Nasqueron Agora - User contributions [en]

Operations grimoire/Deploy with Terraform

2026-03-29T18:27:37Z

Yousra: /* CARP OVH secrets usage */

== Where to work? ==
We deploy from [[Complector]] using <code>/opt/salt/nasqueron-operations</code> as our local copy of rOPS, authoritative for both Salt and Terraform

You need to belong to the <code>ops</code> group to be able to have access and have write-rights on the repository

It's important to work from there to save a shared Terraform state.

== Specific deployment notes ==
=== Vault / OpenBao ===
==== General notes ====
;OpenTofu support
As of 2026-02-07, the Vault provider isn't compiled for FreeBSD. You need to use Terraform instead.

;Vault
You need a Vault token to allow the provider to connect.

You also need to set VAULT_ADDR to https://172.27.27.7:8200 as Vault doesn't listen on 127.0.0.1

<syntaxhighlight lang="shell">
$ export VAULT_ADDR=https://172.27.27.7:8200
$ sudo /opt/salt/nasqueron-operations/utils/vault/issue-admin-token.py > ~/.vault-token

$ cd /opt/salt/nasqueron-operations/terraform/openbao
$ terraform init # if you've a new entry requiring a module, it needs to be installed
$ terraform plan
$ terraform apply
</syntaxhighlight>

It could be interesting to maintain an alternative policy to admin for the Terraform provider, restricted to the current actions. Note benefits would be limited as those include policies management.

==== Propagate secrets (DRP) ====
;No automatic secret rotation
Secrets rotation is disabled with a lifecycle management <code>ignore_changes = [ secret_id, ]</code>.

To rotate a secret, it needs first to be destroyed from terraform state:
<code>terraform destroy -target=module.viperserv_approle.vault_approle_auth_backend_role_secret_id.this</code>

;Full procedure
Once the AppRole have been created in Vault, they need to be provisioned to the relevant configuration files.

<syntaxhighlight lang="shell">
$ cd /opt/salt/nasqueron-operations/terraform/openbao
$ terraform init # if you've a new entry requiring a module, it needs to be installed
$ terraform plan
$ terraform apply

$ cd /opt/salt/nasqueron-operations
$ salt windriver state.sls_id /usr/local/etc/secrets/rhyne-wyse.yaml roles/reports/rhyne-wyse/config
$ salt windriver state.sls_id /srv/viperserv/.credentials roles/viperserv/eggdrop/config
# Missing for router
</syntaxhighlight>

<div class="alert">Important. Each time you'll reprovision the secrets, they will change.

Don't forget to always apply this full procedure.
</div>

=== CARP OVH secrets usage ===

The OVH credentials used by the CARP failover script are stored in Vault at:

'''apps/network/carp-hyper-001-switch'''

These credentials are accessed by router-002 and router-003. They are then used by a script triggered by devd to update the OVH failover IP MAC address depending on the CARP state (MASTER/BACKUP). The script authenticates to Vault using AppRole (via secretsmith) to retrieve them.

The AppRole credentials (role_id and secret_id) used by the script are stored in:

'''ops/secrets/network/router/vault'''

Vault secrets are provisioned using Terraform (openbao module), and must be propagated to the routers using Salt.

The Vault Secrets may need to be rotated, so we need a simple way to propagate
updated credentials to routers. This is enabled by the mechanism
introduced in D4026.

== Table of Terraform states ==

{| class="wikitable"
|+ Terraform and OpenBao states
|-
! Configuration !! State back-end !! Path !! Software to use
|-
| openbao || On disk || /opt/salt/nasqueron-operations/terraform/openbao/terraform.tfstate || Terraform
|}

On disk paths are stored in Complector.

== Troubleshoot ==
=== Error: Module not installed ===

You need to run <code>tofu init</code> to prepare for any new provider.

<syntaxhighlight lang="shell">
$ tofu plan

│ Error: Module not installed
│
│ on rhyne_wyse.tf line 23:
│ 23: module "rhyne_wyse_approle" {
│
│ This module is not yet installed. Run "tofu init" to install all modules required by this configuration.
</syntaxhighlight>

=== Error: Incompatible provider version ===

<syntaxhighlight lang="shell">
$ tofu init

Initializing the backend...
Initializing modules...
- rhyne_wyse_approle in modules/app_credentials

Initializing provider plugins...
- Finding hashicorp/vault versions matching "5.3.0"...
╷
│ Error: Incompatible provider version
│
│ Provider registry.opentofu.org/hashicorp/vault v5.3.0 does not have a package available for your current platform, freebsd_amd64.
│
│ Provider releases are separate from OpenTofu CLI releases, so not all providers are available for all platforms. Other versions of this provider may have different platforms supported.
</syntaxhighlight>

On OpenTofu, some modules are only compiled for Linux, not for FreeBSD.
Switch to Terraform pending a solution to help the OpenTofu builds.

[[Category:Operations grimoire]]
[[Category:Terraform]]

Operations grimoire/Deploy with Terraform

2026-03-29T15:11:03Z

Yousra: /* CARP OVH secrets usage */

== Where to work? ==
We deploy from [[Complector]] using <code>/opt/salt/nasqueron-operations</code> as our local copy of rOPS, authoritative for both Salt and Terraform

You need to belong to the <code>ops</code> group to be able to have access and have write-rights on the repository

It's important to work from there to save a shared Terraform state.

== Specific deployment notes ==
=== Vault / OpenBao ===
==== General notes ====
;OpenTofu support
As of 2026-02-07, the Vault provider isn't compiled for FreeBSD. You need to use Terraform instead.

;Vault
You need a Vault token to allow the provider to connect.

You also need to set VAULT_ADDR to https://172.27.27.7:8200 as Vault doesn't listen on 127.0.0.1

<syntaxhighlight lang="shell">
$ export VAULT_ADDR=https://172.27.27.7:8200
$ sudo /opt/salt/nasqueron-operations/utils/vault/issue-admin-token.py > ~/.vault-token

$ cd /opt/salt/nasqueron-operations/terraform/openbao
$ terraform init # if you've a new entry requiring a module, it needs to be installed
$ terraform plan
$ terraform apply
</syntaxhighlight>

It could be interesting to maintain an alternative policy to admin for the Terraform provider, restricted to the current actions. Note benefits would be limited as those include policies management.

==== Propagate secrets (DRP) ====
;No automatic secret rotation
Secrets rotation is disabled with a lifecycle management <code>ignore_changes = [ secret_id, ]</code>.

To rotate a secret, it needs first to be destroyed from terraform state:
<code>terraform destroy -target=module.viperserv_approle.vault_approle_auth_backend_role_secret_id.this</code>

;Full procedure
Once the AppRole have been created in Vault, they need to be provisioned to the relevant configuration files.

<syntaxhighlight lang="shell">
$ cd /opt/salt/nasqueron-operations/terraform/openbao
$ terraform init # if you've a new entry requiring a module, it needs to be installed
$ terraform plan
$ terraform apply

$ cd /opt/salt/nasqueron-operations
$ salt windriver state.sls_id /usr/local/etc/secrets/rhyne-wyse.yaml roles/reports/rhyne-wyse/config
$ salt windriver state.sls_id /srv/viperserv/.credentials roles/viperserv/eggdrop/config
# Missing for router
</syntaxhighlight>

<div class="alert">Important. Each time you'll reprovision the secrets, they will change.

Don't forget to always apply this full procedure.
</div>

=== CARP OVH secrets usage ===

The OVH credentials used by the CARP failover script are stored in Vault at:

'''apps/network/carp-hyper-001-switch'''

These credentials are accessed by router-002 and router-003. They are then used by a script triggered by devd to update the OVH failover IP MAC address depending on the CARP state (MASTER/BACKUP). The script authenticates to Vault using AppRole (via secretsmith) to retrieve them.

The AppRole credentials (role_id and secret_id) used by the script are stored in:

'''ops/secrets/network/router/vault'''

Vault secrets are provisioned using Terraform (openbao module), and must be propagated to the routers using Salt. This done easily thanks to Linked to {{D|4026}}.

== Table of Terraform states ==

{| class="wikitable"
|+ Terraform and OpenBao states
|-
! Configuration !! State back-end !! Path !! Software to use
|-
| openbao || On disk || /opt/salt/nasqueron-operations/terraform/openbao/terraform.tfstate || Terraform
|}

On disk paths are stored in Complector.

== Troubleshoot ==
=== Error: Module not installed ===

You need to run <code>tofu init</code> to prepare for any new provider.

<syntaxhighlight lang="shell">
$ tofu plan

│ Error: Module not installed
│
│ on rhyne_wyse.tf line 23:
│ 23: module "rhyne_wyse_approle" {
│
│ This module is not yet installed. Run "tofu init" to install all modules required by this configuration.
</syntaxhighlight>

=== Error: Incompatible provider version ===

<syntaxhighlight lang="shell">
$ tofu init

Initializing the backend...
Initializing modules...
- rhyne_wyse_approle in modules/app_credentials

Initializing provider plugins...
- Finding hashicorp/vault versions matching "5.3.0"...
╷
│ Error: Incompatible provider version
│
│ Provider registry.opentofu.org/hashicorp/vault v5.3.0 does not have a package available for your current platform, freebsd_amd64.
│
│ Provider releases are separate from OpenTofu CLI releases, so not all providers are available for all platforms. Other versions of this provider may have different platforms supported.
</syntaxhighlight>

On OpenTofu, some modules are only compiled for Linux, not for FreeBSD.
Switch to Terraform pending a solution to help the OpenTofu builds.

[[Category:Operations grimoire]]
[[Category:Terraform]]

Operations grimoire/Deploy with Terraform

2026-03-29T15:04:21Z

Yousra: /* CARP OVH secrets usage */

== Where to work? ==
We deploy from [[Complector]] using <code>/opt/salt/nasqueron-operations</code> as our local copy of rOPS, authoritative for both Salt and Terraform

You need to belong to the <code>ops</code> group to be able to have access and have write-rights on the repository

It's important to work from there to save a shared Terraform state.

== Specific deployment notes ==
=== Vault / OpenBao ===
==== General notes ====
;OpenTofu support
As of 2026-02-07, the Vault provider isn't compiled for FreeBSD. You need to use Terraform instead.

;Vault
You need a Vault token to allow the provider to connect.

You also need to set VAULT_ADDR to https://172.27.27.7:8200 as Vault doesn't listen on 127.0.0.1

<syntaxhighlight lang="shell">
$ export VAULT_ADDR=https://172.27.27.7:8200
$ sudo /opt/salt/nasqueron-operations/utils/vault/issue-admin-token.py > ~/.vault-token

$ cd /opt/salt/nasqueron-operations/terraform/openbao
$ terraform init # if you've a new entry requiring a module, it needs to be installed
$ terraform plan
$ terraform apply
</syntaxhighlight>

It could be interesting to maintain an alternative policy to admin for the Terraform provider, restricted to the current actions. Note benefits would be limited as those include policies management.

==== Propagate secrets (DRP) ====
;No automatic secret rotation
Secrets rotation is disabled with a lifecycle management <code>ignore_changes = [ secret_id, ]</code>.

To rotate a secret, it needs first to be destroyed from terraform state:
<code>terraform destroy -target=module.viperserv_approle.vault_approle_auth_backend_role_secret_id.this</code>

;Full procedure
Once the AppRole have been created in Vault, they need to be provisioned to the relevant configuration files.

<syntaxhighlight lang="shell">
$ cd /opt/salt/nasqueron-operations/terraform/openbao
$ terraform init # if you've a new entry requiring a module, it needs to be installed
$ terraform plan
$ terraform apply

$ cd /opt/salt/nasqueron-operations
$ salt windriver state.sls_id /usr/local/etc/secrets/rhyne-wyse.yaml roles/reports/rhyne-wyse/config
$ salt windriver state.sls_id /srv/viperserv/.credentials roles/viperserv/eggdrop/config
# Missing for router
</syntaxhighlight>

<div class="alert">Important. Each time you'll reprovision the secrets, they will change.

Don't forget to always apply this full procedure.
</div>

=== CARP OVH secrets usage ===

The OVH credentials used by the CARP failover script are stored in Vault at:

'''apps/network/carp-hyper-001-switch'''

These credentials are accessed by router-002 and router-003. They are then used by a script triggered by devd to update the OVH failover IP MAC address depending on the CARP state (MASTER/BACKUP). The script authenticates to Vault using AppRole (via secretsmith) to retrieve them.

The AppRole credentials (role_id and secret_id) used by the script are stored in:

'''ops/secrets/network/router/vault'''

Vault secrets are provisioned using Terraform (openbao module), and must be propagated to the routers using Salt. This done easily thanks to Linked to {{D|4026}}.

When rotating the secrets OVH :

* Terraform must be applied again to update the secrets in Vault

Failing to do so may cause CARP failover to break due to invalid OVH credentials or expired Vault authentication.

== Table of Terraform states ==

{| class="wikitable"
|+ Terraform and OpenBao states
|-
! Configuration !! State back-end !! Path !! Software to use
|-
| openbao || On disk || /opt/salt/nasqueron-operations/terraform/openbao/terraform.tfstate || Terraform
|}

On disk paths are stored in Complector.

== Troubleshoot ==
=== Error: Module not installed ===

You need to run <code>tofu init</code> to prepare for any new provider.

<syntaxhighlight lang="shell">
$ tofu plan

│ Error: Module not installed
│
│ on rhyne_wyse.tf line 23:
│ 23: module "rhyne_wyse_approle" {
│
│ This module is not yet installed. Run "tofu init" to install all modules required by this configuration.
</syntaxhighlight>

=== Error: Incompatible provider version ===

<syntaxhighlight lang="shell">
$ tofu init

Initializing the backend...
Initializing modules...
- rhyne_wyse_approle in modules/app_credentials

Initializing provider plugins...
- Finding hashicorp/vault versions matching "5.3.0"...
╷
│ Error: Incompatible provider version
│
│ Provider registry.opentofu.org/hashicorp/vault v5.3.0 does not have a package available for your current platform, freebsd_amd64.
│
│ Provider releases are separate from OpenTofu CLI releases, so not all providers are available for all platforms. Other versions of this provider may have different platforms supported.
</syntaxhighlight>

On OpenTofu, some modules are only compiled for Linux, not for FreeBSD.
Switch to Terraform pending a solution to help the OpenTofu builds.

[[Category:Operations grimoire]]
[[Category:Terraform]]

Operations grimoire/Deploy with Terraform

2026-03-29T15:02:37Z

Yousra: /* CARP OVH secrets usage */

== Where to work? ==
We deploy from [[Complector]] using <code>/opt/salt/nasqueron-operations</code> as our local copy of rOPS, authoritative for both Salt and Terraform

You need to belong to the <code>ops</code> group to be able to have access and have write-rights on the repository

It's important to work from there to save a shared Terraform state.

== Specific deployment notes ==
=== Vault / OpenBao ===
==== General notes ====
;OpenTofu support
As of 2026-02-07, the Vault provider isn't compiled for FreeBSD. You need to use Terraform instead.

;Vault
You need a Vault token to allow the provider to connect.

You also need to set VAULT_ADDR to https://172.27.27.7:8200 as Vault doesn't listen on 127.0.0.1

<syntaxhighlight lang="shell">
$ export VAULT_ADDR=https://172.27.27.7:8200
$ sudo /opt/salt/nasqueron-operations/utils/vault/issue-admin-token.py > ~/.vault-token

$ cd /opt/salt/nasqueron-operations/terraform/openbao
$ terraform init # if you've a new entry requiring a module, it needs to be installed
$ terraform plan
$ terraform apply
</syntaxhighlight>

It could be interesting to maintain an alternative policy to admin for the Terraform provider, restricted to the current actions. Note benefits would be limited as those include policies management.

==== Propagate secrets (DRP) ====
;No automatic secret rotation
Secrets rotation is disabled with a lifecycle management <code>ignore_changes = [ secret_id, ]</code>.

To rotate a secret, it needs first to be destroyed from terraform state:
<code>terraform destroy -target=module.viperserv_approle.vault_approle_auth_backend_role_secret_id.this</code>

;Full procedure
Once the AppRole have been created in Vault, they need to be provisioned to the relevant configuration files.

<syntaxhighlight lang="shell">
$ cd /opt/salt/nasqueron-operations/terraform/openbao
$ terraform init # if you've a new entry requiring a module, it needs to be installed
$ terraform plan
$ terraform apply

$ cd /opt/salt/nasqueron-operations
$ salt windriver state.sls_id /usr/local/etc/secrets/rhyne-wyse.yaml roles/reports/rhyne-wyse/config
$ salt windriver state.sls_id /srv/viperserv/.credentials roles/viperserv/eggdrop/config
# Missing for router
</syntaxhighlight>

<div class="alert">Important. Each time you'll reprovision the secrets, they will change.

Don't forget to always apply this full procedure.
</div>

=== CARP OVH secrets usage ===

The OVH credentials used by the CARP failover script are stored in Vault at:

'''apps/network/carp-hyper-001-switch'''

These credentials are accessed by router-002 and router-003. They are then used by a script triggered by devd to update the OVH failover IP MAC address depending on the CARP state (MASTER/BACKUP). The script authenticates to Vault using AppRole (via secretsmith) to retrieve them.

The AppRole credentials (role_id and secret_id) used by the script are stored in:

'''ops/secrets/network/router/vault'''

Vault secrets are provisioned using Terraform (openbao module), and must be propagated to the routers using Salt. This done easily thanks to Linked to {{D|4026}}.

When rotating Vault secrets managed through Terraform:

* Terraform must be applied again to update them in Vault
* if Vault AppRole credentials are changed, they must also be propagated to the routers using Salt

When rotating the secrets OVH :

* Terraform must be applied again to update the secrets in Vault

Failing to do so may cause CARP failover to break due to invalid OVH credentials or expired Vault authentication.

== Table of Terraform states ==

{| class="wikitable"
|+ Terraform and OpenBao states
|-
! Configuration !! State back-end !! Path !! Software to use
|-
| openbao || On disk || /opt/salt/nasqueron-operations/terraform/openbao/terraform.tfstate || Terraform
|}

On disk paths are stored in Complector.

== Troubleshoot ==
=== Error: Module not installed ===

You need to run <code>tofu init</code> to prepare for any new provider.

<syntaxhighlight lang="shell">
$ tofu plan

│ Error: Module not installed
│
│ on rhyne_wyse.tf line 23:
│ 23: module "rhyne_wyse_approle" {
│
│ This module is not yet installed. Run "tofu init" to install all modules required by this configuration.
</syntaxhighlight>

=== Error: Incompatible provider version ===

<syntaxhighlight lang="shell">
$ tofu init

Initializing the backend...
Initializing modules...
- rhyne_wyse_approle in modules/app_credentials

Initializing provider plugins...
- Finding hashicorp/vault versions matching "5.3.0"...
╷
│ Error: Incompatible provider version
│
│ Provider registry.opentofu.org/hashicorp/vault v5.3.0 does not have a package available for your current platform, freebsd_amd64.
│
│ Provider releases are separate from OpenTofu CLI releases, so not all providers are available for all platforms. Other versions of this provider may have different platforms supported.
</syntaxhighlight>

On OpenTofu, some modules are only compiled for Linux, not for FreeBSD.
Switch to Terraform pending a solution to help the OpenTofu builds.

[[Category:Operations grimoire]]
[[Category:Terraform]]

Operations grimoire/Deploy with Terraform

2026-03-29T15:01:12Z

Yousra: /* CARP OVH secrets usage */

== Where to work? ==
We deploy from [[Complector]] using <code>/opt/salt/nasqueron-operations</code> as our local copy of rOPS, authoritative for both Salt and Terraform

You need to belong to the <code>ops</code> group to be able to have access and have write-rights on the repository

It's important to work from there to save a shared Terraform state.

== Specific deployment notes ==
=== Vault / OpenBao ===
==== General notes ====
;OpenTofu support
As of 2026-02-07, the Vault provider isn't compiled for FreeBSD. You need to use Terraform instead.

;Vault
You need a Vault token to allow the provider to connect.

You also need to set VAULT_ADDR to https://172.27.27.7:8200 as Vault doesn't listen on 127.0.0.1

<syntaxhighlight lang="shell">
$ export VAULT_ADDR=https://172.27.27.7:8200
$ sudo /opt/salt/nasqueron-operations/utils/vault/issue-admin-token.py > ~/.vault-token

$ cd /opt/salt/nasqueron-operations/terraform/openbao
$ terraform init # if you've a new entry requiring a module, it needs to be installed
$ terraform plan
$ terraform apply
</syntaxhighlight>

It could be interesting to maintain an alternative policy to admin for the Terraform provider, restricted to the current actions. Note benefits would be limited as those include policies management.

==== Propagate secrets (DRP) ====
;No automatic secret rotation
Secrets rotation is disabled with a lifecycle management <code>ignore_changes = [ secret_id, ]</code>.

To rotate a secret, it needs first to be destroyed from terraform state:
<code>terraform destroy -target=module.viperserv_approle.vault_approle_auth_backend_role_secret_id.this</code>

;Full procedure
Once the AppRole have been created in Vault, they need to be provisioned to the relevant configuration files.

<syntaxhighlight lang="shell">
$ cd /opt/salt/nasqueron-operations/terraform/openbao
$ terraform init # if you've a new entry requiring a module, it needs to be installed
$ terraform plan
$ terraform apply

$ cd /opt/salt/nasqueron-operations
$ salt windriver state.sls_id /usr/local/etc/secrets/rhyne-wyse.yaml roles/reports/rhyne-wyse/config
$ salt windriver state.sls_id /srv/viperserv/.credentials roles/viperserv/eggdrop/config
# Missing for router
</syntaxhighlight>

<div class="alert">Important. Each time you'll reprovision the secrets, they will change.

Don't forget to always apply this full procedure.
</div>

=== CARP OVH secrets usage ===

The OVH credentials used by the CARP failover script are stored in Vault at:

'''apps/network/carp-hyper-001-switch'''

These credentials are accessed by router-002 and router-003. They are then used by a script triggered by devd to update the OVH failover IP MAC address depending on the CARP state (MASTER/BACKUP). The script authenticates to Vault using AppRole (via secretsmith) to retrieve them.

The AppRole credentials (role_id and secret_id) used by the script are stored in:

'''ops/secrets/network/router/vault'''

Vault secrets are provisioned using Terraform (openbao module), and must be propagated to the routers using Salt. This done thanks to Linked to {{D|4026}}

When rotating Vault secrets managed through Terraform:

* Terraform must be applied again to update them in Vault
* if Vault AppRole credentials are changed, they must also be propagated to the routers using Salt

When rotating the secrets OVH :

* Terraform must be applied again to update the secrets in Vault

Failing to do so may cause CARP failover to break due to invalid OVH credentials or expired Vault authentication.

== Table of Terraform states ==

{| class="wikitable"
|+ Terraform and OpenBao states
|-
! Configuration !! State back-end !! Path !! Software to use
|-
| openbao || On disk || /opt/salt/nasqueron-operations/terraform/openbao/terraform.tfstate || Terraform
|}

On disk paths are stored in Complector.

== Troubleshoot ==
=== Error: Module not installed ===

You need to run <code>tofu init</code> to prepare for any new provider.

<syntaxhighlight lang="shell">
$ tofu plan

│ Error: Module not installed
│
│ on rhyne_wyse.tf line 23:
│ 23: module "rhyne_wyse_approle" {
│
│ This module is not yet installed. Run "tofu init" to install all modules required by this configuration.
</syntaxhighlight>

=== Error: Incompatible provider version ===

<syntaxhighlight lang="shell">
$ tofu init

Initializing the backend...
Initializing modules...
- rhyne_wyse_approle in modules/app_credentials

Initializing provider plugins...
- Finding hashicorp/vault versions matching "5.3.0"...
╷
│ Error: Incompatible provider version
│
│ Provider registry.opentofu.org/hashicorp/vault v5.3.0 does not have a package available for your current platform, freebsd_amd64.
│
│ Provider releases are separate from OpenTofu CLI releases, so not all providers are available for all platforms. Other versions of this provider may have different platforms supported.
</syntaxhighlight>

On OpenTofu, some modules are only compiled for Linux, not for FreeBSD.
Switch to Terraform pending a solution to help the OpenTofu builds.

[[Category:Operations grimoire]]
[[Category:Terraform]]

Protocol CARP

2026-03-29T15:00:53Z

Yousra: /* ESXI and CARP */

== Introduction and history ==

« CARP stands for Common Address Redundancy Protocol and its basic functionality is to allow multiple hosts to share a set of IP addresses ».

CARP was developed by the OpenBSD project in 2003 as an alternative to VRRP due to patent concerns related to Cisco's HSRP technology. To avoid potential legal issues and remain compatible with open-source principles, OpenBSD implemented its own redundancy protocol with added cryptographic features. CARP was later implemented directly in the kernel of other BSD systems such as FreeBSD and NetBSD. A userland implementation of CARP, named ucarp, made the protocol usable on Linux without requiring kernel integration.

These patent concerns originated from the development of VRRP by the Internet Engineering Task Force (IETF). Indeed, when the Internet Engineering Task Force (IETF) developed the Virtual Router Redundancy Protocol (VRRP), Cisco Systems indicated that it owned patents related to router redundancy mechanisms used in the protocol Hot Standby Router Protocol (HSRP). This created a legal risk for developers, particularly for open-source projects that wanted to implement VRRP freely.

Sources :

https://en.wikipedia.org/wiki/Common_Address_Redundancy_Protocol

https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf

== Master host and backup hosts ==

CARP creates a redundancy group, meaning several hosts are configured to share a virtual IP address. However, at any given time, only one host uses the shared IP address. This host is called the Master host and it receives and handles all traffic destined to that virtual IP.

When the Master host becomes unavailable (it crashed, turned off, or lost its network connection), the other hosts in the same redundancy group detect the failure. Immediately, one of the Backup hosts is elected as the new Master host. That means that it will take over the shared IP address.

This switch happens automatically, ensuring service continuity without clients noticing any interruption.

[[File:Schema redondance routeurs.png]]

Sources :

https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf

https://wxcafe.net/posts/redondance-routeurs-openbsd-freebsd/

== CARP advertisements ==

- The Master host periodically broadcasts CARP advertisements to the backup hosts.

- The Backup hosts listen but do not send any packets.

'''Each CARP advertisement contains :'''

1. The CARP version and packet type (CARP Header).

2. The VHID (Virtual Host ID), which identifies the redundancy group (CARP Header).

3. The advertisement parameters (advbase and advskew), which determine the host’s priority CARP (CARP Header).

4. The counter

5. The signature in SHA-1

'''CARP Advertisement Packet Format :'''

Each line represent 32 bits, 4 bytes

Length : 36 bytes

<pre>
/*
* The CARP header layout is as follows:
*
* 0 1 2 3
* 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
* |Version| Type | VirtualHostID | AdvSkew | Auth Len |
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
* | Demotion | AdvBase | Checksum |
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
* | Counter (1) |
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
* | Counter (2) |
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
* | SHA-1 HMAC (1) |
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
* | SHA-1 HMAC (2) |
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
* | SHA-1 HMAC (3) |
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
* | SHA-1 HMAC (4) |
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
* | SHA-1 HMAC (5) |
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
*
*/
</pre>

'''commande : tcpdump -i <interface> proto 112'''

[[ File:Carp-advertissement.png ]]

Sources :

https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf

https://github.com/openbsd/src/blob/master/sys/netinet/ip_carp.h

== Cryptography ==

'''All CARP advertisements are cryptographically signed :'''

Steps :

1. The Master host creates an advertisement.

2. It calculates a cryptographic signature, a SHA-1 hash using:

* the shared secret,

* the VHID,

* the virtual IP address,

* the advertisement parameters (advbase and advskew),

* the counter

* and other header fields.

3. It then sends the signed advertisement.

4. The Backup hosts receives the signed advertisement.

5. Each Backup host recalculates the signature using the same shared secret :

- If the calculated signature matches the received signature → the packet is considered valid because authentication and integrity are insured.

- If it does not match → the packet is rejected.

It protects against spoofing attacks : without cryptographic authentication, an attacker could send fake CARP advertisements and attempt to pretend to be or to become the Master host.

'''command : tcpdump -X -ni <interface> proto 112'''

We can observe the complete packet in the capture. Each hexadecimal value represents two bytes. The first 20 bytes correspond to the IP header, while the last 20 bytes correspond to the SHA-1 authentication hash included in the CARP advertisement.

[[ File:Sha1-signature.png ]]

Sources :

https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf

https://www.giac.org/paper/gsec/4031/carp-free-fail-over-protocol/106433

== IP protocol number ==

CARP uses the same IP protocol number as VRRP (112). As a result, packet capture tools such as tcpdump interpret CARP packets as VRRP advertisements like we see above, even though the packets actually belong to the CARP protocol.

Why ??
The OpenBSD developers initially requested unique identifiers from the Internet Assigned Numbers Authority (IANA), but the request was denied because they had not followed the IETF standardization process. As a result, they chose to use protocol number 112, which was already assigned to VRRP. Despite this overlap, VRRP and CARP can coexist on the same network as long as their identifiers (VRRP group ID and CARP VHID) are different.

Sources :

https://en.wikipedia.org/wiki/Common_Address_Redundancy_Protocol

== Multiple VHID : Distribute traffic ==

Using multiple VHIDs in CARP allows several virtual IP addresses to be managed separately. This means different hosts can be Master for different services. For example, one host can manage the web service, while another manages the DNS service. Thanks to this, the hosts can be used efficiently while still ensuring high availability. It would not be practical to use a machine only when the other machine is not responding.

sources :

https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf

== Split-brain ==

Normally:
* one Master

* one or several Backups

If the Master host stops sending advertisements, the Backup hosts know that the master is no longer reachable. However, a problem can occur if the Backup hosts stop receiving each other's advertisements because the link between them is broken. In that case, each node may assume it is the Master.

When communication is restored, CARP resolves the problem:

1. The Masters can see each other
2. The real Master send the packet
3. The hosts compare CARP priority
4. The hosts with lower priority automatically switch back to Backup

sources :

https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf

== Master Election Process ==

1. All hosts start

All hosts begin in BACKUP state.

2. Each host starts a timer, a counter

* The delay depends on the CARP priority.

* The lower the advskew, the shorter the delay.

* The lower the advskew, the higher the CARP priority.

3. The higher-priority host’s timer expires first

That host sends CARP advertisements when the timer expires.

It assigns the virtual IP (VIP) to its interface.

It becomes the MASTER host.

4. The other hosts receive the advertisement

They detect that a higher-priority host is active.

They remain in BACKUP state.

They do not send CARP advertisements.

In CARP, we usually choose the preferred Master by giving it a higher priority (lower advskew).

== Virtual Mac Address ==

The protocol automatically generates a virtual MAC address based on the VHID, ending with the VHID's value (00:00:5e:00:01:XX). This MAC address is identical on both hosts and is used by the MASTER node.

We can see here that the virtual mac address associated to the vhid2 is : 00:00:5e:00:01:02 (02 at the end for vhid2)

'''command : tcpdump -e -ni vmx1 proto 112'''
[[ File:Virtual-mac.png ]]

✔ The CARP virtual MAC address is visible.

❌ The virtual IP address (VIP) is not visible as the source.

In CARP advertisements, the Ethernet source address corresponds to the virtual CARP MAC address derived from the VHID. However, the IP source address is the physical IP address of the router sending the advertisement. The virtual IP address (VIP) is not used as the source IP in these control packets. Instead, the VIP is used for normal traffic between clients and the virtual gateway.

sources :

https://dynfi.com/documentations/dynfi-firewall/configuration_firewall_VIP.html

== Address Multicast CARP ==

In the Common Address Redundancy Protocol (CARP), routers communicate with each other using a multicast address. CARP advertisements are sent to the IPv4 multicast address '''224.0.0.18''', which allows all routers participating in the CARP group to receive the messages simultaneously. At the Ethernet level, this multicast IP address corresponds to the multicast MAC address '''01:00:5e:00:00:12'''.

== Priority vs advskew. ==

In the CARP protocol, the Master router is not directly determined by priority, but rather by a value called advskew. Advskew is a numerical parameter between 0 and 255: the lower its value, the more likely the router is to become the Master. Priority, on the other hand, is a more intuitive abstraction used in automation tools or logical configurations: the higher the priority, the higher the router's expected priority. To link the two, we use the formula: advskew = 255 − priority. Thus, a router with a priority of 200 will have a low advskew (55) and will become the Master, while a router with a priority of 0 will have a maximum advskew (255) and will remain in Backup mode. In short, priority is a value that is easily understood and logical for humans, while advskew is the value actually used by CARP's internal mechanism to determine the Master or Backup role. However, advskew can also be called "CARP priority".

== Preemption ==

In fact, after some research, I discovered that on FreeBSD, the preemption behavior of CARP (Common Address Redundancy Protocol) is controlled by the kernel parameter net.inet.carp.preempt. This parameter allows a router with a higher priority (a lower advskew) to become MASTER, even if another router is already active in that role or if the real Master was restart after. To check if preemption is enabled, you can use the command :

'''sysctl net.inet.carp.preempt'''

If the returned value is 0, preemption is disabled, and the currently MASTER router will retain that role as long as it continues to send its CARP advertisements.

To temporarily enable preemption, simply run : '''sysctl net.inet.carp.preempt=1'''.

To make this behavior permanent, you must add the line '''net.inet.carp.preempt=1''' to the /etc/sysctl.conf file.

Sources :

https://man.freebsd.org/cgi/man.cgi?carp(4)

https://vincentdelft.be/post/post_20241115

== Peer ==

By default, CARP uses multicast to send advertisements. However, multicast traffic can be filtered, restricted and so unreliable. Configuring CARP to use unicast advertisements between peers can help ensure more stable and predictable high-availability operation.

In addition to that, using CARP in unicast mode can be appropriate in a two-router topology, as communication only needs to occur between 2 specific peers.

Above all IPsec (security added in a classic tunnel) protects unicast traffic, it does not protect (or protects very poorly) multicast traffic. So if we want CARP to be used with IPsec, peer is an excellent choice.

Router-002 :

'''ifconfig_vmx1_alias0="inet vhid 2 advskew 100 pass seediez7ae7LooFevohshae4oot0EeR peer 178.32.70.111 alias 51.68.252.230/32"'''

Router-003 :

'''ifconfig_vmx1_alias0="inet vhid 2 advskew 0 pass seediez7ae7LooFevohshae4oot0EeR peer 178.32.70.110 alias 51.68.252.230/32"'''

We see that the Master router-003 sends advertissements to router-002.nasqueron.org/178.32.70.110
[[ File:Peer.png ]]

Sources :

https://man.freebsd.org/cgi/man.cgi?query=carp

https://docs.opnsense.org/manual/how-tos/carp.html

https://www.kernel-panic.it/openbsd/carp/carp4.html

== Basic configuration ==

We want :

router-002 to act as the master, while router-003 should operate as the backup for Intranought network.

router-003 to act as the master, while router-002 should operate as the backup for Public network.

1) Enable CARP Support

On the file ''/boot/loader.conf'', you need to add : '''carp_load="YES"'''

You are telling FreeBSD:

“Load the CARP kernel module automatically at boot time.”

If you want to load it now, but not permanently : '''kldload carp'''

If someone has build a custom FreeBSD kernel, they can include CARP directly by adding this line to the kernel configuration file: '''device carp'''

2) CARP configuration

In each router, on the new appropriate file '''/etc/rc.conf.d/netif/carp''', we will have the CARP configuration

router-002 :

ifconfig_vmx0_alias0="inet vhid 1 advskew 0 pass xxxxxxx alias 172.27.27.13/27"

ifconfig_vmx1_alias0="inet vhid 2 advskew 100 pass xxxxxxx alias 51.68.252.230/32"

router-003 :

ifconfig_vmx0_alias0="inet vhid 1 advskew 100 pass xxxxxxx alias 172.27.27.13/27"

ifconfig_vmx1_alias0="inet vhid 2 advskew 0 pass xxxxxxx alias 51.68.252.230/32"

So we gave a lower advskew (default is 0) to become the master router.

In older versions of FreeBSD (9 and earlier), CARP required creating a separate virtual interface such as carp0, and the shared IP address was attached to that interface. Starting with FreeBSD 10, this was simplified. CARP can now be configured directly on the physical interface using an alias with a VHID. The newer method is easier to configure and maintain, so it is recommended to use it in modern versions of FreeBSD.

commands :

ifconfig carp0 create
ifconfig carp0 vhid 1 pass xxxxxx
ifconfig carp0 inet 172.27.27.X/28'''

sources :

https://docs-archive.freebsd.org/doc/11.0-RELEASE/usr/local/share/doc/freebsd/en/books/handbook/carp.html

== Automatic blue/green deployment ==

In this context, “automatic Blue-Green” does not refer to a traditional application deployment strategy. Instead, it describes a network-level behavior enabled by CARP.

With CARP, two routers are deployed in parallel:

- One acts as the active router (MASTER)

- The other remains in standby (BACKUP)

Both routers are fully operational and connected to the network. Traffic is sent to a shared virtual IP address. At any given time, only the MASTER processes the traffic. However, if the MASTER fails, the BACKUP automatically takes over the virtual IP address without manual intervention.

This behavior resembles Blue-Green deployment because:

1. Two environments exist simultaneously

2. Only one actively handles traffic

3. Traffic can switch from one to the other

4. The switch is transparent to users

The difference is that in traditional Blue-Green deployment, the switch is usually performed manually during an update. With CARP, the switch happens automatically in case of failure.

== ESXI and CARP ==

When using Common Address Redundancy Protocol in a virtualized environment such as VMware ESXi, specific security settings must be adjusted on the virtual switch. CARP relies on the use of a virtual MAC address that can move between routers depending on which device is currently the Master. However, ESXi applies strict security policies by default to prevent virtual machines from sending or receiving traffic with unexpected MAC addresses. To ensure that CARP functions correctly, three settings must be enabled on the vSwitch or port group.

First, '''Promiscuous Mode''' must be enabled. This allows a virtual machine to receive all frames circulating on the virtual network, even if the destination MAC address does not match the interface’s MAC address. This is important for CARP because routers must receive multicast advertisements sent to the CARP group.

Second, '''MAC Address Changes''' must be allowed. CARP dynamically assigns a virtual MAC address to the Master router. When a failover occurs, the Backup router must adopt this same virtual MAC address. If ESXi blocks MAC address changes, the router will not be able to assume the virtual MAC address and the failover mechanism will fail.

Finally, '''Forged Transmits''' must also be permitted. This option allows a virtual machine to send frames with a source MAC address different from the one originally assigned to the virtual network interface. Since CARP advertisements and ARP responses are sent using the virtual MAC address rather than the physical interface MAC address, this setting is required to allow those frames to be transmitted correctly.

Together, these three settings ensure that CARP redundancy mechanisms operate properly in a virtualized ESXi network environment.

Configuration to enable "Promiscuous Mode", “MAC Address Changes” and “Forged Transmits” :

https://knowledge.broadcom.com/external/article/324520/configuring-promiscuous-mode-on-a-virtua.html

Sources :

https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/vsphere-security/securing-vsphere-networking/securing-vsphere-standard-switches.html

Linked to {{T|2227}}

Operations grimoire/Deploy with Terraform

2026-03-29T14:34:51Z

Yousra: /* CARP OVH secrets usage */

== Where to work? ==
We deploy from [[Complector]] using <code>/opt/salt/nasqueron-operations</code> as our local copy of rOPS, authoritative for both Salt and Terraform

You need to belong to the <code>ops</code> group to be able to have access and have write-rights on the repository

It's important to work from there to save a shared Terraform state.

== Specific deployment notes ==
=== Vault / OpenBao ===
==== General notes ====
;OpenTofu support
As of 2026-02-07, the Vault provider isn't compiled for FreeBSD. You need to use Terraform instead.

;Vault
You need a Vault token to allow the provider to connect.

You also need to set VAULT_ADDR to https://172.27.27.7:8200 as Vault doesn't listen on 127.0.0.1

<syntaxhighlight lang="shell">
$ export VAULT_ADDR=https://172.27.27.7:8200
$ sudo /opt/salt/nasqueron-operations/utils/vault/issue-admin-token.py > ~/.vault-token

$ cd /opt/salt/nasqueron-operations/terraform/openbao
$ terraform init # if you've a new entry requiring a module, it needs to be installed
$ terraform plan
$ terraform apply
</syntaxhighlight>

It could be interesting to maintain an alternative policy to admin for the Terraform provider, restricted to the current actions. Note benefits would be limited as those include policies management.

==== Propagate secrets (DRP) ====
;No automatic secret rotation
Secrets rotation is disabled with a lifecycle management <code>ignore_changes = [ secret_id, ]</code>.

To rotate a secret, it needs first to be destroyed from terraform state:
<code>terraform destroy -target=module.viperserv_approle.vault_approle_auth_backend_role_secret_id.this</code>

;Full procedure
Once the AppRole have been created in Vault, they need to be provisioned to the relevant configuration files.

<syntaxhighlight lang="shell">
$ cd /opt/salt/nasqueron-operations/terraform/openbao
$ terraform init # if you've a new entry requiring a module, it needs to be installed
$ terraform plan
$ terraform apply

$ cd /opt/salt/nasqueron-operations
$ salt windriver state.sls_id /usr/local/etc/secrets/rhyne-wyse.yaml roles/reports/rhyne-wyse/config
$ salt windriver state.sls_id /srv/viperserv/.credentials roles/viperserv/eggdrop/config
# Missing for router
</syntaxhighlight>

<div class="alert">Important. Each time you'll reprovision the secrets, they will change.

Don't forget to always apply this full procedure.
</div>

=== CARP OVH secrets usage ===

The OVH credentials used by the CARP failover script are stored in Vault at:

'''apps/network/carp-hyper-001-switch'''

These credentials are accessed by router-002 and router-003. They are then used by a script triggered by devd to update the OVH failover IP MAC address depending on the CARP state (MASTER/BACKUP). The script authenticates to Vault using AppRole (via secretsmith) to retrieve them.

The AppRole credentials (role_id and secret_id) used by the script are stored in:

'''ops/secrets/network/router/vault'''

Vault secrets are provisioned using Terraform (openbao module), and must be propagated to the routers using Salt.

When rotating the secrets OVH :

* Terraform must be applied again to update the secrets in Vault

Failing to do so may cause CARP failover to break due to invalid OVH credentials or expired Vault authentication.

== Table of Terraform states ==

{| class="wikitable"
|+ Terraform and OpenBao states
|-
! Configuration !! State back-end !! Path !! Software to use
|-
| openbao || On disk || /opt/salt/nasqueron-operations/terraform/openbao/terraform.tfstate || Terraform
|}

On disk paths are stored in Complector.

== Troubleshoot ==
=== Error: Module not installed ===

You need to run <code>tofu init</code> to prepare for any new provider.

<syntaxhighlight lang="shell">
$ tofu plan

│ Error: Module not installed
│
│ on rhyne_wyse.tf line 23:
│ 23: module "rhyne_wyse_approle" {
│
│ This module is not yet installed. Run "tofu init" to install all modules required by this configuration.
</syntaxhighlight>

=== Error: Incompatible provider version ===

<syntaxhighlight lang="shell">
$ tofu init

Initializing the backend...
Initializing modules...
- rhyne_wyse_approle in modules/app_credentials

Initializing provider plugins...
- Finding hashicorp/vault versions matching "5.3.0"...
╷
│ Error: Incompatible provider version
│
│ Provider registry.opentofu.org/hashicorp/vault v5.3.0 does not have a package available for your current platform, freebsd_amd64.
│
│ Provider releases are separate from OpenTofu CLI releases, so not all providers are available for all platforms. Other versions of this provider may have different platforms supported.
</syntaxhighlight>

On OpenTofu, some modules are only compiled for Linux, not for FreeBSD.
Switch to Terraform pending a solution to help the OpenTofu builds.

[[Category:Operations grimoire]]
[[Category:Terraform]]

Operations grimoire/Deploy with Terraform

2026-03-29T14:33:57Z

Yousra: /* CARP OVH secrets usage */

== Where to work? ==
We deploy from [[Complector]] using <code>/opt/salt/nasqueron-operations</code> as our local copy of rOPS, authoritative for both Salt and Terraform

You need to belong to the <code>ops</code> group to be able to have access and have write-rights on the repository

It's important to work from there to save a shared Terraform state.

== Specific deployment notes ==
=== Vault / OpenBao ===
==== General notes ====
;OpenTofu support
As of 2026-02-07, the Vault provider isn't compiled for FreeBSD. You need to use Terraform instead.

;Vault
You need a Vault token to allow the provider to connect.

You also need to set VAULT_ADDR to https://172.27.27.7:8200 as Vault doesn't listen on 127.0.0.1

<syntaxhighlight lang="shell">
$ export VAULT_ADDR=https://172.27.27.7:8200
$ sudo /opt/salt/nasqueron-operations/utils/vault/issue-admin-token.py > ~/.vault-token

$ cd /opt/salt/nasqueron-operations/terraform/openbao
$ terraform init # if you've a new entry requiring a module, it needs to be installed
$ terraform plan
$ terraform apply
</syntaxhighlight>

It could be interesting to maintain an alternative policy to admin for the Terraform provider, restricted to the current actions. Note benefits would be limited as those include policies management.

==== Propagate secrets (DRP) ====
;No automatic secret rotation
Secrets rotation is disabled with a lifecycle management <code>ignore_changes = [ secret_id, ]</code>.

To rotate a secret, it needs first to be destroyed from terraform state:
<code>terraform destroy -target=module.viperserv_approle.vault_approle_auth_backend_role_secret_id.this</code>

;Full procedure
Once the AppRole have been created in Vault, they need to be provisioned to the relevant configuration files.

<syntaxhighlight lang="shell">
$ cd /opt/salt/nasqueron-operations/terraform/openbao
$ terraform init # if you've a new entry requiring a module, it needs to be installed
$ terraform plan
$ terraform apply

$ cd /opt/salt/nasqueron-operations
$ salt windriver state.sls_id /usr/local/etc/secrets/rhyne-wyse.yaml roles/reports/rhyne-wyse/config
$ salt windriver state.sls_id /srv/viperserv/.credentials roles/viperserv/eggdrop/config
# Missing for router
</syntaxhighlight>

<div class="alert">Important. Each time you'll reprovision the secrets, they will change.

Don't forget to always apply this full procedure.
</div>

=== CARP OVH secrets usage ===

The OVH credentials used by the CARP failover script are stored in Vault at:

'''apps/network/carp-hyper-001-switch'''

These credentials are accessed by router-002 and router-003. They are then used by a script triggered by devd to update the OVH failover IP MAC address depending on the CARP state (MASTER/BACKUP). The script authenticates to Vault using AppRole (via secretsmith) to retrieve them.

The AppRole credentials (role_id and secret_id) used by the script are stored in:

'''ops/secrets/network/router/vault'''

Vault secrets are provisioned using Terraform (openbao module), and must be propagated to the routers using Salt.

When rotating the secrets Vault or OVH :

* Terraform must be applied again to update the secrets in Vault

Failing to do so may cause CARP failover to break due to invalid OVH credentials or expired Vault authentication.

== Table of Terraform states ==

{| class="wikitable"
|+ Terraform and OpenBao states
|-
! Configuration !! State back-end !! Path !! Software to use
|-
| openbao || On disk || /opt/salt/nasqueron-operations/terraform/openbao/terraform.tfstate || Terraform
|}

On disk paths are stored in Complector.

== Troubleshoot ==
=== Error: Module not installed ===

You need to run <code>tofu init</code> to prepare for any new provider.

<syntaxhighlight lang="shell">
$ tofu plan

│ Error: Module not installed
│
│ on rhyne_wyse.tf line 23:
│ 23: module "rhyne_wyse_approle" {
│
│ This module is not yet installed. Run "tofu init" to install all modules required by this configuration.
</syntaxhighlight>

=== Error: Incompatible provider version ===

<syntaxhighlight lang="shell">
$ tofu init

Initializing the backend...
Initializing modules...
- rhyne_wyse_approle in modules/app_credentials

Initializing provider plugins...
- Finding hashicorp/vault versions matching "5.3.0"...
╷
│ Error: Incompatible provider version
│
│ Provider registry.opentofu.org/hashicorp/vault v5.3.0 does not have a package available for your current platform, freebsd_amd64.
│
│ Provider releases are separate from OpenTofu CLI releases, so not all providers are available for all platforms. Other versions of this provider may have different platforms supported.
</syntaxhighlight>

On OpenTofu, some modules are only compiled for Linux, not for FreeBSD.
Switch to Terraform pending a solution to help the OpenTofu builds.

[[Category:Operations grimoire]]
[[Category:Terraform]]

Operations grimoire/Deploy with Terraform

2026-03-29T14:33:11Z

Yousra: /* CARP OVH secrets usage */

== Where to work? ==
We deploy from [[Complector]] using <code>/opt/salt/nasqueron-operations</code> as our local copy of rOPS, authoritative for both Salt and Terraform

You need to belong to the <code>ops</code> group to be able to have access and have write-rights on the repository

It's important to work from there to save a shared Terraform state.

== Specific deployment notes ==
=== Vault / OpenBao ===
==== General notes ====
;OpenTofu support
As of 2026-02-07, the Vault provider isn't compiled for FreeBSD. You need to use Terraform instead.

;Vault
You need a Vault token to allow the provider to connect.

You also need to set VAULT_ADDR to https://172.27.27.7:8200 as Vault doesn't listen on 127.0.0.1

<syntaxhighlight lang="shell">
$ export VAULT_ADDR=https://172.27.27.7:8200
$ sudo /opt/salt/nasqueron-operations/utils/vault/issue-admin-token.py > ~/.vault-token

$ cd /opt/salt/nasqueron-operations/terraform/openbao
$ terraform init # if you've a new entry requiring a module, it needs to be installed
$ terraform plan
$ terraform apply
</syntaxhighlight>

It could be interesting to maintain an alternative policy to admin for the Terraform provider, restricted to the current actions. Note benefits would be limited as those include policies management.

==== Propagate secrets (DRP) ====
;No automatic secret rotation
Secrets rotation is disabled with a lifecycle management <code>ignore_changes = [ secret_id, ]</code>.

To rotate a secret, it needs first to be destroyed from terraform state:
<code>terraform destroy -target=module.viperserv_approle.vault_approle_auth_backend_role_secret_id.this</code>

;Full procedure
Once the AppRole have been created in Vault, they need to be provisioned to the relevant configuration files.

<syntaxhighlight lang="shell">
$ cd /opt/salt/nasqueron-operations/terraform/openbao
$ terraform init # if you've a new entry requiring a module, it needs to be installed
$ terraform plan
$ terraform apply

$ cd /opt/salt/nasqueron-operations
$ salt windriver state.sls_id /usr/local/etc/secrets/rhyne-wyse.yaml roles/reports/rhyne-wyse/config
$ salt windriver state.sls_id /srv/viperserv/.credentials roles/viperserv/eggdrop/config
# Missing for router
</syntaxhighlight>

<div class="alert">Important. Each time you'll reprovision the secrets, they will change.

Don't forget to always apply this full procedure.
</div>

=== CARP OVH secrets usage ===

The OVH credentials used by the CARP failover script are stored in Vault at:

'''apps/network/carp-hyper-001-switch'''

These credentials are accessed by router-002 and router-003. They are then used by a script triggered by devd to update the OVH failover IP MAC address depending on the CARP state (MASTER/BACKUP). The script authenticates to Vault using AppRole (via secretsmith) to retrieve them.

The AppRole credentials (role_id and secret_id) used by the script are stored in:

'''ops/secrets/network/router/vault'''

Vault secrets are provisioned using Terraform (openbao module), and must be propagated to the routers using Salt.

When rotating the secrets:

* Terraform must be applied again to update the secrets in Vault

Failing to do so may cause CARP failover to break due to invalid OVH credentials or expired Vault authentication.

== Table of Terraform states ==

{| class="wikitable"
|+ Terraform and OpenBao states
|-
! Configuration !! State back-end !! Path !! Software to use
|-
| openbao || On disk || /opt/salt/nasqueron-operations/terraform/openbao/terraform.tfstate || Terraform
|}

On disk paths are stored in Complector.

== Troubleshoot ==
=== Error: Module not installed ===

You need to run <code>tofu init</code> to prepare for any new provider.

<syntaxhighlight lang="shell">
$ tofu plan

│ Error: Module not installed
│
│ on rhyne_wyse.tf line 23:
│ 23: module "rhyne_wyse_approle" {
│
│ This module is not yet installed. Run "tofu init" to install all modules required by this configuration.
</syntaxhighlight>

=== Error: Incompatible provider version ===

<syntaxhighlight lang="shell">
$ tofu init

Initializing the backend...
Initializing modules...
- rhyne_wyse_approle in modules/app_credentials

Initializing provider plugins...
- Finding hashicorp/vault versions matching "5.3.0"...
╷
│ Error: Incompatible provider version
│
│ Provider registry.opentofu.org/hashicorp/vault v5.3.0 does not have a package available for your current platform, freebsd_amd64.
│
│ Provider releases are separate from OpenTofu CLI releases, so not all providers are available for all platforms. Other versions of this provider may have different platforms supported.
</syntaxhighlight>

On OpenTofu, some modules are only compiled for Linux, not for FreeBSD.
Switch to Terraform pending a solution to help the OpenTofu builds.

[[Category:Operations grimoire]]
[[Category:Terraform]]

Operations grimoire/Deploy with Terraform

2026-03-29T14:29:49Z

Yousra: /* CARP OVH secrets usage */

== Where to work? ==
We deploy from [[Complector]] using <code>/opt/salt/nasqueron-operations</code> as our local copy of rOPS, authoritative for both Salt and Terraform

You need to belong to the <code>ops</code> group to be able to have access and have write-rights on the repository

It's important to work from there to save a shared Terraform state.

== Specific deployment notes ==
=== Vault / OpenBao ===
==== General notes ====
;OpenTofu support
As of 2026-02-07, the Vault provider isn't compiled for FreeBSD. You need to use Terraform instead.

;Vault
You need a Vault token to allow the provider to connect.

You also need to set VAULT_ADDR to https://172.27.27.7:8200 as Vault doesn't listen on 127.0.0.1

<syntaxhighlight lang="shell">
$ export VAULT_ADDR=https://172.27.27.7:8200
$ sudo /opt/salt/nasqueron-operations/utils/vault/issue-admin-token.py > ~/.vault-token

$ cd /opt/salt/nasqueron-operations/terraform/openbao
$ terraform init # if you've a new entry requiring a module, it needs to be installed
$ terraform plan
$ terraform apply
</syntaxhighlight>

It could be interesting to maintain an alternative policy to admin for the Terraform provider, restricted to the current actions. Note benefits would be limited as those include policies management.

==== Propagate secrets (DRP) ====
;No automatic secret rotation
Secrets rotation is disabled with a lifecycle management <code>ignore_changes = [ secret_id, ]</code>.

To rotate a secret, it needs first to be destroyed from terraform state:
<code>terraform destroy -target=module.viperserv_approle.vault_approle_auth_backend_role_secret_id.this</code>

;Full procedure
Once the AppRole have been created in Vault, they need to be provisioned to the relevant configuration files.

<syntaxhighlight lang="shell">
$ cd /opt/salt/nasqueron-operations/terraform/openbao
$ terraform init # if you've a new entry requiring a module, it needs to be installed
$ terraform plan
$ terraform apply

$ cd /opt/salt/nasqueron-operations
$ salt windriver state.sls_id /usr/local/etc/secrets/rhyne-wyse.yaml roles/reports/rhyne-wyse/config
$ salt windriver state.sls_id /srv/viperserv/.credentials roles/viperserv/eggdrop/config
# Missing for router
</syntaxhighlight>

<div class="alert">Important. Each time you'll reprovision the secrets, they will change.

Don't forget to always apply this full procedure.
</div>

=== CARP OVH secrets usage ===

The OVH credentials used by the CARP failover script are stored in Vault at:

'''apps/network/carp-hyper-001-switch'''

These credentials are accessed by router-002 and router-003. They are then used by a script triggered by devd to update the OVH failover IP MAC address depending on the CARP state (MASTER/BACKUP). The script authenticates to Vault using AppRole (via secretsmith) to retrieve them.

The AppRole credentials (role_id and secret_id) used by the script are stored in:

'''ops/secrets/network/router/vault'''

Secrets are provisioned using Terraform (openbao module), and must be propagated to the routers using Salt.

When rotating the secrets:

* Terraform must be applied again to update the secrets in Vault

Failing to do so may cause CARP failover to break due to invalid OVH credentials or expired Vault authentication.

== Table of Terraform states ==

{| class="wikitable"
|+ Terraform and OpenBao states
|-
! Configuration !! State back-end !! Path !! Software to use
|-
| openbao || On disk || /opt/salt/nasqueron-operations/terraform/openbao/terraform.tfstate || Terraform
|}

On disk paths are stored in Complector.

== Troubleshoot ==
=== Error: Module not installed ===

You need to run <code>tofu init</code> to prepare for any new provider.

<syntaxhighlight lang="shell">
$ tofu plan

│ Error: Module not installed
│
│ on rhyne_wyse.tf line 23:
│ 23: module "rhyne_wyse_approle" {
│
│ This module is not yet installed. Run "tofu init" to install all modules required by this configuration.
</syntaxhighlight>

=== Error: Incompatible provider version ===

<syntaxhighlight lang="shell">
$ tofu init

Initializing the backend...
Initializing modules...
- rhyne_wyse_approle in modules/app_credentials

Initializing provider plugins...
- Finding hashicorp/vault versions matching "5.3.0"...
╷
│ Error: Incompatible provider version
│
│ Provider registry.opentofu.org/hashicorp/vault v5.3.0 does not have a package available for your current platform, freebsd_amd64.
│
│ Provider releases are separate from OpenTofu CLI releases, so not all providers are available for all platforms. Other versions of this provider may have different platforms supported.
</syntaxhighlight>

On OpenTofu, some modules are only compiled for Linux, not for FreeBSD.
Switch to Terraform pending a solution to help the OpenTofu builds.

[[Category:Operations grimoire]]
[[Category:Terraform]]

Operations grimoire/Deploy with Terraform

2026-03-29T14:24:27Z

Yousra: /* CARP OVH secrets usage */

== Where to work? ==
We deploy from [[Complector]] using <code>/opt/salt/nasqueron-operations</code> as our local copy of rOPS, authoritative for both Salt and Terraform

You need to belong to the <code>ops</code> group to be able to have access and have write-rights on the repository

It's important to work from there to save a shared Terraform state.

== Specific deployment notes ==
=== Vault / OpenBao ===
==== General notes ====
;OpenTofu support
As of 2026-02-07, the Vault provider isn't compiled for FreeBSD. You need to use Terraform instead.

;Vault
You need a Vault token to allow the provider to connect.

You also need to set VAULT_ADDR to https://172.27.27.7:8200 as Vault doesn't listen on 127.0.0.1

<syntaxhighlight lang="shell">
$ export VAULT_ADDR=https://172.27.27.7:8200
$ sudo /opt/salt/nasqueron-operations/utils/vault/issue-admin-token.py > ~/.vault-token

$ cd /opt/salt/nasqueron-operations/terraform/openbao
$ terraform init # if you've a new entry requiring a module, it needs to be installed
$ terraform plan
$ terraform apply
</syntaxhighlight>

It could be interesting to maintain an alternative policy to admin for the Terraform provider, restricted to the current actions. Note benefits would be limited as those include policies management.

==== Propagate secrets (DRP) ====
;No automatic secret rotation
Secrets rotation is disabled with a lifecycle management <code>ignore_changes = [ secret_id, ]</code>.

To rotate a secret, it needs first to be destroyed from terraform state:
<code>terraform destroy -target=module.viperserv_approle.vault_approle_auth_backend_role_secret_id.this</code>

;Full procedure
Once the AppRole have been created in Vault, they need to be provisioned to the relevant configuration files.

<syntaxhighlight lang="shell">
$ cd /opt/salt/nasqueron-operations/terraform/openbao
$ terraform init # if you've a new entry requiring a module, it needs to be installed
$ terraform plan
$ terraform apply

$ cd /opt/salt/nasqueron-operations
$ salt windriver state.sls_id /usr/local/etc/secrets/rhyne-wyse.yaml roles/reports/rhyne-wyse/config
$ salt windriver state.sls_id /srv/viperserv/.credentials roles/viperserv/eggdrop/config
# Missing for router
</syntaxhighlight>

<div class="alert">Important. Each time you'll reprovision the secrets, they will change.

Don't forget to always apply this full procedure.
</div>

=== CARP OVH secrets usage ===

The OVH credentials used by the CARP failover script are stored in Vault at:

'''apps/network/carp-hyper-001-switch'''

These credentials are accessed by router-002 and router-003. They are then used by a script triggered by devd to update the OVH failover IP MAC address depending on the CARP state (MASTER/BACKUP). The script authenticates to Vault using AppRole (via secretsmith) to retrieve them.

The AppRole credentials (role_id and secret_id) used by the script are stored in:

'''ops/secrets/network/router/vault'''

These are provisioned via Terraform and deployed to the routers using Salt.

Secrets are provisioned using Terraform (openbao module), and must be propagated to the routers using Salt.

When rotating these secrets:

* Terraform must be applied again to update the secrets in Vault
* Salt states must be re-applied to propagate updated AppRole credentials to the routers

Failing to do so may cause CARP failover to break due to invalid OVH credentials or expired Vault authentication.

== Table of Terraform states ==

{| class="wikitable"
|+ Terraform and OpenBao states
|-
! Configuration !! State back-end !! Path !! Software to use
|-
| openbao || On disk || /opt/salt/nasqueron-operations/terraform/openbao/terraform.tfstate || Terraform
|}

On disk paths are stored in Complector.

== Troubleshoot ==
=== Error: Module not installed ===

You need to run <code>tofu init</code> to prepare for any new provider.

<syntaxhighlight lang="shell">
$ tofu plan

│ Error: Module not installed
│
│ on rhyne_wyse.tf line 23:
│ 23: module "rhyne_wyse_approle" {
│
│ This module is not yet installed. Run "tofu init" to install all modules required by this configuration.
</syntaxhighlight>

=== Error: Incompatible provider version ===

<syntaxhighlight lang="shell">
$ tofu init

Initializing the backend...
Initializing modules...
- rhyne_wyse_approle in modules/app_credentials

Initializing provider plugins...
- Finding hashicorp/vault versions matching "5.3.0"...
╷
│ Error: Incompatible provider version
│
│ Provider registry.opentofu.org/hashicorp/vault v5.3.0 does not have a package available for your current platform, freebsd_amd64.
│
│ Provider releases are separate from OpenTofu CLI releases, so not all providers are available for all platforms. Other versions of this provider may have different platforms supported.
</syntaxhighlight>

On OpenTofu, some modules are only compiled for Linux, not for FreeBSD.
Switch to Terraform pending a solution to help the OpenTofu builds.

[[Category:Operations grimoire]]
[[Category:Terraform]]

Operations grimoire/Deploy with Terraform

2026-03-29T14:22:45Z

Yousra: /* CARP OVH secrets usage */

== Where to work? ==
We deploy from [[Complector]] using <code>/opt/salt/nasqueron-operations</code> as our local copy of rOPS, authoritative for both Salt and Terraform

You need to belong to the <code>ops</code> group to be able to have access and have write-rights on the repository

It's important to work from there to save a shared Terraform state.

== Specific deployment notes ==
=== Vault / OpenBao ===
==== General notes ====
;OpenTofu support
As of 2026-02-07, the Vault provider isn't compiled for FreeBSD. You need to use Terraform instead.

;Vault
You need a Vault token to allow the provider to connect.

You also need to set VAULT_ADDR to https://172.27.27.7:8200 as Vault doesn't listen on 127.0.0.1

<syntaxhighlight lang="shell">
$ export VAULT_ADDR=https://172.27.27.7:8200
$ sudo /opt/salt/nasqueron-operations/utils/vault/issue-admin-token.py > ~/.vault-token

$ cd /opt/salt/nasqueron-operations/terraform/openbao
$ terraform init # if you've a new entry requiring a module, it needs to be installed
$ terraform plan
$ terraform apply
</syntaxhighlight>

It could be interesting to maintain an alternative policy to admin for the Terraform provider, restricted to the current actions. Note benefits would be limited as those include policies management.

==== Propagate secrets (DRP) ====
;No automatic secret rotation
Secrets rotation is disabled with a lifecycle management <code>ignore_changes = [ secret_id, ]</code>.

To rotate a secret, it needs first to be destroyed from terraform state:
<code>terraform destroy -target=module.viperserv_approle.vault_approle_auth_backend_role_secret_id.this</code>

;Full procedure
Once the AppRole have been created in Vault, they need to be provisioned to the relevant configuration files.

<syntaxhighlight lang="shell">
$ cd /opt/salt/nasqueron-operations/terraform/openbao
$ terraform init # if you've a new entry requiring a module, it needs to be installed
$ terraform plan
$ terraform apply

$ cd /opt/salt/nasqueron-operations
$ salt windriver state.sls_id /usr/local/etc/secrets/rhyne-wyse.yaml roles/reports/rhyne-wyse/config
$ salt windriver state.sls_id /srv/viperserv/.credentials roles/viperserv/eggdrop/config
# Missing for router
</syntaxhighlight>

<div class="alert">Important. Each time you'll reprovision the secrets, they will change.

Don't forget to always apply this full procedure.
</div>

=== CARP OVH secrets usage ===

The OVH credentials used by the CARP failover script are stored in Vault at:

'''apps/network/carp-hyper-001-switch'''

These credentials are accessed by router-002 and router-003.
The script authenticates to Vault using AppRole (via secretsmith) to retrieve them.

They are then used by a script triggered by devd to update the OVH failover IP MAC address depending on the CARP state (MASTER/BACKUP).

The AppRole credentials (role_id and secret_id) used by the routers are stored in:

'''ops/secrets/network/router/vault'''

These are provisioned via Terraform and deployed to the routers using Salt.

Secrets are provisioned using Terraform (openbao module), and must be propagated to the routers using Salt.

When rotating these secrets:

* Terraform must be applied again to update the secrets in Vault
* Salt states must be re-applied to propagate updated AppRole credentials to the routers

Failing to do so may cause CARP failover to break due to invalid OVH credentials or expired Vault authentication.

== Table of Terraform states ==

{| class="wikitable"
|+ Terraform and OpenBao states
|-
! Configuration !! State back-end !! Path !! Software to use
|-
| openbao || On disk || /opt/salt/nasqueron-operations/terraform/openbao/terraform.tfstate || Terraform
|}

On disk paths are stored in Complector.

== Troubleshoot ==
=== Error: Module not installed ===

You need to run <code>tofu init</code> to prepare for any new provider.

<syntaxhighlight lang="shell">
$ tofu plan

│ Error: Module not installed
│
│ on rhyne_wyse.tf line 23:
│ 23: module "rhyne_wyse_approle" {
│
│ This module is not yet installed. Run "tofu init" to install all modules required by this configuration.
</syntaxhighlight>

=== Error: Incompatible provider version ===

<syntaxhighlight lang="shell">
$ tofu init

Initializing the backend...
Initializing modules...
- rhyne_wyse_approle in modules/app_credentials

Initializing provider plugins...
- Finding hashicorp/vault versions matching "5.3.0"...
╷
│ Error: Incompatible provider version
│
│ Provider registry.opentofu.org/hashicorp/vault v5.3.0 does not have a package available for your current platform, freebsd_amd64.
│
│ Provider releases are separate from OpenTofu CLI releases, so not all providers are available for all platforms. Other versions of this provider may have different platforms supported.
</syntaxhighlight>

On OpenTofu, some modules are only compiled for Linux, not for FreeBSD.
Switch to Terraform pending a solution to help the OpenTofu builds.

[[Category:Operations grimoire]]
[[Category:Terraform]]

Operations grimoire/Deploy with Terraform

2026-03-29T14:21:13Z

Yousra: /* CARP OVH secrets usage */

== Where to work? ==
We deploy from [[Complector]] using <code>/opt/salt/nasqueron-operations</code> as our local copy of rOPS, authoritative for both Salt and Terraform

You need to belong to the <code>ops</code> group to be able to have access and have write-rights on the repository

It's important to work from there to save a shared Terraform state.

== Specific deployment notes ==
=== Vault / OpenBao ===
==== General notes ====
;OpenTofu support
As of 2026-02-07, the Vault provider isn't compiled for FreeBSD. You need to use Terraform instead.

;Vault
You need a Vault token to allow the provider to connect.

You also need to set VAULT_ADDR to https://172.27.27.7:8200 as Vault doesn't listen on 127.0.0.1

<syntaxhighlight lang="shell">
$ export VAULT_ADDR=https://172.27.27.7:8200
$ sudo /opt/salt/nasqueron-operations/utils/vault/issue-admin-token.py > ~/.vault-token

$ cd /opt/salt/nasqueron-operations/terraform/openbao
$ terraform init # if you've a new entry requiring a module, it needs to be installed
$ terraform plan
$ terraform apply
</syntaxhighlight>

It could be interesting to maintain an alternative policy to admin for the Terraform provider, restricted to the current actions. Note benefits would be limited as those include policies management.

==== Propagate secrets (DRP) ====
;No automatic secret rotation
Secrets rotation is disabled with a lifecycle management <code>ignore_changes = [ secret_id, ]</code>.

To rotate a secret, it needs first to be destroyed from terraform state:
<code>terraform destroy -target=module.viperserv_approle.vault_approle_auth_backend_role_secret_id.this</code>

;Full procedure
Once the AppRole have been created in Vault, they need to be provisioned to the relevant configuration files.

<syntaxhighlight lang="shell">
$ cd /opt/salt/nasqueron-operations/terraform/openbao
$ terraform init # if you've a new entry requiring a module, it needs to be installed
$ terraform plan
$ terraform apply

$ cd /opt/salt/nasqueron-operations
$ salt windriver state.sls_id /usr/local/etc/secrets/rhyne-wyse.yaml roles/reports/rhyne-wyse/config
$ salt windriver state.sls_id /srv/viperserv/.credentials roles/viperserv/eggdrop/config
# Missing for router
</syntaxhighlight>

<div class="alert">Important. Each time you'll reprovision the secrets, they will change.

Don't forget to always apply this full procedure.
</div>

=== CARP OVH secrets usage ===

The OVH credentials used by the CARP failover script are stored in Vault at:

'''apps/network/carp-hyper-001-switch'''

These credentials are accessed by router-002 and router-003 using AppRole authentication (via secretsmith).
They are used by a script triggered by devd to update the OVH failover IP MAC address depending on the CARP state (MASTER/BACKUP).

The AppRole credentials (role_id and secret_id) used by the routers are stored in:

'''ops/secrets/network/router/vault'''

These are provisioned via Terraform and deployed to the routers using Salt.

Secrets are provisioned using Terraform (openbao module), and must be propagated to the routers using Salt.

When rotating these secrets:

* Terraform must be applied again to update the secrets in Vault
* Salt states must be re-applied to propagate updated AppRole credentials to the routers

Failing to do so may cause CARP failover to break due to invalid OVH credentials or expired Vault authentication.

== Table of Terraform states ==

{| class="wikitable"
|+ Terraform and OpenBao states
|-
! Configuration !! State back-end !! Path !! Software to use
|-
| openbao || On disk || /opt/salt/nasqueron-operations/terraform/openbao/terraform.tfstate || Terraform
|}

On disk paths are stored in Complector.

== Troubleshoot ==
=== Error: Module not installed ===

You need to run <code>tofu init</code> to prepare for any new provider.

<syntaxhighlight lang="shell">
$ tofu plan

│ Error: Module not installed
│
│ on rhyne_wyse.tf line 23:
│ 23: module "rhyne_wyse_approle" {
│
│ This module is not yet installed. Run "tofu init" to install all modules required by this configuration.
</syntaxhighlight>

=== Error: Incompatible provider version ===

<syntaxhighlight lang="shell">
$ tofu init

Initializing the backend...
Initializing modules...
- rhyne_wyse_approle in modules/app_credentials

Initializing provider plugins...
- Finding hashicorp/vault versions matching "5.3.0"...
╷
│ Error: Incompatible provider version
│
│ Provider registry.opentofu.org/hashicorp/vault v5.3.0 does not have a package available for your current platform, freebsd_amd64.
│
│ Provider releases are separate from OpenTofu CLI releases, so not all providers are available for all platforms. Other versions of this provider may have different platforms supported.
</syntaxhighlight>

On OpenTofu, some modules are only compiled for Linux, not for FreeBSD.
Switch to Terraform pending a solution to help the OpenTofu builds.

[[Category:Operations grimoire]]
[[Category:Terraform]]

Operations grimoire/Deploy with Terraform

2026-03-29T14:20:50Z

Yousra: /* CARP OVH secrets usage */

== Where to work? ==
We deploy from [[Complector]] using <code>/opt/salt/nasqueron-operations</code> as our local copy of rOPS, authoritative for both Salt and Terraform

You need to belong to the <code>ops</code> group to be able to have access and have write-rights on the repository

It's important to work from there to save a shared Terraform state.

== Specific deployment notes ==
=== Vault / OpenBao ===
==== General notes ====
;OpenTofu support
As of 2026-02-07, the Vault provider isn't compiled for FreeBSD. You need to use Terraform instead.

;Vault
You need a Vault token to allow the provider to connect.

You also need to set VAULT_ADDR to https://172.27.27.7:8200 as Vault doesn't listen on 127.0.0.1

<syntaxhighlight lang="shell">
$ export VAULT_ADDR=https://172.27.27.7:8200
$ sudo /opt/salt/nasqueron-operations/utils/vault/issue-admin-token.py > ~/.vault-token

$ cd /opt/salt/nasqueron-operations/terraform/openbao
$ terraform init # if you've a new entry requiring a module, it needs to be installed
$ terraform plan
$ terraform apply
</syntaxhighlight>

It could be interesting to maintain an alternative policy to admin for the Terraform provider, restricted to the current actions. Note benefits would be limited as those include policies management.

==== Propagate secrets (DRP) ====
;No automatic secret rotation
Secrets rotation is disabled with a lifecycle management <code>ignore_changes = [ secret_id, ]</code>.

To rotate a secret, it needs first to be destroyed from terraform state:
<code>terraform destroy -target=module.viperserv_approle.vault_approle_auth_backend_role_secret_id.this</code>

;Full procedure
Once the AppRole have been created in Vault, they need to be provisioned to the relevant configuration files.

<syntaxhighlight lang="shell">
$ cd /opt/salt/nasqueron-operations/terraform/openbao
$ terraform init # if you've a new entry requiring a module, it needs to be installed
$ terraform plan
$ terraform apply

$ cd /opt/salt/nasqueron-operations
$ salt windriver state.sls_id /usr/local/etc/secrets/rhyne-wyse.yaml roles/reports/rhyne-wyse/config
$ salt windriver state.sls_id /srv/viperserv/.credentials roles/viperserv/eggdrop/config
# Missing for router
</syntaxhighlight>

<div class="alert">Important. Each time you'll reprovision the secrets, they will change.

Don't forget to always apply this full procedure.
</div>

=== CARP OVH secrets usage ===

The OVH credentials used by the CARP failover script are stored in Vault at:

apps/network/carp-hyper-001-switch

These credentials are accessed by router-002 and router-003 using AppRole authentication (via secretsmith).
They are used by a script triggered by devd to update the OVH failover IP MAC address depending on the CARP state (MASTER/BACKUP).

The AppRole credentials (role_id and secret_id) used by the routers are stored in:

ops/secrets/network/router/vault

These are provisioned via Terraform and deployed to the routers using Salt.

Secrets are provisioned using Terraform (openbao module), and must be propagated to the routers using Salt.

When rotating these secrets:

* Terraform must be applied again to update the secrets in Vault
* Salt states must be re-applied to propagate updated AppRole credentials to the routers

Failing to do so may cause CARP failover to break due to invalid OVH credentials or expired Vault authentication.

== Table of Terraform states ==

{| class="wikitable"
|+ Terraform and OpenBao states
|-
! Configuration !! State back-end !! Path !! Software to use
|-
| openbao || On disk || /opt/salt/nasqueron-operations/terraform/openbao/terraform.tfstate || Terraform
|}

On disk paths are stored in Complector.

== Troubleshoot ==
=== Error: Module not installed ===

You need to run <code>tofu init</code> to prepare for any new provider.

<syntaxhighlight lang="shell">
$ tofu plan

│ Error: Module not installed
│
│ on rhyne_wyse.tf line 23:
│ 23: module "rhyne_wyse_approle" {
│
│ This module is not yet installed. Run "tofu init" to install all modules required by this configuration.
</syntaxhighlight>

=== Error: Incompatible provider version ===

<syntaxhighlight lang="shell">
$ tofu init

Initializing the backend...
Initializing modules...
- rhyne_wyse_approle in modules/app_credentials

Initializing provider plugins...
- Finding hashicorp/vault versions matching "5.3.0"...
╷
│ Error: Incompatible provider version
│
│ Provider registry.opentofu.org/hashicorp/vault v5.3.0 does not have a package available for your current platform, freebsd_amd64.
│
│ Provider releases are separate from OpenTofu CLI releases, so not all providers are available for all platforms. Other versions of this provider may have different platforms supported.
</syntaxhighlight>

On OpenTofu, some modules are only compiled for Linux, not for FreeBSD.
Switch to Terraform pending a solution to help the OpenTofu builds.

[[Category:Operations grimoire]]
[[Category:Terraform]]

Operations grimoire/Deploy with Terraform

2026-03-29T14:19:11Z

Yousra: /* CARP OVH secrets usage */

== Where to work? ==
We deploy from [[Complector]] using <code>/opt/salt/nasqueron-operations</code> as our local copy of rOPS, authoritative for both Salt and Terraform

You need to belong to the <code>ops</code> group to be able to have access and have write-rights on the repository

It's important to work from there to save a shared Terraform state.

== Specific deployment notes ==
=== Vault / OpenBao ===
==== General notes ====
;OpenTofu support
As of 2026-02-07, the Vault provider isn't compiled for FreeBSD. You need to use Terraform instead.

;Vault
You need a Vault token to allow the provider to connect.

You also need to set VAULT_ADDR to https://172.27.27.7:8200 as Vault doesn't listen on 127.0.0.1

<syntaxhighlight lang="shell">
$ export VAULT_ADDR=https://172.27.27.7:8200
$ sudo /opt/salt/nasqueron-operations/utils/vault/issue-admin-token.py > ~/.vault-token

$ cd /opt/salt/nasqueron-operations/terraform/openbao
$ terraform init # if you've a new entry requiring a module, it needs to be installed
$ terraform plan
$ terraform apply
</syntaxhighlight>

It could be interesting to maintain an alternative policy to admin for the Terraform provider, restricted to the current actions. Note benefits would be limited as those include policies management.

==== Propagate secrets (DRP) ====
;No automatic secret rotation
Secrets rotation is disabled with a lifecycle management <code>ignore_changes = [ secret_id, ]</code>.

To rotate a secret, it needs first to be destroyed from terraform state:
<code>terraform destroy -target=module.viperserv_approle.vault_approle_auth_backend_role_secret_id.this</code>

;Full procedure
Once the AppRole have been created in Vault, they need to be provisioned to the relevant configuration files.

<syntaxhighlight lang="shell">
$ cd /opt/salt/nasqueron-operations/terraform/openbao
$ terraform init # if you've a new entry requiring a module, it needs to be installed
$ terraform plan
$ terraform apply

$ cd /opt/salt/nasqueron-operations
$ salt windriver state.sls_id /usr/local/etc/secrets/rhyne-wyse.yaml roles/reports/rhyne-wyse/config
$ salt windriver state.sls_id /srv/viperserv/.credentials roles/viperserv/eggdrop/config
# Missing for router
</syntaxhighlight>

<div class="alert">Important. Each time you'll reprovision the secrets, they will change.

Don't forget to always apply this full procedure.
</div>

=== CARP OVH secrets usage ===

The OVH credentials used by the CARP failover script are stored in Vault at:

'''apps/network/carp-hyper-001-switch'''

These credentials are accessed by router-002 and router-003 and are used by a script triggered by devd to update the OVH failover IP MAC address depending on the CARP state (MASTER/BACKUP, that is using AppRole authentication (via secretsmith),

Secrets are provisioned using Terraform (openbao module), and must be propagated to the routers using Salt.

When rotating these secrets:
* Terraform must be applied again

Failing to do so may cause CARP failover to break due to invalid OVH credentials.

== Table of Terraform states ==

{| class="wikitable"
|+ Terraform and OpenBao states
|-
! Configuration !! State back-end !! Path !! Software to use
|-
| openbao || On disk || /opt/salt/nasqueron-operations/terraform/openbao/terraform.tfstate || Terraform
|}

On disk paths are stored in Complector.

== Troubleshoot ==
=== Error: Module not installed ===

You need to run <code>tofu init</code> to prepare for any new provider.

<syntaxhighlight lang="shell">
$ tofu plan

│ Error: Module not installed
│
│ on rhyne_wyse.tf line 23:
│ 23: module "rhyne_wyse_approle" {
│
│ This module is not yet installed. Run "tofu init" to install all modules required by this configuration.
</syntaxhighlight>

=== Error: Incompatible provider version ===

<syntaxhighlight lang="shell">
$ tofu init

Initializing the backend...
Initializing modules...
- rhyne_wyse_approle in modules/app_credentials

Initializing provider plugins...
- Finding hashicorp/vault versions matching "5.3.0"...
╷
│ Error: Incompatible provider version
│
│ Provider registry.opentofu.org/hashicorp/vault v5.3.0 does not have a package available for your current platform, freebsd_amd64.
│
│ Provider releases are separate from OpenTofu CLI releases, so not all providers are available for all platforms. Other versions of this provider may have different platforms supported.
</syntaxhighlight>

On OpenTofu, some modules are only compiled for Linux, not for FreeBSD.
Switch to Terraform pending a solution to help the OpenTofu builds.

[[Category:Operations grimoire]]
[[Category:Terraform]]

Operations grimoire/Deploy with Terraform

2026-03-29T14:17:06Z

Yousra: /* CARP OVH secrets usage */

== Where to work? ==
We deploy from [[Complector]] using <code>/opt/salt/nasqueron-operations</code> as our local copy of rOPS, authoritative for both Salt and Terraform

You need to belong to the <code>ops</code> group to be able to have access and have write-rights on the repository

It's important to work from there to save a shared Terraform state.

== Specific deployment notes ==
=== Vault / OpenBao ===
==== General notes ====
;OpenTofu support
As of 2026-02-07, the Vault provider isn't compiled for FreeBSD. You need to use Terraform instead.

;Vault
You need a Vault token to allow the provider to connect.

You also need to set VAULT_ADDR to https://172.27.27.7:8200 as Vault doesn't listen on 127.0.0.1

<syntaxhighlight lang="shell">
$ export VAULT_ADDR=https://172.27.27.7:8200
$ sudo /opt/salt/nasqueron-operations/utils/vault/issue-admin-token.py > ~/.vault-token

$ cd /opt/salt/nasqueron-operations/terraform/openbao
$ terraform init # if you've a new entry requiring a module, it needs to be installed
$ terraform plan
$ terraform apply
</syntaxhighlight>

It could be interesting to maintain an alternative policy to admin for the Terraform provider, restricted to the current actions. Note benefits would be limited as those include policies management.

==== Propagate secrets (DRP) ====
;No automatic secret rotation
Secrets rotation is disabled with a lifecycle management <code>ignore_changes = [ secret_id, ]</code>.

To rotate a secret, it needs first to be destroyed from terraform state:
<code>terraform destroy -target=module.viperserv_approle.vault_approle_auth_backend_role_secret_id.this</code>

;Full procedure
Once the AppRole have been created in Vault, they need to be provisioned to the relevant configuration files.

<syntaxhighlight lang="shell">
$ cd /opt/salt/nasqueron-operations/terraform/openbao
$ terraform init # if you've a new entry requiring a module, it needs to be installed
$ terraform plan
$ terraform apply

$ cd /opt/salt/nasqueron-operations
$ salt windriver state.sls_id /usr/local/etc/secrets/rhyne-wyse.yaml roles/reports/rhyne-wyse/config
$ salt windriver state.sls_id /srv/viperserv/.credentials roles/viperserv/eggdrop/config
# Missing for router
</syntaxhighlight>

<div class="alert">Important. Each time you'll reprovision the secrets, they will change.

Don't forget to always apply this full procedure.
</div>

=== CARP OVH secrets usage ===

The OVH credentials used by the CARP failover script are stored in Vault at:

'''apps/network/carp-hyper-001-switch'''

These credentials are accessed by router-002 and router-003 and are used by a script triggered by devd to update the OVH failover IP MAC address depending on the CARP state (MASTER/BACKUP, that is using AppRole authentication (via secretsmith),

Secrets are provisioned using Terraform (openbao module), and must be propagated to the routers using Salt.

When rotating these secrets:
* Terraform must be applied again
* Salt states must be re-applied to update the configuration on routers

Failing to do so may cause CARP failover to break due to invalid OVH credentials.

== Table of Terraform states ==

{| class="wikitable"
|+ Terraform and OpenBao states
|-
! Configuration !! State back-end !! Path !! Software to use
|-
| openbao || On disk || /opt/salt/nasqueron-operations/terraform/openbao/terraform.tfstate || Terraform
|}

On disk paths are stored in Complector.

== Troubleshoot ==
=== Error: Module not installed ===

You need to run <code>tofu init</code> to prepare for any new provider.

<syntaxhighlight lang="shell">
$ tofu plan

│ Error: Module not installed
│
│ on rhyne_wyse.tf line 23:
│ 23: module "rhyne_wyse_approle" {
│
│ This module is not yet installed. Run "tofu init" to install all modules required by this configuration.
</syntaxhighlight>

=== Error: Incompatible provider version ===

<syntaxhighlight lang="shell">
$ tofu init

Initializing the backend...
Initializing modules...
- rhyne_wyse_approle in modules/app_credentials

Initializing provider plugins...
- Finding hashicorp/vault versions matching "5.3.0"...
╷
│ Error: Incompatible provider version
│
│ Provider registry.opentofu.org/hashicorp/vault v5.3.0 does not have a package available for your current platform, freebsd_amd64.
│
│ Provider releases are separate from OpenTofu CLI releases, so not all providers are available for all platforms. Other versions of this provider may have different platforms supported.
</syntaxhighlight>

On OpenTofu, some modules are only compiled for Linux, not for FreeBSD.
Switch to Terraform pending a solution to help the OpenTofu builds.

[[Category:Operations grimoire]]
[[Category:Terraform]]

Operations grimoire/Deploy with Terraform

2026-03-29T14:09:45Z

Yousra: /* CARP OVH secrets usage */

== Where to work? ==
We deploy from [[Complector]] using <code>/opt/salt/nasqueron-operations</code> as our local copy of rOPS, authoritative for both Salt and Terraform

You need to belong to the <code>ops</code> group to be able to have access and have write-rights on the repository

It's important to work from there to save a shared Terraform state.

== Specific deployment notes ==
=== Vault / OpenBao ===
==== General notes ====
;OpenTofu support
As of 2026-02-07, the Vault provider isn't compiled for FreeBSD. You need to use Terraform instead.

;Vault
You need a Vault token to allow the provider to connect.

You also need to set VAULT_ADDR to https://172.27.27.7:8200 as Vault doesn't listen on 127.0.0.1

<syntaxhighlight lang="shell">
$ export VAULT_ADDR=https://172.27.27.7:8200
$ sudo /opt/salt/nasqueron-operations/utils/vault/issue-admin-token.py > ~/.vault-token

$ cd /opt/salt/nasqueron-operations/terraform/openbao
$ terraform init # if you've a new entry requiring a module, it needs to be installed
$ terraform plan
$ terraform apply
</syntaxhighlight>

It could be interesting to maintain an alternative policy to admin for the Terraform provider, restricted to the current actions. Note benefits would be limited as those include policies management.

==== Propagate secrets (DRP) ====
;No automatic secret rotation
Secrets rotation is disabled with a lifecycle management <code>ignore_changes = [ secret_id, ]</code>.

To rotate a secret, it needs first to be destroyed from terraform state:
<code>terraform destroy -target=module.viperserv_approle.vault_approle_auth_backend_role_secret_id.this</code>

;Full procedure
Once the AppRole have been created in Vault, they need to be provisioned to the relevant configuration files.

<syntaxhighlight lang="shell">
$ cd /opt/salt/nasqueron-operations/terraform/openbao
$ terraform init # if you've a new entry requiring a module, it needs to be installed
$ terraform plan
$ terraform apply

$ cd /opt/salt/nasqueron-operations
$ salt windriver state.sls_id /usr/local/etc/secrets/rhyne-wyse.yaml roles/reports/rhyne-wyse/config
$ salt windriver state.sls_id /srv/viperserv/.credentials roles/viperserv/eggdrop/config
# Missing for router
</syntaxhighlight>

<div class="alert">Important. Each time you'll reprovision the secrets, they will change.

Don't forget to always apply this full procedure.
</div>

=== CARP OVH secrets usage ===

The OVH credentials used by the CARP failover script are stored in Vault at:

'''apps/network/carp-hyper-001-switch'''

These credentials are accessed by router-002 and router-003 that are using AppRole authentication (via secretsmith), and are used by a script triggered by devd to update the OVH failover IP MAC address depending on the CARP state (MASTER/BACKUP).

Secrets are provisioned using Terraform (openbao module), and must be propagated to the routers using Salt.

When rotating these secrets:
* Terraform must be applied again
* Salt states must be re-applied to update the configuration on routers

Failing to do so may cause CARP failover to break due to invalid OVH credentials.

== Table of Terraform states ==

{| class="wikitable"
|+ Terraform and OpenBao states
|-
! Configuration !! State back-end !! Path !! Software to use
|-
| openbao || On disk || /opt/salt/nasqueron-operations/terraform/openbao/terraform.tfstate || Terraform
|}

On disk paths are stored in Complector.

== Troubleshoot ==
=== Error: Module not installed ===

You need to run <code>tofu init</code> to prepare for any new provider.

<syntaxhighlight lang="shell">
$ tofu plan

│ Error: Module not installed
│
│ on rhyne_wyse.tf line 23:
│ 23: module "rhyne_wyse_approle" {
│
│ This module is not yet installed. Run "tofu init" to install all modules required by this configuration.
</syntaxhighlight>

=== Error: Incompatible provider version ===

<syntaxhighlight lang="shell">
$ tofu init

Initializing the backend...
Initializing modules...
- rhyne_wyse_approle in modules/app_credentials

Initializing provider plugins...
- Finding hashicorp/vault versions matching "5.3.0"...
╷
│ Error: Incompatible provider version
│
│ Provider registry.opentofu.org/hashicorp/vault v5.3.0 does not have a package available for your current platform, freebsd_amd64.
│
│ Provider releases are separate from OpenTofu CLI releases, so not all providers are available for all platforms. Other versions of this provider may have different platforms supported.
</syntaxhighlight>

On OpenTofu, some modules are only compiled for Linux, not for FreeBSD.
Switch to Terraform pending a solution to help the OpenTofu builds.

[[Category:Operations grimoire]]
[[Category:Terraform]]

Operations grimoire/Deploy with Terraform

2026-03-29T14:08:58Z

Yousra: /* CARP OVH secrets usage */

== Where to work? ==
We deploy from [[Complector]] using <code>/opt/salt/nasqueron-operations</code> as our local copy of rOPS, authoritative for both Salt and Terraform

You need to belong to the <code>ops</code> group to be able to have access and have write-rights on the repository

It's important to work from there to save a shared Terraform state.

== Specific deployment notes ==
=== Vault / OpenBao ===
==== General notes ====
;OpenTofu support
As of 2026-02-07, the Vault provider isn't compiled for FreeBSD. You need to use Terraform instead.

;Vault
You need a Vault token to allow the provider to connect.

You also need to set VAULT_ADDR to https://172.27.27.7:8200 as Vault doesn't listen on 127.0.0.1

<syntaxhighlight lang="shell">
$ export VAULT_ADDR=https://172.27.27.7:8200
$ sudo /opt/salt/nasqueron-operations/utils/vault/issue-admin-token.py > ~/.vault-token

$ cd /opt/salt/nasqueron-operations/terraform/openbao
$ terraform init # if you've a new entry requiring a module, it needs to be installed
$ terraform plan
$ terraform apply
</syntaxhighlight>

It could be interesting to maintain an alternative policy to admin for the Terraform provider, restricted to the current actions. Note benefits would be limited as those include policies management.

==== Propagate secrets (DRP) ====
;No automatic secret rotation
Secrets rotation is disabled with a lifecycle management <code>ignore_changes = [ secret_id, ]</code>.

To rotate a secret, it needs first to be destroyed from terraform state:
<code>terraform destroy -target=module.viperserv_approle.vault_approle_auth_backend_role_secret_id.this</code>

;Full procedure
Once the AppRole have been created in Vault, they need to be provisioned to the relevant configuration files.

<syntaxhighlight lang="shell">
$ cd /opt/salt/nasqueron-operations/terraform/openbao
$ terraform init # if you've a new entry requiring a module, it needs to be installed
$ terraform plan
$ terraform apply

$ cd /opt/salt/nasqueron-operations
$ salt windriver state.sls_id /usr/local/etc/secrets/rhyne-wyse.yaml roles/reports/rhyne-wyse/config
$ salt windriver state.sls_id /srv/viperserv/.credentials roles/viperserv/eggdrop/config
# Missing for router
</syntaxhighlight>

<div class="alert">Important. Each time you'll reprovision the secrets, they will change.

Don't forget to always apply this full procedure.
</div>

=== CARP OVH secrets usage ===

The OVH credentials used by the CARP failover script are stored in Vault at:

'''apps/network/carp-hyper-001-switch'''

These credentials are accessed by routers using AppRole authentication (via secretsmith), and are used by a script triggered by devd to update the OVH failover IP MAC address depending on the CARP state (MASTER/BACKUP).

Secrets are provisioned using Terraform (openbao module), and must be propagated to the routers using Salt.

When rotating these secrets:
* Terraform must be applied again
* Salt states must be re-applied to update the configuration on routers

Failing to do so may cause CARP failover to break due to invalid OVH credentials.

== Table of Terraform states ==

{| class="wikitable"
|+ Terraform and OpenBao states
|-
! Configuration !! State back-end !! Path !! Software to use
|-
| openbao || On disk || /opt/salt/nasqueron-operations/terraform/openbao/terraform.tfstate || Terraform
|}

On disk paths are stored in Complector.

== Troubleshoot ==
=== Error: Module not installed ===

You need to run <code>tofu init</code> to prepare for any new provider.

<syntaxhighlight lang="shell">
$ tofu plan

│ Error: Module not installed
│
│ on rhyne_wyse.tf line 23:
│ 23: module "rhyne_wyse_approle" {
│
│ This module is not yet installed. Run "tofu init" to install all modules required by this configuration.
</syntaxhighlight>

=== Error: Incompatible provider version ===

<syntaxhighlight lang="shell">
$ tofu init

Initializing the backend...
Initializing modules...
- rhyne_wyse_approle in modules/app_credentials

Initializing provider plugins...
- Finding hashicorp/vault versions matching "5.3.0"...
╷
│ Error: Incompatible provider version
│
│ Provider registry.opentofu.org/hashicorp/vault v5.3.0 does not have a package available for your current platform, freebsd_amd64.
│
│ Provider releases are separate from OpenTofu CLI releases, so not all providers are available for all platforms. Other versions of this provider may have different platforms supported.
</syntaxhighlight>

On OpenTofu, some modules are only compiled for Linux, not for FreeBSD.
Switch to Terraform pending a solution to help the OpenTofu builds.

[[Category:Operations grimoire]]
[[Category:Terraform]]

Operations grimoire/Deploy with Terraform

2026-03-29T14:00:31Z

Yousra: /* Propagate secrets (DRP) */

== Where to work? ==
We deploy from [[Complector]] using <code>/opt/salt/nasqueron-operations</code> as our local copy of rOPS, authoritative for both Salt and Terraform

You need to belong to the <code>ops</code> group to be able to have access and have write-rights on the repository

It's important to work from there to save a shared Terraform state.

== Specific deployment notes ==
=== Vault / OpenBao ===
==== General notes ====
;OpenTofu support
As of 2026-02-07, the Vault provider isn't compiled for FreeBSD. You need to use Terraform instead.

;Vault
You need a Vault token to allow the provider to connect.

You also need to set VAULT_ADDR to https://172.27.27.7:8200 as Vault doesn't listen on 127.0.0.1

<syntaxhighlight lang="shell">
$ export VAULT_ADDR=https://172.27.27.7:8200
$ sudo /opt/salt/nasqueron-operations/utils/vault/issue-admin-token.py > ~/.vault-token

$ cd /opt/salt/nasqueron-operations/terraform/openbao
$ terraform init # if you've a new entry requiring a module, it needs to be installed
$ terraform plan
$ terraform apply
</syntaxhighlight>

It could be interesting to maintain an alternative policy to admin for the Terraform provider, restricted to the current actions. Note benefits would be limited as those include policies management.

==== Propagate secrets (DRP) ====
;No automatic secret rotation
Secrets rotation is disabled with a lifecycle management <code>ignore_changes = [ secret_id, ]</code>.

To rotate a secret, it needs first to be destroyed from terraform state:
<code>terraform destroy -target=module.viperserv_approle.vault_approle_auth_backend_role_secret_id.this</code>

;Full procedure
Once the AppRole have been created in Vault, they need to be provisioned to the relevant configuration files.

<syntaxhighlight lang="shell">
$ cd /opt/salt/nasqueron-operations/terraform/openbao
$ terraform init # if you've a new entry requiring a module, it needs to be installed
$ terraform plan
$ terraform apply

$ cd /opt/salt/nasqueron-operations
$ salt windriver state.sls_id /usr/local/etc/secrets/rhyne-wyse.yaml roles/reports/rhyne-wyse/config
$ salt windriver state.sls_id /srv/viperserv/.credentials roles/viperserv/eggdrop/config
# Missing for router
</syntaxhighlight>

<div class="alert">Important. Each time you'll reprovision the secrets, they will change.

Don't forget to always apply this full procedure.
</div>

=== CARP OVH secrets usage ===

The OVH credentials used by the CARP failover script are stored in Vault at:

apps/network/carp-hyper-001-switch

These credentials are accessed by routers using AppRole authentication (via secretsmith), and are used by a script triggered by devd to update the OVH failover IP MAC address depending on the CARP state (MASTER/BACKUP).

Secrets are provisioned using Terraform (openbao module), and must be propagated to the routers using Salt.

When rotating these secrets:
* Terraform must be applied again
* Salt states must be re-applied to update the configuration on routers

Failing to do so may cause CARP failover to break due to invalid OVH credentials.

== Table of Terraform states ==

{| class="wikitable"
|+ Terraform and OpenBao states
|-
! Configuration !! State back-end !! Path !! Software to use
|-
| openbao || On disk || /opt/salt/nasqueron-operations/terraform/openbao/terraform.tfstate || Terraform
|}

On disk paths are stored in Complector.

== Troubleshoot ==
=== Error: Module not installed ===

You need to run <code>tofu init</code> to prepare for any new provider.

<syntaxhighlight lang="shell">
$ tofu plan

│ Error: Module not installed
│
│ on rhyne_wyse.tf line 23:
│ 23: module "rhyne_wyse_approle" {
│
│ This module is not yet installed. Run "tofu init" to install all modules required by this configuration.
</syntaxhighlight>

=== Error: Incompatible provider version ===

<syntaxhighlight lang="shell">
$ tofu init

Initializing the backend...
Initializing modules...
- rhyne_wyse_approle in modules/app_credentials

Initializing provider plugins...
- Finding hashicorp/vault versions matching "5.3.0"...
╷
│ Error: Incompatible provider version
│
│ Provider registry.opentofu.org/hashicorp/vault v5.3.0 does not have a package available for your current platform, freebsd_amd64.
│
│ Provider releases are separate from OpenTofu CLI releases, so not all providers are available for all platforms. Other versions of this provider may have different platforms supported.
</syntaxhighlight>

On OpenTofu, some modules are only compiled for Linux, not for FreeBSD.
Switch to Terraform pending a solution to help the OpenTofu builds.

[[Category:Operations grimoire]]
[[Category:Terraform]]

Protocol CARP

2026-03-22T20:24:31Z

Yousra:

2026-03-09T19:17:11Z

Yousra: /* Virtual Mac Address */

== Introduction and history ==

« CARP stands for Common Address Redundancy Protocol and its basic functionality is to allow multiple hosts to share a set of IP addresses ».

CARP was developed by the OpenBSD project in 2003 as an alternative to VRRP due to patent concerns related to Cisco's HSRP technology. To avoid potential legal issues and remain compatible with open-source principles, OpenBSD implemented its own redundancy protocol with added cryptographic features. CARP was later adopted by other systems such as FreeBSD and NetBSD. A userland implementation of CARP, named ucarp, made the protocol usable on Linux without requiring kernel integration.

These patent concerns originated from the development of VRRP by the Internet Engineering Task Force (IETF). Indeed, when the Internet Engineering Task Force (IETF) developed the Virtual Router Redundancy Protocol (VRRP), Cisco Systems indicated that it owned patents related to router redundancy mechanisms used in the protocol Hot Standby Router Protocol (HSRP). This created a legal risk for developers, particularly for open-source projects that wanted to implement VRRP freely.

Sources :

https://en.wikipedia.org/wiki/Common_Address_Redundancy_Protocol

https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf

== Master host and backup hosts ==

CARP creates a redundancy group, meaning several hosts are configured to share a virtual IP address. However, at any given time, only one host uses the shared IP address. This host is called the Master host and it receives and handles all traffic destined to that virtual IP.

When the Master host becomes unavailable (it crashed, turned off, or lost its network connection), the other hosts in the same redundancy group detect the failure. Immediately, one of the Backup hosts is elected as the new Master host. That means that it will take over the shared IP address.

This switch happens automatically, ensuring service continuity without clients noticing any interruption.

[[File:Schema redondance routeurs.png]]

Sources :

https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf

https://wxcafe.net/posts/redondance-routeurs-openbsd-freebsd/

== CARP advertisements ==

- The Master host periodically broadcasts CARP advertisements to the backup hosts.

- The Backup hosts listen but do not send any packets.

'''Each CARP advertisement contains :'''

1. The CARP version and packet type (CARP Header).

2. The VHID (Virtual Host ID), which identifies the redundancy group (CARP Header).

3. The advertisement parameters (advbase and advskew), which determine the host’s priority CARP (CARP Header).

4. The counter

5. The signature in SHA-1

'''CARP Advertisement Packet Format :'''

Each line represent 32 bits, 4 bytes

Length : 36 bytes

<pre>
/*
* The CARP header layout is as follows:
*
* 0 1 2 3
* 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
* |Version| Type | VirtualHostID | AdvSkew | Auth Len |
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
* | Demotion | AdvBase | Checksum |
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
* | Counter (1) |
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
* | Counter (2) |
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
* | SHA-1 HMAC (1) |
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
* | SHA-1 HMAC (2) |
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
* | SHA-1 HMAC (3) |
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
* | SHA-1 HMAC (4) |
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
* | SHA-1 HMAC (5) |
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
*
*/
</pre>

'''commande : tcpdump -i <interface> proto 112'''

[[ File:Carp-advertissement.png ]]

Sources :

https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf

https://github.com/openbsd/src/blob/master/sys/netinet/ip_carp.h

== Cryptography ==

'''All CARP advertisements are cryptographically signed :'''

Steps :

1. The Master host creates an advertisement.

2. It calculates a cryptographic signature, a SHA-1 hash using:

* the shared secret,

* the VHID,

* the virtual IP address,

* the advertisement parameters (advbase and advskew),

* the counter

* and other header fields.

3. It then sends the signed advertisement.

4. The Backup hosts receives the signed advertisement.

5. Each Backup host recalculates the signature using the same shared secret :

- If the calculated signature matches the received signature → the packet is considered valid because authentication and integrity are insured.

- If it does not match → the packet is rejected.

It protects against spoofing attacks : without cryptographic authentication, an attacker could send fake CARP advertisements and attempt to pretend to be or to become the Master host.

'''command : tcpdump -X -ni <interface> proto 112'''

We can observe the complete packet in the capture. Each hexadecimal value represents two bytes. The first 20 bytes correspond to the IP header, while the last 20 bytes correspond to the SHA-1 authentication hash included in the CARP advertisement.

[[ File:Sha1-signature.png ]]

Sources :

https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf

https://www.giac.org/paper/gsec/4031/carp-free-fail-over-protocol/106433

== IP protocol number ==

CARP uses the same IP protocol number as VRRP (112). As a result, packet capture tools such as tcpdump interpret CARP packets as VRRP advertisements like we see above, even though the packets actually belong to the CARP protocol.

Why ??
The OpenBSD developers initially requested unique identifiers from the Internet Assigned Numbers Authority (IANA), but the request was denied because they had not followed the IETF standardization process. As a result, they chose to use protocol number 112, which was already assigned to VRRP. Despite this overlap, VRRP and CARP can coexist on the same network as long as their identifiers (VRRP group ID and CARP VHID) are different.

Sources :

https://en.wikipedia.org/wiki/Common_Address_Redundancy_Protocol

== Multiple VHID : Distribute traffic ==

Using multiple VHIDs in CARP allows several virtual IP addresses to be managed separately. This means different hosts can be Master for different services. For example, one host can manage the web service, while another manages the DNS service. Thanks to this, the hosts can be used efficiently while still ensuring high availability. It would not be practical to use a machine only when the other machine is not responding.

sources :

https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf

== Split-brain ==

Normally:
* one Master

* one or several Backups

If the Master host stops sending advertisements, the Backup hosts know that the master is no longer reachable. However, a problem can occur if the Backup hosts stop receiving each other's advertisements because the link between them is broken. In that case, each node may assume it is the Master.

When communication is restored, CARP resolves the problem:

1. The Masters can see each other
2. The real Master send the packet
3. The hosts compare CARP priority
4. The hosts with lower priority automatically switch back to Backup

sources :

https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf

== Master Election Process ==

1. All hosts start

All hosts begin in BACKUP state.

2. Each host starts a timer

* The delay depends on the CARP priority (advskew).

* The lower the advskew, the shorter the delay.

* The lower the advskew, the higher the priority.

3. The higher-priority host’s timer expires first

That host sends CARP advertisements when the timer expires.

It assigns the virtual IP (VIP) to its interface.

It becomes the MASTER host.

4. The other hosts receive the advertisement

They detect that a higher-priority host is active.

They remain in BACKUP state.

They do not send CARP advertisements.

In CARP, we usually choose the preferred Master by giving it a higher priority (lower advskew).

== Virtual Mac Address ==

The protocol automatically generates a virtual MAC address based on the VHID, ending with the VHID's value (00:00:5e:00:01:XX). This MAC address is identical on both hosts and is used by the MASTER node.

We can see here that the virtual mac address associated to the VIP 51.68.252.230 is : 00:00:5e:00:01:02 (02 at the end for vhid2)

'''command : tcpdump -e -ni vmx1 proto 112'''
[[ File:Virtual-mac.png ]]

✔ The CARP virtual MAC address is visible.

❌ The virtual IP address (VIP) is not visible as the source.

In CARP advertisements, the Ethernet source address corresponds to the virtual CARP MAC address derived from the VHID. However, the IP source address is the physical IP address of the router sending the advertisement. The virtual IP address (VIP) is not used as the source IP in these control packets. Instead, the VIP is used for normal traffic between clients and the virtual gateway.

sources :

https://dynfi.com/documentations/dynfi-firewall/configuration_firewall_VIP.html

== Basic configuration ==

We want :

router-002 to act as the master, while router-003 should operate as the backup for Intranought network.

router-003 to act as the master, while router-002 should operate as the backup for Public network.

1) Enable CARP Support

On the file ''/boot/loader.conf'', you need to add : '''carp_load="YES"'''

You are telling FreeBSD:

“Load the CARP kernel module automatically at boot time.”

If you want to load it now, but not permanently : '''kldload carp'''

If someone has build a custom FreeBSD kernel, they can include CARP directly by adding this line to the kernel configuration file: '''device carp'''

2) CARP configuration

In each router, on the new appropriate file '''/etc/rc.conf.d/netif/carp''', we will have the CARP configuration

router-002 :

ifconfig_vmx0_alias0="inet vhid 1 advskew 0 pass xxxxxxx alias 172.27.27.13/27"

ifconfig_vmx1_alias0="inet vhid 2 advskew 100 pass xxxxxxx alias 51.68.252.230/32"

router-003 :

ifconfig_vmx0_alias0="inet vhid 1 advskew 100 pass xxxxxxx alias 172.27.27.13/27"

ifconfig_vmx1_alias0="inet vhid 2 advskew 0 pass xxxxxxx alias 51.68.252.230/32"

So we gave a lower advskew (default is 0) to become the master router.

In older versions of FreeBSD (9 and earlier), CARP required creating a separate virtual interface such as carp0, and the shared IP address was attached to that interface. Starting with FreeBSD 10, this was simplified. CARP can now be configured directly on the physical interface using an alias with a VHID. The newer method is easier to configure and maintain, so it is recommended to use it in modern versions of FreeBSD.

commands :

ifconfig carp0 create
ifconfig carp0 vhid 1 pass xxxxxx
ifconfig carp0 inet 172.27.27.X/28'''

sources :

https://docs-archive.freebsd.org/doc/11.0-RELEASE/usr/local/share/doc/freebsd/en/books/handbook/carp.html

== Automatic blue/green deployment ==

In this context, “automatic Blue-Green” does not refer to a traditional application deployment strategy. Instead, it describes a network-level behavior enabled by CARP.

With CARP, two routers are deployed in parallel:

- One acts as the active router (MASTER)

- The other remains in standby (BACKUP)

Both routers are fully operational and connected to the network. Traffic is sent to a shared virtual IP address. At any given time, only the MASTER processes the traffic. However, if the MASTER fails, the BACKUP automatically takes over the virtual IP address without manual intervention.

This behavior resembles Blue-Green deployment because:

1. Two environments exist simultaneously

2. Only one actively handles traffic

3. Traffic can switch from one to the other

4. The switch is transparent to users

The difference is that in traditional Blue-Green deployment, the switch is usually performed manually during an update. With CARP, the switch happens automatically in case of failure.

== Priority vs advskew. ==

In the CARP protocol, the master router is not directly determined by priority, but rather by a value called advskew. Advskew is a numerical parameter between 0 and 255: the lower its value, the more likely the router is to become the master. Priority, on the other hand, is a more intuitive abstraction used in automation tools or logical configurations: the higher the priority, the higher the router's expected priority. To link the two, we use the formula: advskew = 255 − priority. Thus, a router with a priority of 200 will have a low advskew (55) and will become the master, while a router with a priority of 0 will have a maximum advskew (255) and will remain in backup mode. In short, priority is a value that is easily understood and logical for humans, while advskew is the value actually used by CARP's internal mechanism to determine the master or backup role.

Linked to {{T|2227}}

== Distribute traffic ==

The current infrastructure relies on two distinct networks at the ESXI hypervisor level:

- An internal network 172.27.27.0/28, referred to as IntraNought, used for internal communication and management.

- A public network used for public IP addresses exposed to the Internet.

The objective now is to evaluate the possibility of implementing an active/active model so that both routers would receive traffic during normal operation, while still maintaining high availability through CARP.

On the internal network (IntraNought), configuring CARP does not introduce a significant complexity. One router can operate as master and the other as backup. We will configure a first VHID.

However, on the public network, a constraint exists. Public IP addresses (IPFO) provided by OVH are associated with a specific MAC address. If both routers need to use the same public IP address, they must present the same MAC address.

By default, the ESXi vSwitch blocks situations where two virtual machines use the same MAC address. This requires enabling specific settings such as Promiscuous Mode, MAC Address Changes and Forged Transmits. It is technically feasible, but this adds complexity and may require network isolation by using a dedicated vSwitch for the routers.

Finally, even if both routers are active, they run on the same hypervisor. This means the hypervisor is a Single Point of Failure (SPOF). If the physical host fails, both routers would go down at the same time, so the redundancy only exists at the virtual level, not at the hardware level.

'''But at this point, a key question remains: If two VMs share the same MAC address, how does the vSwitch decide which VM should receive the traffic? Could it send the frame to the wrong VM? Or both ?'''

In our setup, two routers share the same virtual public IP and therefore the same virtual MAC address using CARP. However, only one router is active (the master) at a time.

When the master sends traffic, the ESXi vSwitch learns the virtual MAC address from the source MAC of the frames it receives. It then associates that MAC address with the master’s port. As a result, any incoming traffic destined for that MAC is forwarded only to the active router.

The backup router has the same virtual MAC configured, but it does not actively use it while in backup state. It does not respond to ARP requests and does not send traffic using that MAC. Therefore, the vSwitch does not associate the MAC with the backup’s port.

If a failover occurs, the backup router becomes master and starts sending traffic with the same virtual MAC. The vSwitch detects that the MAC now appears on a different port and updates its table. Traffic is then forwarded to the new master.

Security settings such as “MAC Address Changes” and “Forged Transmits” must be set to Accept to allow this MAC movement between ports and prevent ESXi from blocking the traffic as a potential MAC spoofing attempt.

Regarding Promiscuous Mode, VMware documentation states: “Promiscuous mode eliminates any reception filtering that the virtual machine adapter performs so that the guest operating system receives all traffic observed on the wire.” This means that the filtering normally performed by the virtual network adapter based on the destination MAC address is disabled. As a result, the guest operating system can receive all traffic visible on that VLAN. However, this setting does not modify the forwarding logic of the vSwitch itself. The vSwitch continues to perform standard Layer 2 MAC learning and forwards unicast traffic to a single port at a time. Therefore, even with Promiscuous Mode enabled, the vSwitch does not behave like a hub. Promiscuous mode is mainly used for traffic monitoring or packet sniffing, it is not required in our case.

Also, VMware documentation explicitly mentions legitimate scenarios where multiple adapters share the same MAC address, such as Microsoft NLB in unicast mode. This confirms that using a shared virtual MAC with CARP is a supported and valid design pattern.

Configuration to enable “MAC Address Changes” and “Forged Transmits” :

https://knowledge.broadcom.com/external/article/324520/configuring-promiscuous-mode-on-a-virtua.html
Source documentation:

https://wxcafe.net/posts/redondance-routeurs-openbsd-freebsd/

https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf

https://docs-archive.freebsd.org/doc/11.0-RELEASE/usr/local/share/doc/freebsd/en/books/handbook/carp.html

https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/vsphere-security/securing-vsphere-networking/securing-vsphere-standard-switches.html

https://dynfi.com/documentations/dynfi-firewall/configuration_firewall_VIP.html

Protocol CARP

2026-03-09T19:16:16Z

Yousra: /* Virtual Mac Address */

== Introduction and history ==

« CARP stands for Common Address Redundancy Protocol and its basic functionality is to allow multiple hosts to share a set of IP addresses ».

CARP was developed by the OpenBSD project in 2003 as an alternative to VRRP due to patent concerns related to Cisco's HSRP technology. To avoid potential legal issues and remain compatible with open-source principles, OpenBSD implemented its own redundancy protocol with added cryptographic features. CARP was later adopted by other systems such as FreeBSD and NetBSD. A userland implementation of CARP, named ucarp, made the protocol usable on Linux without requiring kernel integration.

These patent concerns originated from the development of VRRP by the Internet Engineering Task Force (IETF). Indeed, when the Internet Engineering Task Force (IETF) developed the Virtual Router Redundancy Protocol (VRRP), Cisco Systems indicated that it owned patents related to router redundancy mechanisms used in the protocol Hot Standby Router Protocol (HSRP). This created a legal risk for developers, particularly for open-source projects that wanted to implement VRRP freely.

Sources :

https://en.wikipedia.org/wiki/Common_Address_Redundancy_Protocol

https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf

== Master host and backup hosts ==

CARP creates a redundancy group, meaning several hosts are configured to share a virtual IP address. However, at any given time, only one host uses the shared IP address. This host is called the Master host and it receives and handles all traffic destined to that virtual IP.

When the Master host becomes unavailable (it crashed, turned off, or lost its network connection), the other hosts in the same redundancy group detect the failure. Immediately, one of the Backup hosts is elected as the new Master host. That means that it will take over the shared IP address.

This switch happens automatically, ensuring service continuity without clients noticing any interruption.

[[File:Schema redondance routeurs.png]]

Sources :

https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf

https://wxcafe.net/posts/redondance-routeurs-openbsd-freebsd/

== CARP advertisements ==

- The Master host periodically broadcasts CARP advertisements to the backup hosts.

- The Backup hosts listen but do not send any packets.

'''Each CARP advertisement contains :'''

1. The CARP version and packet type (CARP Header).

2. The VHID (Virtual Host ID), which identifies the redundancy group (CARP Header).

3. The advertisement parameters (advbase and advskew), which determine the host’s priority CARP (CARP Header).

4. The counter

5. The signature in SHA-1

'''CARP Advertisement Packet Format :'''

Each line represent 32 bits, 4 bytes

Length : 36 bytes

<pre>
/*
* The CARP header layout is as follows:
*
* 0 1 2 3
* 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
* |Version| Type | VirtualHostID | AdvSkew | Auth Len |
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
* | Demotion | AdvBase | Checksum |
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
* | Counter (1) |
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
* | Counter (2) |
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
* | SHA-1 HMAC (1) |
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
* | SHA-1 HMAC (2) |
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
* | SHA-1 HMAC (3) |
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
* | SHA-1 HMAC (4) |
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
* | SHA-1 HMAC (5) |
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
*
*/
</pre>

'''commande : tcpdump -i <interface> proto 112'''

[[ File:Carp-advertissement.png ]]

Sources :

https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf

https://github.com/openbsd/src/blob/master/sys/netinet/ip_carp.h

== Cryptography ==

'''All CARP advertisements are cryptographically signed :'''

Steps :

1. The Master host creates an advertisement.

2. It calculates a cryptographic signature, a SHA-1 hash using:

* the shared secret,

* the VHID,

* the virtual IP address,

* the advertisement parameters (advbase and advskew),

* the counter

* and other header fields.

3. It then sends the signed advertisement.

4. The Backup hosts receives the signed advertisement.

5. Each Backup host recalculates the signature using the same shared secret :

- If the calculated signature matches the received signature → the packet is considered valid because authentication and integrity are insured.

- If it does not match → the packet is rejected.

It protects against spoofing attacks : without cryptographic authentication, an attacker could send fake CARP advertisements and attempt to pretend to be or to become the Master host.

'''command : tcpdump -X -ni <interface> proto 112'''

We can observe the complete packet in the capture. Each hexadecimal value represents two bytes. The first 20 bytes correspond to the IP header, while the last 20 bytes correspond to the SHA-1 authentication hash included in the CARP advertisement.

[[ File:Sha1-signature.png ]]

Sources :

https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf

https://www.giac.org/paper/gsec/4031/carp-free-fail-over-protocol/106433

== IP protocol number ==

CARP uses the same IP protocol number as VRRP (112). As a result, packet capture tools such as tcpdump interpret CARP packets as VRRP advertisements like we see above, even though the packets actually belong to the CARP protocol.

Why ??
The OpenBSD developers initially requested unique identifiers from the Internet Assigned Numbers Authority (IANA), but the request was denied because they had not followed the IETF standardization process. As a result, they chose to use protocol number 112, which was already assigned to VRRP. Despite this overlap, VRRP and CARP can coexist on the same network as long as their identifiers (VRRP group ID and CARP VHID) are different.

Sources :

https://en.wikipedia.org/wiki/Common_Address_Redundancy_Protocol

== Multiple VHID : Distribute traffic ==

Using multiple VHIDs in CARP allows several virtual IP addresses to be managed separately. This means different hosts can be Master for different services. For example, one host can manage the web service, while another manages the DNS service. Thanks to this, the hosts can be used efficiently while still ensuring high availability. It would not be practical to use a machine only when the other machine is not responding.

sources :

https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf

== Split-brain ==

Normally:
* one Master

* one or several Backups

If the Master host stops sending advertisements, the Backup hosts know that the master is no longer reachable. However, a problem can occur if the Backup hosts stop receiving each other's advertisements because the link between them is broken. In that case, each node may assume it is the Master.

When communication is restored, CARP resolves the problem:

1. The Masters can see each other
2. The real Master send the packet
3. The hosts compare CARP priority
4. The hosts with lower priority automatically switch back to Backup

sources :

https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf

== Master Election Process ==

1. All hosts start

All hosts begin in BACKUP state.

2. Each host starts a timer

* The delay depends on the CARP priority (advskew).

* The lower the advskew, the shorter the delay.

* The lower the advskew, the higher the priority.

3. The higher-priority host’s timer expires first

That host sends CARP advertisements when the timer expires.

It assigns the virtual IP (VIP) to its interface.

It becomes the MASTER host.

4. The other hosts receive the advertisement

They detect that a higher-priority host is active.

They remain in BACKUP state.

They do not send CARP advertisements.

In CARP, we usually choose the preferred Master by giving it a higher priority (lower advskew).

== Virtual Mac Address ==

The protocol automatically generates a virtual MAC address based on the VHID, ending with the VHID's value (00:00:5e:00:01:XX). This MAC address is identical on both hosts and is used by the MASTER node.

We can see here that the virtual mac address associated to the VIP 51.68.252.230 is : 00:00:5e:00:01:02 (02 at the end for vhid2)

'''command : tcpdump -e -ni vmx1 proto 112'''
[[ File:Virtual-mac.png ]]

✔ The CARP virtual MAC address is visible.
❌ The virtual IP address (VIP) is not visible as the source.

In CARP advertisements, the Ethernet source address corresponds to the virtual CARP MAC address derived from the VHID. However, the IP source address is the physical IP address of the router sending the advertisement. The virtual IP address (VIP) is not used as the source IP in these control packets. Instead, the VIP is used for normal traffic between clients and the virtual gateway.

sources :

https://dynfi.com/documentations/dynfi-firewall/configuration_firewall_VIP.html

== Basic configuration ==

We want :

router-002 to act as the master, while router-003 should operate as the backup for Intranought network.

router-003 to act as the master, while router-002 should operate as the backup for Public network.

1) Enable CARP Support

On the file ''/boot/loader.conf'', you need to add : '''carp_load="YES"'''

You are telling FreeBSD:

“Load the CARP kernel module automatically at boot time.”

If you want to load it now, but not permanently : '''kldload carp'''

If someone has build a custom FreeBSD kernel, they can include CARP directly by adding this line to the kernel configuration file: '''device carp'''

2) CARP configuration

In each router, on the new appropriate file '''/etc/rc.conf.d/netif/carp''', we will have the CARP configuration

router-002 :

ifconfig_vmx0_alias0="inet vhid 1 advskew 0 pass xxxxxxx alias 172.27.27.13/27"

ifconfig_vmx1_alias0="inet vhid 2 advskew 100 pass xxxxxxx alias 51.68.252.230/32"

router-003 :

ifconfig_vmx0_alias0="inet vhid 1 advskew 100 pass xxxxxxx alias 172.27.27.13/27"

ifconfig_vmx1_alias0="inet vhid 2 advskew 0 pass xxxxxxx alias 51.68.252.230/32"

So we gave a lower advskew (default is 0) to become the master router.

In older versions of FreeBSD (9 and earlier), CARP required creating a separate virtual interface such as carp0, and the shared IP address was attached to that interface. Starting with FreeBSD 10, this was simplified. CARP can now be configured directly on the physical interface using an alias with a VHID. The newer method is easier to configure and maintain, so it is recommended to use it in modern versions of FreeBSD.

commands :

ifconfig carp0 create
ifconfig carp0 vhid 1 pass xxxxxx
ifconfig carp0 inet 172.27.27.X/28'''

sources :

https://docs-archive.freebsd.org/doc/11.0-RELEASE/usr/local/share/doc/freebsd/en/books/handbook/carp.html

== Automatic blue/green deployment ==

In this context, “automatic Blue-Green” does not refer to a traditional application deployment strategy. Instead, it describes a network-level behavior enabled by CARP.

With CARP, two routers are deployed in parallel:

- One acts as the active router (MASTER)

- The other remains in standby (BACKUP)

Both routers are fully operational and connected to the network. Traffic is sent to a shared virtual IP address. At any given time, only the MASTER processes the traffic. However, if the MASTER fails, the BACKUP automatically takes over the virtual IP address without manual intervention.

This behavior resembles Blue-Green deployment because:

1. Two environments exist simultaneously

2. Only one actively handles traffic

3. Traffic can switch from one to the other

4. The switch is transparent to users

The difference is that in traditional Blue-Green deployment, the switch is usually performed manually during an update. With CARP, the switch happens automatically in case of failure.

== Priority vs advskew. ==

In the CARP protocol, the master router is not directly determined by priority, but rather by a value called advskew. Advskew is a numerical parameter between 0 and 255: the lower its value, the more likely the router is to become the master. Priority, on the other hand, is a more intuitive abstraction used in automation tools or logical configurations: the higher the priority, the higher the router's expected priority. To link the two, we use the formula: advskew = 255 − priority. Thus, a router with a priority of 200 will have a low advskew (55) and will become the master, while a router with a priority of 0 will have a maximum advskew (255) and will remain in backup mode. In short, priority is a value that is easily understood and logical for humans, while advskew is the value actually used by CARP's internal mechanism to determine the master or backup role.

Linked to {{T|2227}}

== Distribute traffic ==

The current infrastructure relies on two distinct networks at the ESXI hypervisor level:

- An internal network 172.27.27.0/28, referred to as IntraNought, used for internal communication and management.

- A public network used for public IP addresses exposed to the Internet.

The objective now is to evaluate the possibility of implementing an active/active model so that both routers would receive traffic during normal operation, while still maintaining high availability through CARP.

On the internal network (IntraNought), configuring CARP does not introduce a significant complexity. One router can operate as master and the other as backup. We will configure a first VHID.

However, on the public network, a constraint exists. Public IP addresses (IPFO) provided by OVH are associated with a specific MAC address. If both routers need to use the same public IP address, they must present the same MAC address.

By default, the ESXi vSwitch blocks situations where two virtual machines use the same MAC address. This requires enabling specific settings such as Promiscuous Mode, MAC Address Changes and Forged Transmits. It is technically feasible, but this adds complexity and may require network isolation by using a dedicated vSwitch for the routers.

Finally, even if both routers are active, they run on the same hypervisor. This means the hypervisor is a Single Point of Failure (SPOF). If the physical host fails, both routers would go down at the same time, so the redundancy only exists at the virtual level, not at the hardware level.

'''But at this point, a key question remains: If two VMs share the same MAC address, how does the vSwitch decide which VM should receive the traffic? Could it send the frame to the wrong VM? Or both ?'''

In our setup, two routers share the same virtual public IP and therefore the same virtual MAC address using CARP. However, only one router is active (the master) at a time.

When the master sends traffic, the ESXi vSwitch learns the virtual MAC address from the source MAC of the frames it receives. It then associates that MAC address with the master’s port. As a result, any incoming traffic destined for that MAC is forwarded only to the active router.

The backup router has the same virtual MAC configured, but it does not actively use it while in backup state. It does not respond to ARP requests and does not send traffic using that MAC. Therefore, the vSwitch does not associate the MAC with the backup’s port.

If a failover occurs, the backup router becomes master and starts sending traffic with the same virtual MAC. The vSwitch detects that the MAC now appears on a different port and updates its table. Traffic is then forwarded to the new master.

Security settings such as “MAC Address Changes” and “Forged Transmits” must be set to Accept to allow this MAC movement between ports and prevent ESXi from blocking the traffic as a potential MAC spoofing attempt.

Regarding Promiscuous Mode, VMware documentation states: “Promiscuous mode eliminates any reception filtering that the virtual machine adapter performs so that the guest operating system receives all traffic observed on the wire.” This means that the filtering normally performed by the virtual network adapter based on the destination MAC address is disabled. As a result, the guest operating system can receive all traffic visible on that VLAN. However, this setting does not modify the forwarding logic of the vSwitch itself. The vSwitch continues to perform standard Layer 2 MAC learning and forwards unicast traffic to a single port at a time. Therefore, even with Promiscuous Mode enabled, the vSwitch does not behave like a hub. Promiscuous mode is mainly used for traffic monitoring or packet sniffing, it is not required in our case.

Also, VMware documentation explicitly mentions legitimate scenarios where multiple adapters share the same MAC address, such as Microsoft NLB in unicast mode. This confirms that using a shared virtual MAC with CARP is a supported and valid design pattern.

Configuration to enable “MAC Address Changes” and “Forged Transmits” :

https://knowledge.broadcom.com/external/article/324520/configuring-promiscuous-mode-on-a-virtua.html
Source documentation:

https://wxcafe.net/posts/redondance-routeurs-openbsd-freebsd/

https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf

https://docs-archive.freebsd.org/doc/11.0-RELEASE/usr/local/share/doc/freebsd/en/books/handbook/carp.html

https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/vsphere-security/securing-vsphere-networking/securing-vsphere-standard-switches.html

https://dynfi.com/documentations/dynfi-firewall/configuration_firewall_VIP.html

File:Virtual-mac.png

2026-03-09T17:38:58Z

Yousra:

Protocol CARP

2026-03-09T17:26:46Z

Yousra: /* Basic configuration */

== Introduction and history ==

« CARP stands for Common Address Redundancy Protocol and its basic functionality is to allow multiple hosts to share a set of IP addresses ».

CARP was developed by the OpenBSD project in 2003 as an alternative to VRRP due to patent concerns related to Cisco's HSRP technology. To avoid potential legal issues and remain compatible with open-source principles, OpenBSD implemented its own redundancy protocol with added cryptographic features. CARP was later adopted by other systems such as FreeBSD and NetBSD. A userland implementation of CARP, named ucarp, made the protocol usable on Linux without requiring kernel integration.

These patent concerns originated from the development of VRRP by the Internet Engineering Task Force (IETF). Indeed, when the Internet Engineering Task Force (IETF) developed the Virtual Router Redundancy Protocol (VRRP), Cisco Systems indicated that it owned patents related to router redundancy mechanisms used in the protocol Hot Standby Router Protocol (HSRP). This created a legal risk for developers, particularly for open-source projects that wanted to implement VRRP freely.

Sources :

https://en.wikipedia.org/wiki/Common_Address_Redundancy_Protocol

https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf

== Master host and backup hosts ==

CARP creates a redundancy group, meaning several hosts are configured to share a virtual IP address. However, at any given time, only one host uses the shared IP address. This host is called the Master host and it receives and handles all traffic destined to that virtual IP.

When the Master host becomes unavailable (it crashed, turned off, or lost its network connection), the other hosts in the same redundancy group detect the failure. Immediately, one of the Backup hosts is elected as the new Master host. That means that it will take over the shared IP address.

This switch happens automatically, ensuring service continuity without clients noticing any interruption.

[[File:Schema redondance routeurs.png]]

Sources :

https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf

https://wxcafe.net/posts/redondance-routeurs-openbsd-freebsd/

== CARP advertisements ==

- The Master host periodically broadcasts CARP advertisements to the backup hosts.

- The Backup hosts listen but do not send any packets.

'''Each CARP advertisement contains :'''

1. The CARP version and packet type (CARP Header).

2. The VHID (Virtual Host ID), which identifies the redundancy group (CARP Header).

3. The advertisement parameters (advbase and advskew), which determine the host’s priority CARP (CARP Header).

4. The counter

5. The signature in SHA-1

'''CARP Advertisement Packet Format :'''

Each line represent 32 bits, 4 bytes

Length : 36 bytes

<pre>
/*
* The CARP header layout is as follows:
*
* 0 1 2 3
* 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
* |Version| Type | VirtualHostID | AdvSkew | Auth Len |
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
* | Demotion | AdvBase | Checksum |
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
* | Counter (1) |
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
* | Counter (2) |
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
* | SHA-1 HMAC (1) |
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
* | SHA-1 HMAC (2) |
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
* | SHA-1 HMAC (3) |
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
* | SHA-1 HMAC (4) |
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
* | SHA-1 HMAC (5) |
* +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
*
*/
</pre>

'''commande : tcpdump -i <interface> proto 112'''

[[ File:Carp-advertissement.png ]]

Sources :

https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf

https://github.com/openbsd/src/blob/master/sys/netinet/ip_carp.h

== Cryptography ==

'''All CARP advertisements are cryptographically signed :'''

Steps :

1. The Master host creates an advertisement.

2. It calculates a cryptographic signature, a SHA-1 hash using:

* the shared secret,

* the VHID,

* the virtual IP address,

* the advertisement parameters (advbase and advskew),

* the counter

* and other header fields.

3. It then sends the signed advertisement.

4. The Backup hosts receives the signed advertisement.

5. Each Backup host recalculates the signature using the same shared secret :

- If the calculated signature matches the received signature → the packet is considered valid because authentication and integrity are insured.

- If it does not match → the packet is rejected.

It protects against spoofing attacks : without cryptographic authentication, an attacker could send fake CARP advertisements and attempt to pretend to be or to become the Master host.

'''command : tcpdump -X -ni <interface> proto 112'''

We can observe the complete packet in the capture. Each hexadecimal value represents two bytes. The first 20 bytes correspond to the IP header, while the last 20 bytes correspond to the SHA-1 authentication hash included in the CARP advertisement.

[[ File:Sha1-signature.png ]]

Sources :

https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf

https://www.giac.org/paper/gsec/4031/carp-free-fail-over-protocol/106433

== IP protocol number ==

CARP uses the same IP protocol number as VRRP (112). As a result, packet capture tools such as tcpdump interpret CARP packets as VRRP advertisements like we see above, even though the packets actually belong to the CARP protocol.

Why ??
The OpenBSD developers initially requested unique identifiers from the Internet Assigned Numbers Authority (IANA), but the request was denied because they had not followed the IETF standardization process. As a result, they chose to use protocol number 112, which was already assigned to VRRP. Despite this overlap, VRRP and CARP can coexist on the same network as long as their identifiers (VRRP group ID and CARP VHID) are different.

Sources :

https://en.wikipedia.org/wiki/Common_Address_Redundancy_Protocol

== Multiple VHID : Distribute traffic ==

Using multiple VHIDs in CARP allows several virtual IP addresses to be managed separately. This means different hosts can be Master for different services. For example, one host can manage the web service, while another manages the DNS service. Thanks to this, the hosts can be used efficiently while still ensuring high availability. It would not be practical to use a machine only when the other machine is not responding.

sources :

https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf

== Split-brain ==

Normally:
* one Master

* one or several Backups

If the Master host stops sending advertisements, the Backup hosts know that the master is no longer reachable. However, a problem can occur if the Backup hosts stop receiving each other's advertisements because the link between them is broken. In that case, each node may assume it is the Master.

When communication is restored, CARP resolves the problem:

1. The Masters can see each other
2. The real Master send the packet
3. The hosts compare CARP priority
4. The hosts with lower priority automatically switch back to Backup

sources :

https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf

== Master Election Process ==

1. All hosts start

All hosts begin in BACKUP state.

2. Each host starts a timer

* The delay depends on the CARP priority (advskew).

* The lower the advskew, the shorter the delay.

* The lower the advskew, the higher the priority.

3. The higher-priority host’s timer expires first

That host sends CARP advertisements when the timer expires.

It assigns the virtual IP (VIP) to its interface.

It becomes the MASTER host.

4. The other hosts receive the advertisement

They detect that a higher-priority host is active.

They remain in BACKUP state.

They do not send CARP advertisements.

In CARP, we usually choose the preferred Master by giving it a higher priority (lower advskew).

== Virtual Mac Address ==

The protocol automatically generates a virtual MAC address based on the VHID, ending with the VHID's value (00:00:5e:00:01:XX). This MAC address is identical on both hosts and is used by the MASTER node.

sources :

https://dynfi.com/documentations/dynfi-firewall/configuration_firewall_VIP.html

== Basic configuration ==

We want :

router-002 to act as the master, while router-003 should operate as the backup for Intranought network.

router-003 to act as the master, while router-002 should operate as the backup for Public network.

1) Enable CARP Support

On the file ''/boot/loader.conf'', you need to add : '''carp_load="YES"'''

You are telling FreeBSD:

“Load the CARP kernel module automatically at boot time.”

If you want to load it now, but not permanently : '''kldload carp'''

If someone has build a custom FreeBSD kernel, they can include CARP directly by adding this line to the kernel configuration file: '''device carp'''

2) CARP configuration

In each router, on the new appropriate file '''/etc/rc.conf.d/netif/carp''', we will have the CARP configuration

router-002 :

ifconfig_vmx0_alias0="inet vhid 1 advskew 0 pass xxxxxxx alias 172.27.27.13/27"

ifconfig_vmx1_alias0="inet vhid 2 advskew 100 pass xxxxxxx alias 51.68.252.230/32"

router-003 :

ifconfig_vmx0_alias0="inet vhid 1 advskew 100 pass xxxxxxx alias 172.27.27.13/27"

ifconfig_vmx1_alias0="inet vhid 2 advskew 0 pass xxxxxxx alias 51.68.252.230/32"

So we gave a lower advskew (default is 0) to become the master router.

In older versions of FreeBSD (9 and earlier), CARP required creating a separate virtual interface such as carp0, and the shared IP address was attached to that interface. Starting with FreeBSD 10, this was simplified. CARP can now be configured directly on the physical interface using an alias with a VHID. The newer method is easier to configure and maintain, so it is recommended to use it in modern versions of FreeBSD.

commands :

ifconfig carp0 create
ifconfig carp0 vhid 1 pass xxxxxx
ifconfig carp0 inet 172.27.27.X/28'''

sources :

https://docs-archive.freebsd.org/doc/11.0-RELEASE/usr/local/share/doc/freebsd/en/books/handbook/carp.html

== Automatic blue/green deployment ==

In this context, “automatic Blue-Green” does not refer to a traditional application deployment strategy. Instead, it describes a network-level behavior enabled by CARP.

With CARP, two routers are deployed in parallel:

- One acts as the active router (MASTER)

- The other remains in standby (BACKUP)

Both routers are fully operational and connected to the network. Traffic is sent to a shared virtual IP address. At any given time, only the MASTER processes the traffic. However, if the MASTER fails, the BACKUP automatically takes over the virtual IP address without manual intervention.

This behavior resembles Blue-Green deployment because:

1. Two environments exist simultaneously

2. Only one actively handles traffic

3. Traffic can switch from one to the other

4. The switch is transparent to users

The difference is that in traditional Blue-Green deployment, the switch is usually performed manually during an update. With CARP, the switch happens automatically in case of failure.

== Priority vs advskew. ==

In the CARP protocol, the master router is not directly determined by priority, but rather by a value called advskew. Advskew is a numerical parameter between 0 and 255: the lower its value, the more likely the router is to become the master. Priority, on the other hand, is a more intuitive abstraction used in automation tools or logical configurations: the higher the priority, the higher the router's expected priority. To link the two, we use the formula: advskew = 255 − priority. Thus, a router with a priority of 200 will have a low advskew (55) and will become the master, while a router with a priority of 0 will have a maximum advskew (255) and will remain in backup mode. In short, priority is a value that is easily understood and logical for humans, while advskew is the value actually used by CARP's internal mechanism to determine the master or backup role.

Linked to {{T|2227}}

== Distribute traffic ==

The current infrastructure relies on two distinct networks at the ESXI hypervisor level:

- An internal network 172.27.27.0/28, referred to as IntraNought, used for internal communication and management.

- A public network used for public IP addresses exposed to the Internet.

The objective now is to evaluate the possibility of implementing an active/active model so that both routers would receive traffic during normal operation, while still maintaining high availability through CARP.

On the internal network (IntraNought), configuring CARP does not introduce a significant complexity. One router can operate as master and the other as backup. We will configure a first VHID.

However, on the public network, a constraint exists. Public IP addresses (IPFO) provided by OVH are associated with a specific MAC address. If both routers need to use the same public IP address, they must present the same MAC address.

By default, the ESXi vSwitch blocks situations where two virtual machines use the same MAC address. This requires enabling specific settings such as Promiscuous Mode, MAC Address Changes and Forged Transmits. It is technically feasible, but this adds complexity and may require network isolation by using a dedicated vSwitch for the routers.

Finally, even if both routers are active, they run on the same hypervisor. This means the hypervisor is a Single Point of Failure (SPOF). If the physical host fails, both routers would go down at the same time, so the redundancy only exists at the virtual level, not at the hardware level.

'''But at this point, a key question remains: If two VMs share the same MAC address, how does the vSwitch decide which VM should receive the traffic? Could it send the frame to the wrong VM? Or both ?'''

In our setup, two routers share the same virtual public IP and therefore the same virtual MAC address using CARP. However, only one router is active (the master) at a time.

When the master sends traffic, the ESXi vSwitch learns the virtual MAC address from the source MAC of the frames it receives. It then associates that MAC address with the master’s port. As a result, any incoming traffic destined for that MAC is forwarded only to the active router.

The backup router has the same virtual MAC configured, but it does not actively use it while in backup state. It does not respond to ARP requests and does not send traffic using that MAC. Therefore, the vSwitch does not associate the MAC with the backup’s port.

If a failover occurs, the backup router becomes master and starts sending traffic with the same virtual MAC. The vSwitch detects that the MAC now appears on a different port and updates its table. Traffic is then forwarded to the new master.

Security settings such as “MAC Address Changes” and “Forged Transmits” must be set to Accept to allow this MAC movement between ports and prevent ESXi from blocking the traffic as a potential MAC spoofing attempt.

Regarding Promiscuous Mode, VMware documentation states: “Promiscuous mode eliminates any reception filtering that the virtual machine adapter performs so that the guest operating system receives all traffic observed on the wire.” This means that the filtering normally performed by the virtual network adapter based on the destination MAC address is disabled. As a result, the guest operating system can receive all traffic visible on that VLAN. However, this setting does not modify the forwarding logic of the vSwitch itself. The vSwitch continues to perform standard Layer 2 MAC learning and forwards unicast traffic to a single port at a time. Therefore, even with Promiscuous Mode enabled, the vSwitch does not behave like a hub. Promiscuous mode is mainly used for traffic monitoring or packet sniffing, it is not required in our case.

Also, VMware documentation explicitly mentions legitimate scenarios where multiple adapters share the same MAC address, such as Microsoft NLB in unicast mode. This confirms that using a shared virtual MAC with CARP is a supported and valid design pattern.

Configuration to enable “MAC Address Changes” and “Forged Transmits” :

https://knowledge.broadcom.com/external/article/324520/configuring-promiscuous-mode-on-a-virtua.html
Source documentation:

https://wxcafe.net/posts/redondance-routeurs-openbsd-freebsd/

https://freebsdfoundation.org/wp-content/uploads/2022/11/zaborski_CARP.pdf

https://docs-archive.freebsd.org/doc/11.0-RELEASE/usr/local/share/doc/freebsd/en/books/handbook/carp.html

https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/vsphere-security/securing-vsphere-networking/securing-vsphere-standard-switches.html

https://dynfi.com/documentations/dynfi-firewall/configuration_firewall_VIP.html