Operations grimoire/Deploy with Terraform

Overview

In addition to Salt, Nasqueron manages Infrastructure as Code using Terraform and OpenTofu. We deploy from Complector using /opt/salt/nasqueron-operations as our local copy of rOPS, authoritative for both Salt and Terraform.

It's important to work from there to save a shared Terraform state.

We provide a tf wrapper that automatically runs OpenTofu or Terraform based on the directory and mounts the encrypted state directory on the fly.

Paths

Terraform states are stored on a local backend. To protect sensitive infrastructure state (which often contains plaintext secrets and topology data) at rest, we use a split-architecture approach combining ZFS and PEFS (Private Encrypted File System).

Paths for Terraform / OpenTofu deployment

Path	Type	Purpose	Filesystem Properties
/opt/salt/nasqueron-operations/terraform/	Standard Dir	Terraform/OpenTofu declarative code	Standard ZFS
/opt/terraform.enc	ZFS Dataset	Underlying encrypted block storage for PEFS	compression=zstd
/opt/terraform/encrypted	PEFS Mount	Encrypted state files. Mounted only during TF operations.	AES-256-XTS
/opt/terraform/tf-data	ZFS Dataset	Provider cache and modules. TF_DATA_DIR target.	compression=lz4, recordsize=128K (optimized for Go binaries)

Access control

• PEFS operations (mounting, adding keys) are executed as the deploy user.
• The wrapper verifies the invoking user belongs to the ops or deploy group before proceeding.
• The PEFS passphrase to read the Terraform states encryption key is stored in HashiCorp Vault at ops/infra/complector/terraform/pefs.

Standard operations

The tf wrapper automatically mounts PEFS, fetches the decryption key from Vault, executes the command, and unmounts PEFS upon exit, even on failure or Ctrl+C.

You can use it exactly like you would use terraform or tofu commands:

$ cd /opt/salt/nasqueron-operations/terraform/ovh-ops-backups
$ tf init
$ tf plan
$ tf apply

Specific deployment notes

Maintenance of .tfstate files

For manual state surgery or moving files, use the explicit mount commands. These bypass the automatic unmount.

# Mount PEFS and load keys for manual maintenance
tf mount

# Perform manual operations (e.g., moving state files)
cd /opt/terraform/encrypted/tf-states
# ...

# Unmount when finished
tf unmount

Choose between opentofu or terraform

By default, tofu is used.

The wrapper maintains an allowlist of project that still require the HashiCorp Terraform versions.

The main blocker is hashicorp/vault, not compiled for FreeBSD on OpenTofu registry.

To add or remove a project there, edit the STILL_REQUIRE_TERRAFORM variable in rOPS: roles/salt-primary/opentofu/files/tf.sh.

New project configuration

Every Terraform project must explicitly configure its local backend to write state into the PEFS-encrypted mount.

Add this to your project's backend.tf or providers.tf:

terraform {
  backend "local" {
    path = "/opt/terraform/encrypted/tf-states/<project-name>/terraform.tfstate"
  }
}

Note: The provider cache path does not need to be configured in HCL. The wrapper automatically exports TF_DATA_DIR=/opt/terraform/tf-data/<project-name>.

Vault / OpenBao

General notes

OpenTofu support

As of 2026-09-06, the Vault provider isn't compiled for FreeBSD. You need to use Terraform instead.

Vault

You need a Vault token to allow the provider to connect.

You also need to set VAULT_ADDR to https://172.27.27.7:8200 as Vault doesn't listen on 127.0.0.1

$ export VAULT_ADDR=https://172.27.27.7:8200
$ sudo /opt/salt/nasqueron-operations/utils/vault/issue-admin-token.py > ~/.vault-token
    
$ cd /opt/salt/nasqueron-operations/terraform/openbao
$ tf init # if you've a new entry requiring a module, it needs to be installed
$ tf plan
$ tf apply

It could be interesting to maintain an alternative policy to admin for the Terraform provider, restricted to the current actions. Note benefits would be limited as those include policies management.

Propagate secrets (DRP)

No automatic secret rotation

Secrets rotation is disabled with a lifecycle management ignore_changes = [ secret_id, ].

To rotate a secret, it needs first to be destroyed from terraform state: terraform destroy -target=module.viperserv_approle.vault_approle_auth_backend_role_secret_id.this

Full procedure

Once the AppRole have been created in Vault, they need to be provisioned to the relevant configuration files.

$ cd /opt/salt/nasqueron-operations/terraform/openbao
$ tf init # if you've a new entry requiring a module, it needs to be installed
$ tf plan
$ tf apply

$ cd /opt/salt/nasqueron-operations
$ salt windriver state.sls_id /usr/local/etc/secrets/rhyne-wyse.yaml roles/reports/rhyne-wyse/config
$ salt windriver state.sls_id /srv/viperserv/.credentials roles/viperserv/eggdrop/config
# Missing for router

Important. Each time you'll reprovision the secrets, they will change.

Don't forget to always apply this full procedure.

CARP OVH secrets usage

The OVH credentials used by the CARP failover script are stored in Vault at:

apps/network/carp-hyper-001-switch

These credentials are accessed by router-002 and router-003. They are then used by a script triggered by devd to update the OVH failover IP MAC address depending on the CARP state (MASTER/BACKUP). The script authenticates to Vault using AppRole (via secretsmith) to retrieve them.

The AppRole credentials (role_id and secret_id) used by the script are stored in:

ops/secrets/network/router/vault

Vault secrets are provisioned using Terraform (openbao module), and must be propagated to the routers using Salt.

The Vault Secrets may need to be rotated, so we need a simple way to propagate updated credentials to routers. This is enabled by the mechanism introduced in D4026.

ovh-ops-backups

This configuration is handled by Terraform.

It requires a full access to public cloud project for backups, as the project is fully configured by those Terraform files.

You'll need to get application key, application secret and consumer_key in your $HOME/.ovh.conf on Complector. You can also generate such files with ovhcloud login.

First Terraform configuration (DRP)

Initialize Terraform working directories

Terraform and OpenTofu are currently deployed from Complector. The working directories are encrypted, a wrapper decrypt them for each operation.

You need to restore from a backup the content of /opt/terraform.enc or if not available reinitialize a new encrypted directory. Currently, the procedure is valid for FreeBSD, as it requires PEFS to encrypt working director.

1. On the salt primary server, deploy the full roles/salt-primary, or at least the roles/salt-primary/opentofu unit
2. Ensure pefs is active with kldstat | grep pefs
3. Ensure Vault is up, then vault kv put ops/infra/complector/terraform/pefs password=$(openssl rand -hex 48)
4. Run initialize-pefs ops/infra/complector/terraform/pefs /opt/terraform.enc /opt/terraform/encrypted

The vault previous version of the secret, if still available, is for the previous directory, keep it if we restore afterhand. But you need a new password for your new fresh encrypted directory.

Troubleshoot

Error: Module not installed

You need to run tf init to prepare for any new provider.

$ tf plan

│ Error: Module not installed
│ 
│   on rhyne_wyse.tf line 23:
│   23: module "rhyne_wyse_approle" {
│ 
│ This module is not yet installed. Run "tofu init" to install all modules required by this configuration.

Error: Incompatible provider version

$ tf init

Initializing the backend...
Initializing modules...
- rhyne_wyse_approle in modules/app_credentials

Initializing provider plugins...
- Finding hashicorp/vault versions matching "5.3.0"...
╷
│ Error: Incompatible provider version
│ 
│ Provider registry.opentofu.org/hashicorp/vault v5.3.0 does not have a package available for your current platform, freebsd_amd64.
│ 
│ Provider releases are separate from OpenTofu CLI releases, so not all providers are available for all platforms. Other versions of this provider may have different platforms supported.

On OpenTofu, some modules are only compiled for Linux, not for FreeBSD. Switch to Terraform pending a solution to help the OpenTofu builds.

To allow the wrapper to run terraform commands, add your project directory to the STILL_REQUIRE_TERRAFORM variable in rOPS: roles/salt-primary/opentofu/files/tf.sh.

State rollback

When declaring new Terraform resources, it's possible to remove from the state an item, without actually destroying it at infrastructure level:

   $ tf state rm <state name>
   Removed ….…
   Successfully removed 1 resource instance(s).