User:Dereckson/Sandbox
From Nasqueron Agora
Terraform Secure State Workflow
Overview
This workflow provides a secure, auditable method for managing Terraform and OpenTofu state on FreeBSD using local encryption. All sensitive infrastructure state is encrypted at rest using PEFS, secrets are injected from Vault at runtime, and backups are stored in OVH Object Storage with Object Lock for immutability.
The core principle is: no plaintext state or secrets ever exist outside the PEFS encrypted volume, and the volume is mounted only for the duration of a single Terraform operation.
Architecture
Schema
┌─────────────────────────────────────────────────────────────────────────────┐
│ DEVELOPER TERMINAL ON COMPLECTOR │
│ │
│ $ cd /opt/salt/nasqueron-operations/terraform/<project> │
│ $ tf plan │
│ $ tf apply │
└──────────────────────────────────┬──────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ WRAPPER /usr/local/bin/tf │
│ │
│ 1. Validate current directory is a top-level project under │
│ /opt/salt/nasqueron-operations/terraform/ │
│ │
│ 2. Determine runtime: │
│ • openbao, ovh-ops-backups → terraform │
│ • everything else → tofu │
│ │
│ 3. Mount PEFS if not already mounted: │
│ pefs mount /opt/terraform.enc /opt/terraform/encrypted │
│ │
│ 4. Load encryption key if not already in kernel: │
│ vault kv get -field=password ops/infra/complector/terraform/pefs │
│ | │
│ └──► pefs addkey -c /opt/terraform -j - │
│ │
│ 5. Prepare directories and environment: │
│ mkdir -p /opt/terraform/encrypted/tf-state/<project> │
│ export TF_DATA_DIR=/opt/terraform/tf-data/<project> │
│ │
│ 6. Execute Terraform/OpenTofu: │
│ terraform <args> -or- tofu <args> │
│ │
│ 7. On exit (trap EXIT): │
│ pefs umount /opt/terraform │
└──────────────────────────────────┬──────────────────────────────────────────┘
│
┌──────────────┴──────────────────┐
│ │
▼ ▼
┌───────────────────────────────────────┐ ┌─────────────────────────────────┐
│ ENCRYPTED VOLUME (PEFS) │ │ PUBLIC REPOSITORY │
│ /opt/terraform.enc/ │ │ /opt/salt/nasqueron-operations/ │
│ │ │ │
│ Mounted at /opt/terraform/encrypted │ │ terraform/ │
│ while tf is running. Sensitive. │ │ ├── ovh-ops-backups/ │
│ │ │ │ ├── main.tf │
│ /o/t/encrypted/tf-state/<project>/ │ │ │ ├── backend.tf │
│ └── terraform.tfstate │ │ │ └── modules/ │
│ └── terraform.tfstate.bak │ │ │ │
│───────────────────────────────────────│ │ ├── openbao/ │
│ WORKING DIRECTORY │ │ │ ├── main.tf │
│ /opt/terraform/ │ │ │ └── ... │
│ │ │ │ │
│ /o/t/tf-data/<project>/ unencrypted │ │ └── <other-projects>/ │
│ └── .teraform/ │ │ │
│ └── providers/ │ │ Contains only public code. │
│ └── modules/ │ │ │
│ │ │ │
└───────────────────────────────────────┘ └─────────────────────────────────┘
│
│ (encrypted at rest by PEFS AES-256-XTS)
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ BACKUP PIPELINE (Restic) │
│ │
│ Restic /opt/terraform.enc │
│ │ │
│ ▼ │
│ OVH Object Storage (S3-compatible) │
│ • Bucket: nasqueron-backups-amaris │
│ • Region: EU-WEST-PAR │
│ • Versioning: enabled │
│ • Object Lock: governance mode, 90-day retention │
│ • Per-client prefix isolation via IAM policies │
└─────────────────────────────────────────────────────────────────────────────┘
Components
| Component | Role | Location |
|---|---|---|
/usr/local/bin/tf-ops
|
Wrapper script that orchestrates the secure workflow | /usr/local/bin/tf (symlink or alias)
|
| PEFS | AES-256-XTS stacked filesystem for encrypting state at rest | /opt/terraform.enc → mounted at /opt/terraform
|
| Vault | Stores the PEFS passphrase and all infrastructure secrets | Path: ops/infra/complector/terraform/pefs
|
| Terraform / OpenTofu | Infrastructure-as-code engine | Selected per-project by the wrapper |
| Restic | Incremental, deduplicating backup client | Backs up the raw /opt/terraform.enc directory
|
| OVH Object Storage | S3-compatible remote storage with Object Lock | Buckets: nasqueron-backups-{amaris,darak,vakor}
|
Security Model
Encryption Layers
The workflow applies defense in depth across three layers:
| Layer | Technology | Protects Against |
|---|---|---|
| At-rest encryption | PEFS (AES-256-XTS) | Disk theft, unauthorized filesystem access, unencrypted backups |
| Secret injection | HashiCorp Vault (KV v2) | Plaintext passwords in scripts, config files, or shell history |
| Remote backup immutability | OVH Object Lock (Governance) | Ransomware deletion of backups, accidental pruning |
Secret Flow
Vault KV v2 PEFS Kernel Terraform
(ops/infra/complector/ ──► (decrypts /opt/terraform.enc ──► reads/writes
terraform/pefs) in-memory, never touches encrypted state
disk in plaintext) inside PEFS mount
The PEFS passphrase is never written to disk. It flows directly from the Vault API into pefs addkey via a Unix pipe (-j - reads from stdin). The plaintext passphrase exists only in process memory for the duration of the addkey system call.
Trust Boundaries
- The public repository contains only declarative infrastructure code. It is safe for Salt to read, for CI to lint, and for developers to clone.
- The PEFS volume contains all sensitive runtime artifacts (state files, provider caches, module downloads). It is encrypted and unmounted by default.
- The Vault server is the single source of truth for the PEFS passphrase. Access is governed by Vault ACL policies.
- The OVH backup stores only the encrypted PEFS volume. The backup provider never sees plaintext state or secrets.
The tf Wrapper Script
Usage
# From any top-level Terraform project directory: $ cd /opt/salt/nasqueron-operations/terraform/ovh-ops-backups $ tf init $ tf plan $ tf apply -auto-approve $ tf state list $ tf output -json
The script must be executed from within a top-level subdirectory of /opt/salt/nasqueron-operations/terraform/. Running it from a nested module directory will result in an error.
Behavior
| Step | Action | Failure Handling |
|---|---|---|
| 1 | Validate working directory | Exits with error code 1 if not in a valid project directory |
| 2 | Select Terraform or OpenTofu binary | Falls back to tofu unless the project is in the whitelist
|
| 3 | Mount PEFS if unmounted | Exits with error code 2 if pefs mount fails
|
| 4 | Load key from Vault if not in kernel | Exits with error code 1 if Vault or pefs addkey fails
|
| 5 | Create state and cache directories | Uses mkdir -p; idempotent
|
| 6 | Execute Terraform/OpenTofu | Inherits the child process exit code |
| 7 | Unmount PEFS via trap EXIT
|
Always runs, even on Ctrl+C or script failure |
Runtime Selection
The wrapper maintains a whitelist of projects that still require the HashiCorp terraform binary. All other projects default to tofu (OpenTofu).
STILL_REQUIRE_TERRAFORM="openbao ovh-ops-backups"
This allows a gradual migration from Terraform to OpenTofu without changing the developer workflow.
Configuration Reference
Repository Structure
/opt/salt/nasqueron-operations/
└── terraform/
├── ovh-ops-backups/ # Uses terraform (whitelisted)
│ ├── main.tf
│ ├── backend.tf # Must point to PEFS state path
│ └── modules/
├── openbao/ # Uses terraform (whitelisted)
│ └── ...
├── some-other-project/ # Uses tofu (default)
│ └── ...
└── ...
PEFS Layout
When mounted, the PEFS volume exposes the following structure:
/opt/terraform/
├── tf-state/
│ ├── ovh-ops-backups/
│ │ ├── terraform.tfstate
│ │ └── terraform.tfstate.backup
│ ├── openbao/
│ │ └── terraform.tfstate
│ └── ...
└── tf-data/
├── ovh-ops-backups/
│ └── .terraform/
│ ├── providers/
│ └── modules/
├── openbao/
│ └── .terraform/
└── ...
Vault Paths
| Path | Field | Purpose |
|---|---|---|
ops/infra/complector/terraform/pefs
|
password
|
PEFS passphrase used to decrypt /opt/terraform.enc
|
ops/secrets/backups/ovh/s3/<account>
|
access_key, secret_key
|
Per-client S3 credentials for Restic backups |
Terraform Backend Configuration
Each project must configure its local backend to write state into the PEFS-mounted directory. Example backend.tf:
terraform {
backend "local" {
path = "/opt/terraform/tf-state/ovh-ops-backups/terraform.tfstate"
}
}
Operations
Initial Setup
- Create the PEFS encrypted volume:
mkdir -p /opt/terraform.enc pefs mount /opt/terraform.enc /opt/terraform pefs addkey -c /opt/terraform # Store the passphrase in Vault: vault kv put ops/infra/complector/terraform/pefs password="<your-passphrase>" pefs umount /opt/terraform
- Install the wrapper script to
/usr/local/bin/tf-opsand create a symlink:
chmod +x /usr/local/bin/tf-ops ln -s /usr/local/bin/tf-ops /usr/local/bin/tf
- Ensure Vault CLI is authenticated:
vault login
Daily Usage
$ pm go terraform-ovh-ops-backups # or: cd /opt/.../terraform/ovh-ops-backups $ tf init $ tf plan $ tf apply
The wrapper handles mounting, key loading, and unmounting transparently. No manual PEFS commands are required.
Backup and Restore
Backups are performed by Restic against the raw encrypted volume:
restic -r s3:https://s3.eu-west-par.io.cloud.ovh.net/nasqueron-backups-amaris \
backup /opt/terraform.enc
To restore on a new machine:
restic -r s3:https://s3.eu-west-par.io.cloud.ovh.net/nasqueron-backups-amaris \
restore latest --target /opt/terraform.enc
pefs mount /opt/terraform.enc /opt/terraform
vault kv get -field=password ops/infra/complector/terraform/pefs | \
pefs addkey -c /opt/terraform -j -
Disaster Recovery
If the Vault server is unavailable, the PEFS volume cannot be unlocked. Ensure Vault itself is backed up and highly available. The PEFS passphrase should also be stored in a secure offline location (e.g., a hardware security module, a sealed envelope in a safe, or an encrypted password manager) as an emergency recovery path.
References
- PEFS — Private Encrypted File System for FreeBSD
- PEFS official documentation
- PEFS Developer Summit 2011 presentation
- Vault — HashiCorp Vault secrets management
- OpenTofu
- Restic backup tool
- OVH Object Storage
