User:Dereckson/Sandbox

Terraform Secure State Workflow

Overview

This workflow provides a secure, auditable method for managing Terraform and OpenTofu state on FreeBSD using local encryption. All sensitive infrastructure state is encrypted at rest using PEFS, secrets are injected from Vault at runtime, and backups are stored in OVH Object Storage with Object Lock for immutability.

The core principle is: no plaintext state or secrets ever exist outside the PEFS encrypted volume, and the volume is mounted only for the duration of a single Terraform operation.

Architecture

Schema

┌─────────────────────────────────────────────────────────────────────────────┐
│                      DEVELOPER TERMINAL ON COMPLECTOR                       │
│                                                                             │
│  $ cd /opt/salt/nasqueron-operations/terraform/<project>                    │
│  $ tf plan                                                                  │
│  $ tf apply                                                                 │
└──────────────────────────────────┬──────────────────────────────────────────┘
                                   │
                                  ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                        WRAPPER /usr/local/bin/tf                            │
│                                                                             │
│  1. Validate current directory is a top-level project under                 │
│     /opt/salt/nasqueron-operations/terraform/                               │
│                                                                             │
│  2. Determine runtime:                                                      │
│     • openbao, ovh-ops-backups  →  terraform                                │
│     • everything else           →  tofu                                     │
│                                                                             │
│  3. Mount PEFS if not already mounted:                                      │
│     pefs mount /opt/terraform.enc /opt/terraform/encrypted                  │
│                                                                             │
│  4. Load encryption key if not already in kernel:                           │
│     vault kv get -field=password ops/infra/complector/terraform/pefs        │
│	|                                                                     │
│	└──►  pefs addkey -c /opt/terraform -j -                              │
│                                                                             │
│  5. Prepare directories and environment:                                    │
│     mkdir -p /opt/terraform/encrypted/tf-state/<project>                    │
│     export TF_DATA_DIR=/opt/terraform/tf-data/<project>                     │
│                                                                             │
│  6. Execute Terraform/OpenTofu:                                             │
│     terraform <args>  -or-  tofu <args>                                     │
│                                                                             │
│  7. On exit (trap EXIT):                                                    │
│     pefs umount /opt/terraform                                              │
└──────────────────────────────────┬──────────────────────────────────────────┘
                                   │
                    ┌──────────────┴──────────────────┐
                    │                                 │
                   ▼                                  ▼
┌───────────────────────────────────────┐   ┌─────────────────────────────────┐
│        ENCRYPTED VOLUME (PEFS)        │   │         PUBLIC REPOSITORY       │
│        /opt/terraform.enc/            │   │ /opt/salt/nasqueron-operations/ │
│                                       │   │                                 │
│  Mounted at /opt/terraform/encrypted  │   │  terraform/                     │
│  while tf is running. Sensitive.	│   │    ├── ovh-ops-backups/         │
│                                       │   │    │     ├── main.tf            │
│  /o/t/encrypted/tf-state/<project>/   │   │    │     ├── backend.tf         │
│    └── terraform.tfstate              │   │    │     └── modules/           │
│    └── terraform.tfstate.bak          │   │    │                            │
│───────────────────────────────────────│   │    ├── openbao/                 │
│        WORKING DIRECTORY              │   │    │     ├── main.tf            │
│        /opt/terraform/                │   │    │     └── ...                │
│                                       │   │    │                            │
│  /o/t/tf-data/<project>/ unencrypted  │   │    └── <other-projects>/        │
│    └── .teraform/                     │   │                                 │
│    └── providers/                     │   │  Contains only public code.     │
│    └── modules/                       │   │                                 │
│                                       │   │                                 │
└───────────────────────────────────────┘   └─────────────────────────────────┘
                    │
                    │  (encrypted at rest by PEFS AES-256-XTS)
                    │
                   ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                        BACKUP PIPELINE (Restic)                             │
│                                                                             │
│  Restic /opt/terraform.enc                                                  │
│	│                                                                     │
│      ▼                                                                      │
│  OVH Object Storage (S3-compatible)                                         │
│  • Bucket: nasqueron-backups-amaris                                         │
│  • Region: EU-WEST-PAR                                                      │
│  • Versioning: enabled                                                      │
│  • Object Lock: governance mode, 90-day retention                           │
│  • Per-client prefix isolation via IAM policies                             │
└─────────────────────────────────────────────────────────────────────────────┘

Components

Component	Role	Location
/usr/local/bin/tf-ops	Wrapper script that orchestrates the secure workflow	/usr/local/bin/tf (symlink or alias)
PEFS	AES-256-XTS stacked filesystem for encrypting state at rest	/opt/terraform.enc → mounted at /opt/terraform
Vault	Stores the PEFS passphrase and all infrastructure secrets	Path: ops/infra/complector/terraform/pefs
Terraform / OpenTofu	Infrastructure-as-code engine	Selected per-project by the wrapper
Restic	Incremental, deduplicating backup client	Backs up the raw /opt/terraform.enc directory
OVH Object Storage	S3-compatible remote storage with Object Lock	Buckets: nasqueron-backups-{amaris,darak,vakor}

Security Model

Encryption Layers

The workflow applies defense in depth across three layers:

Layer	Technology	Protects Against
At-rest encryption	PEFS (AES-256-XTS)	Disk theft, unauthorized filesystem access, unencrypted backups
Secret injection	HashiCorp Vault (KV v2)	Plaintext passwords in scripts, config files, or shell history
Remote backup immutability	OVH Object Lock (Governance)	Ransomware deletion of backups, accidental pruning

Secret Flow

Vault KV v2                    PEFS Kernel                    Terraform
(ops/infra/complector/    ──►  (decrypts /opt/terraform.enc   ──►  reads/writes
 terraform/pefs)                in-memory, never touches           encrypted state
                                disk in plaintext)                 inside PEFS mount

The PEFS passphrase is never written to disk. It flows directly from the Vault API into pefs addkey via a Unix pipe (-j - reads from stdin). The plaintext passphrase exists only in process memory for the duration of the addkey system call.

Trust Boundaries

• The public repository contains only declarative infrastructure code. It is safe for Salt to read, for CI to lint, and for developers to clone.
• The PEFS volume contains all sensitive runtime artifacts (state files, provider caches, module downloads). It is encrypted and unmounted by default.
• The Vault server is the single source of truth for the PEFS passphrase. Access is governed by Vault ACL policies.
• The OVH backup stores only the encrypted PEFS volume. The backup provider never sees plaintext state or secrets.

The tf Wrapper Script

Usage

# From any top-level Terraform project directory:
$ cd /opt/salt/nasqueron-operations/terraform/ovh-ops-backups
$ tf init
$ tf plan
$ tf apply -auto-approve
$ tf state list
$ tf output -json

The script must be executed from within a top-level subdirectory of /opt/salt/nasqueron-operations/terraform/. Running it from a nested module directory will result in an error.

Behavior

Step	Action	Failure Handling
1	Validate working directory	Exits with error code 1 if not in a valid project directory
2	Select Terraform or OpenTofu binary	Falls back to tofu unless the project is in the whitelist
3	Mount PEFS if unmounted	Exits with error code 2 if pefs mount fails
4	Load key from Vault if not in kernel	Exits with error code 1 if Vault or pefs addkey fails
5	Create state and cache directories	Uses mkdir -p; idempotent
6	Execute Terraform/OpenTofu	Inherits the child process exit code
7	Unmount PEFS via trap EXIT	Always runs, even on Ctrl+C or script failure

Runtime Selection

The wrapper maintains a whitelist of projects that still require the HashiCorp terraform binary. All other projects default to tofu (OpenTofu).

STILL_REQUIRE_TERRAFORM="openbao ovh-ops-backups"

This allows a gradual migration from Terraform to OpenTofu without changing the developer workflow.

Configuration Reference

Repository Structure

/opt/salt/nasqueron-operations/
└── terraform/
    ├── ovh-ops-backups/          # Uses terraform (whitelisted)
    │   ├── main.tf
    │   ├── backend.tf            # Must point to PEFS state path
    │   └── modules/
    ├── openbao/                  # Uses terraform (whitelisted)
    │   └── ...
    ├── some-other-project/       # Uses tofu (default)
    │   └── ...
    └── ...

PEFS Layout

When mounted, the PEFS volume exposes the following structure:

/opt/terraform/
├── tf-state/
│   ├── ovh-ops-backups/
│   │   ├── terraform.tfstate
│   │   └── terraform.tfstate.backup
│   ├── openbao/
│   │   └── terraform.tfstate
│   └── ...
└── tf-data/
    ├── ovh-ops-backups/
    │   └── .terraform/
    │       ├── providers/
    │       └── modules/
    ├── openbao/
    │   └── .terraform/
    └── ...

Vault Paths

Path	Field	Purpose
ops/infra/complector/terraform/pefs	password	PEFS passphrase used to decrypt /opt/terraform.enc
ops/secrets/backups/ovh/s3/<account>	access_key, secret_key	Per-client S3 credentials for Restic backups

Terraform Backend Configuration

Each project must configure its local backend to write state into the PEFS-mounted directory. Example backend.tf:

terraform {
  backend "local" {
    path = "/opt/terraform/tf-state/ovh-ops-backups/terraform.tfstate"
  }
}

Operations

Initial Setup

1. Create the PEFS encrypted volume:

mkdir -p /opt/terraform.enc
pefs mount /opt/terraform.enc /opt/terraform
pefs addkey -c /opt/terraform
# Store the passphrase in Vault:
vault kv put ops/infra/complector/terraform/pefs password="<your-passphrase>"
pefs umount /opt/terraform

1. Install the wrapper script to /usr/local/bin/tf-ops and create a symlink:

chmod +x /usr/local/bin/tf-ops
ln -s /usr/local/bin/tf-ops /usr/local/bin/tf

1. Ensure Vault CLI is authenticated:

vault login

Daily Usage

$ pm go terraform-ovh-ops-backups   # or: cd /opt/.../terraform/ovh-ops-backups
$ tf init
$ tf plan
$ tf apply

The wrapper handles mounting, key loading, and unmounting transparently. No manual PEFS commands are required.

Backup and Restore

Backups are performed by Restic against the raw encrypted volume:

restic -r s3:https://s3.eu-west-par.io.cloud.ovh.net/nasqueron-backups-amaris \
       backup /opt/terraform.enc

To restore on a new machine:

restic -r s3:https://s3.eu-west-par.io.cloud.ovh.net/nasqueron-backups-amaris \
       restore latest --target /opt/terraform.enc

pefs mount /opt/terraform.enc /opt/terraform
vault kv get -field=password ops/infra/complector/terraform/pefs | \
    pefs addkey -c /opt/terraform -j -

Disaster Recovery

If the Vault server is unavailable, the PEFS volume cannot be unlocked. Ensure Vault itself is backed up and highly available. The PEFS passphrase should also be stored in a secure offline location (e.g., a hardware security module, a sealed envelope in a safe, or an encrypted password manager) as an emergency recovery path.

References

• PEFS — Private Encrypted File System for FreeBSD
• PEFS official documentation
• PEFS Developer Summit 2011 presentation
• Vault — HashiCorp Vault secrets management
• OpenTofu
• Restic backup tool
• OVH Object Storage