Operations grimoire/Deploy with Terraform - Revision history

Dereckson: /* Standard operations */

2026-09-20T20:41:01Z

Standard operations

@@ Line 58: / Line 58: @@
 $ tf apply
 </syntaxhighlight>
 == Specific deployment notes ==

Dereckson: Proofread

2026-09-06T21:08:51Z

Proofread

← Older revision		Revision as of 21:08, 6 September 2026
Line 3:		Line 3:
	In addition to Salt, Nasqueron manages Infrastructure as Code using Terraform and OpenTofu. We deploy from [[Complector]] using <code>/opt/salt/nasqueron-operations</code> as our local copy of rOPS, authoritative for both Salt and Terraform.		In addition to Salt, Nasqueron manages Infrastructure as Code using Terraform and OpenTofu. We deploy from [[Complector]] using <code>/opt/salt/nasqueron-operations</code> as our local copy of rOPS, authoritative for both Salt and Terraform.

	~~It's important to work from there to save~~ a ~~shared~~ Terraform state.		We provide a <code>tf</code> wrapper that automatically runs OpenTofu or Terraform based on the directory and mounts the encrypted state directory on the fly.

	~~We provide a~~ <~~code~~>~~tf</code> wrapper~~ that ~~automatically runs OpenTofu or~~ Terraform ~~based on the directory and mounts the encrypted~~ state ~~directory on the fly~~.		<div class="alert">It's important to work on Complector, so that all deployments use the shared Terraform state.</div>

	=== Paths ===		=== Paths ===
Line 36:		Line 36:
	\| <code>/opt/terraform/tf-data</code>		\| <code>/opt/terraform/tf-data</code>
	\| ZFS Dataset		\| ZFS Dataset
	\| '''Provider cache and modules.''' <code>TF_DATA_DIR</code> target.		\| '''Provider cache and modules files.''' <code>TF_DATA_DIR</code> target.
	\| <code>compression=lz4</code>, <code>recordsize=128K</code> (optimized for Go binaries)		\| <code>compression=lz4</code>, <code>recordsize=128K</code> (optimized for Go binaries)
	\|}		\|}
Line 43:		Line 43:

	* PEFS operations (mounting, adding keys) are executed as the <code>deploy</code> user.		* PEFS operations (mounting, adding keys) are executed as the <code>deploy</code> user.
	* The wrapper verifies the invoking user belongs to the <code>ops</code> or <code>deploy</code> group before proceeding.		* The wrapper verifies that the invoking user belongs to the <code>ops</code> or <code>deploy</code> group before proceeding.
	* The PEFS passphrase ~~to read the Terraform states encryption key~~ is stored in HashiCorp Vault at <code>ops/infra/complector/terraform/pefs</code>.		* The PEFS passphrase is stored in HashiCorp Vault at <code>ops/infra/complector/terraform/pefs</code>.

	== Standard operations ==		== Standard operations ==
Line 78:		Line 78:
	By default, tofu is used.		By default, tofu is used.

	The wrapper maintains an allowlist of ~~project~~ that still require ~~the~~ HashiCorp Terraform ~~versions~~.		The wrapper maintains an allowlist of projects that still require HashiCorp Terraform.

	The main blocker is <code>hashicorp/vault</code>, not compiled for FreeBSD on OpenTofu registry.		The main blocker is <code>hashicorp/vault</code>, not compiled for FreeBSD on OpenTofu registry.

	To add or remove a project ~~there~~, edit the <code>STILL_REQUIRE_TERRAFORM</code> variable in {{Ops file\|roles/salt-primary/opentofu/files/tf.sh}}.		To add or remove a project from this list, edit the <code>STILL_REQUIRE_TERRAFORM</code> variable in {{Ops file\|roles/salt-primary/opentofu/files/tf.sh}}.

	=== New project configuration ===		=== New project configuration ===
Line 115:		Line 115:

	$ cd /opt/salt/nasqueron-operations/terraform/openbao		$ cd /opt/salt/nasqueron-operations/terraform/openbao
	$ tf init # ~~if you've a new entry requiring a module, it needs to be installed~~		$ tf init # install any newly required providers and modules
	$ tf plan		$ tf plan
	$ tf apply		$ tf apply
	</syntaxhighlight>		</syntaxhighlight>

	It could be interesting to maintain an alternative policy to admin for the Terraform provider, restricted to the current actions. Note benefits would be limited as those include policies management.		It could be interesting to maintain an alternative policy to admin for the Terraform provider, restricted to the current actions. Note that the benefits would be limited as those include policies management.

	==== Propagate secrets (DRP) ====		==== Propagate secrets (DRP) ====
Line 130:		Line 130:

	;Full procedure		;Full procedure
	Once the AppRole ~~have~~ been created in Vault, ~~they~~ need to be provisioned to the relevant configuration files.		Once the AppRole has been created in Vault, its credentials need to be provisioned to the relevant configuration files.

	<syntaxhighlight lang="shell">		<syntaxhighlight lang="shell">
Line 144:		Line 144:
	</syntaxhighlight>		</syntaxhighlight>

	<div class="alert">Important. Each time you~~'ll~~ reprovision the secrets, they will change.		<div class="alert">Important: Each time you reprovision the secrets, they will change.

	Don't forget to always apply this full procedure.		Don't forget to always apply this full procedure.
Line 163:		Line 163:
	Vault secrets are provisioned using Terraform (openbao module), and must be propagated to the routers using Salt.		Vault secrets are provisioned using Terraform (openbao module), and must be propagated to the routers using Salt.

	The Vault ~~Secrets~~ may need to be rotated, so we need a simple way to propagate		The Vault secrets may need to be rotated, so we need a simple way to propagate
	updated credentials to routers. This is enabled by the mechanism		updated credentials to routers. This is enabled by the mechanism
	introduced in D4026.		introduced in D4026.
Line 170:		Line 170:
	This configuration is handled by Terraform.		This configuration is handled by Terraform.

	It requires a full access to public cloud project for backups, as the project is fully configured by ~~those~~ Terraform files.		It requires full access to the public cloud project for backups, as the project is fully configured by these Terraform files.

	You'll need to get application key, application secret and ~~consumer_key~~ in your $HOME/.ovh.conf on Complector.		You'll need to get the application key, application secret, and consumer key and write them in your $HOME/.ovh.conf on Complector.
	You can also generate ~~such files~~ with <code>ovhcloud login</code>.		You can also generate the file with <code>ovhcloud login</code>.

	== First Terraform configuration (DRP) ==		== First Terraform configuration (DRP) ==
	=== Initialize Terraform working directories ===		=== Initialize Terraform working directories ===
	Terraform and OpenTofu are currently deployed from Complector.		Terraform and OpenTofu projects are currently deployed from Complector.
	The ~~working directories~~ are encrypted, a wrapper ~~decrypt~~ them for each operation.		The states are encrypted, and a wrapper decrypts them for each operation.

	You need to restore ~~from a backup~~ the ~~content~~ of /opt/terraform.enc or if not available ~~reinitialize~~ a new encrypted directory.		You need to restore the contents of /opt/terraform.enc from a backup, or if it is not available, initialize a new encrypted directory.
	Currently, the procedure is ~~valid for~~ FreeBSD, as it requires PEFS to encrypt ~~working director~~.		Currently, the procedure is specific to FreeBSD, as it requires PEFS to encrypt directories.

	# On the salt primary server, deploy the full roles/salt-primary, or at least the roles/salt-primary/opentofu unit		# On the salt primary server, deploy the full roles/salt-primary, or at least the roles/salt-primary/opentofu unit
Line 188:		Line 188:
	# Run <code>initialize-pefs ops/infra/complector/terraform/pefs /opt/terraform.enc /opt/terraform/encrypted</code>		# Run <code>initialize-pefs ops/infra/complector/terraform/pefs /opt/terraform.enc /opt/terraform/encrypted</code>

	The ~~vault~~ previous version of the secret, if still available, ~~is for~~ the previous directory, keep it if we restore ~~afterhand~~.		The previous version of the Vault secret, if still available, belongs to the previous directory; keep it if you need to restore that directory. However, you need a new password when creating a fresh encrypted directory.
	~~But~~ you need a new password ~~for your new~~ fresh encrypted directory.

	== Troubleshoot ==		== Troubleshoot ==
Line 226:		Line 225:
	</syntaxhighlight>		</syntaxhighlight>

	On OpenTofu, some ~~modules~~ are only compiled for Linux, not for FreeBSD.		On OpenTofu, some providers are only compiled for Linux, not for FreeBSD.
	Switch to Terraform pending a solution to help the OpenTofu builds.		Switch to Terraform pending a solution to help the OpenTofu builds.

Dereckson: Per T2359. Explain the tf wrapper, and the directories where are stored the states/providers. Switch from tofu/terraform to tf command.

2026-09-06T18:54:32Z

Per T2359. Explain the tf wrapper, and the directories where are stored the states/providers. Switch from tofu/terraform to tf command.

Show changes

Dereckson at 13:50, 6 September 2026

2026-09-06T13:50:17Z

← Older revision		Revision as of 13:50, 6 September 2026
Line 95:		Line 95:

	On disk paths are stored in Complector.		On disk paths are stored in Complector.

			== First Terraform configuration (DRP) ==
			=== Initialize Terraform working directories ===
			Terraform and OpenTofu are currently deployed from Complector.
			The working directories are encrypted, a wrapper decrypt them for each operation.

			You need to restore from a backup the content of /opt/terraform.enc or if not available reinitialize a new encrypted directory.
			Currently, the procedure is valid for FreeBSD, as it requires PEFS to encrypt working director.

			# On the salt primary server, deploy the full roles/salt-primary, or at least the roles/salt-primary/opentofu unit
			# Ensure pefs is active with <code>kldstat \| grep pefs</code>
			# Ensure Vault is up, then <code>vault kv put ops/infra/complector/terraform/pefs password=$(openssl rand -hex 48)</code>
			# Run <code>initialize-pefs ops/infra/complector/terraform/pefs /opt/terraform.enc /opt/terraform</code>

			The vault previous version of the secret, if still available, is for the previous directory, keep it if we restore afterhand.
			But you need a new password for your new fresh encrypted directory.

	== Troubleshoot ==		== Troubleshoot ==

Dereckson: /* ovh-ops-backups */ Terraform, not OpenTofu

2026-09-05T19:43:53Z

ovh-ops-backups: Terraform, not OpenTofu

← Older revision		Revision as of 19:43, 5 September 2026
Line 75:		Line 75:

	=== ovh-ops-backups ===		=== ovh-ops-backups ===
	This ~~Terraform~~ configuration is handled by ~~OpenTofu~~.		This configuration is handled by Terraform.

	It requires a full access to public cloud project for backups, as the project is fully configured by those Terraform files.		It requires a full access to public cloud project for backups, as the project is fully configured by those Terraform files.

Dereckson: /* Table of Terraform states */ Terraform too for ovh-ops-backups as we need to store credentials to Vault

2026-09-05T19:43:23Z

Table of Terraform states: Terraform too for ovh-ops-backups as we need to store credentials to Vault

← Older revision		Revision as of 19:43, 5 September 2026
Line 91:		Line 91:
	\| openbao \|\| On disk \|\| /opt/salt/nasqueron-operations/terraform/openbao/terraform.tfstate \|\| Terraform		\| openbao \|\| On disk \|\| /opt/salt/nasqueron-operations/terraform/openbao/terraform.tfstate \|\| Terraform
	\|-		\|-
	\| ovh-ops-backups \|\| On disk \|\| /opt/salt/nasqueron-operations/terraform/ovh-ops-backups/terraform.tfstate \|\| ~~OpenTofu~~		\| ovh-ops-backups \|\| On disk \|\| /opt/salt/nasqueron-operations/terraform/ovh-ops-backups/terraform.tfstate \|\| Terraform
	\|}		\|}

Dereckson: /* Table of Terraform states */ row break

2026-09-05T18:24:14Z

Table of Terraform states: row break

← Older revision		Revision as of 18:24, 5 September 2026
Line 90:		Line 90:
	\|-		\|-
	\| openbao \|\| On disk \|\| /opt/salt/nasqueron-operations/terraform/openbao/terraform.tfstate \|\| Terraform		\| openbao \|\| On disk \|\| /opt/salt/nasqueron-operations/terraform/openbao/terraform.tfstate \|\| Terraform
			\|-
	\| ovh-ops-backups \|\| On disk \|\| /opt/salt/nasqueron-operations/terraform/ovh-ops-backups/terraform.tfstate \|\| OpenTofu		\| ovh-ops-backups \|\| On disk \|\| /opt/salt/nasqueron-operations/terraform/ovh-ops-backups/terraform.tfstate \|\| OpenTofu
	\|}		\|}

Dereckson: /* Specific deployment notes */

2026-09-05T18:19:55Z

Specific deployment notes

← Older revision		Revision as of 18:19, 5 September 2026
Line 56:		Line 56:
	</div>		</div>

	=== CARP OVH secrets usage ===		==== CARP OVH secrets usage ====

	The OVH credentials used by the CARP failover script are stored in Vault at:		The OVH credentials used by the CARP failover script are stored in Vault at:
Line 73:		Line 73:
	updated credentials to routers. This is enabled by the mechanism		updated credentials to routers. This is enabled by the mechanism
	introduced in D4026.		introduced in D4026.

			=== ovh-ops-backups ===
			This Terraform configuration is handled by OpenTofu.

			It requires a full access to public cloud project for backups, as the project is fully configured by those Terraform files.

			You'll need to get application key, application secret and consumer_key in your $HOME/.ovh.conf on Complector.
			You can also generate such files with <code>ovhcloud login</code>.

	== Table of Terraform states ==		== Table of Terraform states ==

Dereckson: /* Table of Terraform states */

2026-09-05T18:04:37Z

Table of Terraform states

← Older revision		Revision as of 18:04, 5 September 2026
Line 77:		Line 77:

	{\| class="wikitable"		{\| class="wikitable"
	\|+ Terraform and ~~OpenBao~~ states		\|+ Terraform and OpenTofu states
	\|-		\|-
	! Configuration !! State back-end !! Path !! Software to use		! Configuration !! State back-end !! Path !! Software to use
	\|-		\|-
	\| openbao \|\| On disk \|\| /opt/salt/nasqueron-operations/terraform/openbao/terraform.tfstate \|\| Terraform		\| openbao \|\| On disk \|\| /opt/salt/nasqueron-operations/terraform/openbao/terraform.tfstate \|\| Terraform
			\| ovh-ops-backups \|\| On disk \|\| /opt/salt/nasqueron-operations/terraform/ovh-ops-backups/terraform.tfstate \|\| OpenTofu
	\|}		\|}

Dereckson: /* Troubleshoot */ Document tofu state rm

2026-07-23T13:19:56Z

Troubleshoot: Document tofu state rm

← Older revision		Revision as of 13:19, 23 July 2026
Line 123:		Line 123:
	On OpenTofu, some modules are only compiled for Linux, not for FreeBSD.		On OpenTofu, some modules are only compiled for Linux, not for FreeBSD.
	Switch to Terraform pending a solution to help the OpenTofu builds.		Switch to Terraform pending a solution to help the OpenTofu builds.

			=== State rollback ===

			When declaring new Terraform resources, it's possible to remove from the state an item, without actually destroying it at infrastructure level:

			$ tofu state rm <state name>
			Removed ….…
			Successfully removed 1 resource instance(s).

	[[Category:Operations grimoire]]		[[Category:Operations grimoire]]
	[[Category:Terraform]]		[[Category:Terraform]]