Operations grimoire/Vault: Difference between revisions
Line 77: | Line 77: | ||
== Troubleshoot == | == Troubleshoot == | ||
=== Connecting to Vault === | === Connecting to Vault === | ||
==== Access to web UI ==== | |||
Vault isn't listening any public IP, but accept connections from internal network, so you can open a SSH tunnel against <code>complector:8200</code> and then browse https://localhost:8200: | |||
<code>ssh -L 8200:complector:8200 router-001.nasqueron.org</code> | |||
==== curl: (60) SSL certificate problem ==== | ==== curl: (60) SSL certificate problem ==== | ||
Curl is compiled against a certificate bundle (<code>curl-config --ca</code> for the path). That bundle doesn't contain the pki_root certificate. We deploy this certificate in /etc/ssl/certs, but curl won't check there if not instructed to. | Curl is compiled against a certificate bundle (<code>curl-config --ca</code> for the path). That bundle doesn't contain the pki_root certificate. We deploy this certificate in /etc/ssl/certs, but curl won't check there if not instructed to. | ||
Line 87: | Line 92: | ||
* with <code>curl -v</code>, the path should correctly there: <code>* CApath: /etc/ssl/certs</code> in addition to a CAfile. | * with <code>curl -v</code>, the path should correctly there: <code>* CApath: /etc/ssl/certs</code> in addition to a CAfile. | ||
* with <code>file /etc/ssl/certs/* | grep -i nasqueron</code> you should have a result ; if not, put the certificate in /usr/local/share/certs and relaunch the tool to link it in /etc/ssl/certs, for example <code>certctl rehash</code> on FreeBSD (needs root access). | * with <code>file /etc/ssl/certs/* | grep -i nasqueron</code> you should have a result ; if not, put the certificate in /usr/local/share/certs and relaunch the tool to link it in /etc/ssl/certs, for example <code>certctl rehash</code> on FreeBSD (needs root access). | ||
=== Certificates === | |||
==== Renew Vault HTTP certificates === | |||
Vault uses TLS certificates managed by its own CA. That can create a chicken/egg issue if certificates have expired to connect to it, as the Vault client strictly requires valid TLS certificated to issue commands to renew certificates. | |||
Connect to the web UI works as your browser is able to bypass certificate requirement. Opening a SSH tunnel with <code>-L 8200:complector:8200</code> should work for that. | |||
Certificates configuration is available at https://devcentral.nasqueron.org/D2639 | |||
== Current status == | == Current status == | ||
* Secrets in pillar/credentials/zr.sls have been migrated to Vault under ops/secrets/<key> | * Secrets in pillar/credentials/zr.sls have been migrated to Vault under ops/secrets/<key> |
Revision as of 09:52, 23 January 2023
In July 2016, we selected Vault to store credentials.
Vault reference
kv engine
The kv engine contains the following paths:
* ops/secrets: credentials like passwords, API tokens, private keys deployed to servers - a reference for machines * ops/internal: credentials to third-party services internally shared amongst ops - a reference for humans * ops/privacy: privacy information
Secrets
Secrets are directly and manually managed in Vault. If we need to run a disaster recovery procedure, we'll roll any secret by defining them again and deploy from rOPS those new secrets.
Privacy data
Personal identity information (PII) shared inside the Nasqueron Operations squad are more convenient to manage as a repository.
Such access to the repository is restricted to commit data, and ensure this data is deployed to the servers needing it. Any other use isn't allowed.
Information to the repository is then published in Vault, and referred in rOPS as Vault credentials.
Note: The ops/privacy path is only intended for PII of current or previous members of the Ops SIG, like IP addresses allowed to connect to restricted resources like the ops VPN. This is not acceptable to put 3rd party information in this repository.
pki engine
Services listening to private IPs use the Vault PKI to generate certificates:
- pki_root: root CA, certificate available at https://api.nasqueron.org/infra/security/pki/root/ca
- pki_vault: intermediate CA used for Vault itself
Certificates should be issued short-term and frequently renewed by automated services.
transit engine
Encryption as a service is available for applications under /transit path.
See https://learn.hashicorp.com/tutorials/vault/eaas-transit.
Applications should use a policy restricting to a subpath for that app, not to the whole engine.
Salt configuration
Get a secret through Salt
The information in the ops/secrets and ops/privacy kv engines are integrated in the Salt pillar:
ext_pillar:
# Credentials to deploy to servers
- vault:
conf: path=ops/secrets
nesting_key: credentials
# Personal identity information from Nasqueron Operations SIG members
- vault:
conf: path=ops/privacy
nesting_key: privacy
You can then access to this information in Salt states using the following references:
Category | Example of path in Vault | Example of pillar key in Salt |
---|---|---|
Secrets | ops/secrets/foo | pillar['credentials']['foo'] |
Privacy data | ops/privacy/ops-cidr | pillar['privacy']['ops-cidr'] |
The ops/internal paths aren't accessible through Vault.
Policies
DRP
Any step we need to do to configure the engines should be added to the rOPS: roles/vault/bootstrap unit.
If we need to recover Vault, two solutions are available:
- restore the storage, to keep data (credentials, certificates including the root CA one)
- recreate it from scratch through the roles/vault/bootstrap unit
Troubleshoot
Connecting to Vault
Access to web UI
Vault isn't listening any public IP, but accept connections from internal network, so you can open a SSH tunnel against complector:8200
and then browse https://localhost:8200:
ssh -L 8200:complector:8200 router-001.nasqueron.org
curl: (60) SSL certificate problem
Curl is compiled against a certificate bundle (curl-config --ca
for the path). That bundle doesn't contain the pki_root certificate. We deploy this certificate in /etc/ssl/certs, but curl won't check there if not instructed to.
If you build an immutable environment like a container, add the certificate to the bundle.
On other machines, create a curl configuration file in ~/.curlrc (it doesn't seem currently possible to create a system one) with capath=/etc/ssl/certs
.
If you've still the issue after that, check two things:
- with
curl -v
, the path should correctly there:* CApath: /etc/ssl/certs
in addition to a CAfile. - with
file /etc/ssl/certs/* | grep -i nasqueron
you should have a result ; if not, put the certificate in /usr/local/share/certs and relaunch the tool to link it in /etc/ssl/certs, for examplecertctl rehash
on FreeBSD (needs root access).
Certificates
= Renew Vault HTTP certificates
Vault uses TLS certificates managed by its own CA. That can create a chicken/egg issue if certificates have expired to connect to it, as the Vault client strictly requires valid TLS certificated to issue commands to renew certificates.
Connect to the web UI works as your browser is able to bypass certificate requirement. Opening a SSH tunnel with -L 8200:complector:8200
should work for that.
Certificates configuration is available at https://devcentral.nasqueron.org/D2639
Current status
- Secrets in pillar/credentials/zr.sls have been migrated to Vault under ops/secrets/<key>