Dev zone/Vault
This page explains how to query vault from an application.
For infrastructure point of view, see Operations grimoire/Vault and Operations grimoire/Eglide/Vault.
Guides
Python with hvac
In Python, the hvac library is recommended.
Step 1 - Connect to Vault
Connection is straightforward and works out of the box even without parameters:
VAULT_CA_CERTIFICATE = "/usr/local/share/certs/nasqueron-vault-ca.crt"
def connect_to_vault():
return hvac.Client(
verify=VAULT_CA_CERTIFICATE,
)
TLS connection needs to be validated with our root certificate.
The exception is when we write operations code allowing us to renew the Vault certificates itself, where we use
verify=False.
For a user script, that's it, the Vault address will be read from the VAULT_ADDR environment variable and a token should be configured by the user itself.
Comprehensive checks for VAULT_ADDR and tokens, with explicit error messages can be found in rOPS: utils/vault/wordpress-provision-secrets.py.
For an application, you can set URL from configuration (pass it with url=) and probably need to login too with credentials. AppRole is encouraged in that case.
# Those values are expected to be read from a configuration file
vault_certificate_path = "/usr/local/share/certs/nasqueron-vault-ca.crt"
vault_url = "https://172.27.27.7:8200"
vault_role_id = "00000000-0000-0000-0000-000000000000"
vault_secret_id = "00000000-0000-0000-0000-000000000000"
vault_client = hvac.Client(
url=vault_url,
verify=vault_certificate_path,
)
vault_client.auth.approle.login(
role_id=vault_role_id,
secret_id=vault_secret_id,
)
Step 2 - Query a kv secret
Most operations with Vault uses the kv2 engine to get secrets as a collection of keys and values.
Secrets are normally located in apps/ for your application or ops/ for infrastructure credentials.
With hvac, best practice is to query them separately if you target only Nasqueron internal infrastructure:
def read_secret(vault_client, mount_point, prefix, key):
secret = vault_client.secrets.kv.read_secret_version(
mount_point=mount_point,
path=prefix + "/" + key,
)
return secret["data"]["data"]
def read_ops_secret(vault_client, key):
return read_secret(vault_client, "ops", "secrets", key)
def read_app_secret(vault_client, key):
return read_secret(vault_client, "apps", "yourappname", key)
For applications to distribute, best is to write a full path starting with apps/ and ops/ secrets in a configuration. You can use the first part of the path as mount point:
def read_secret(vault_client, secret_path):
tokens = secret_path.split("/")
secret_mount = tokens[0]
secret_path = "/".join(tokens[1:])
secret = vault_client.secrets.kv.read_secret_version(
mount_point=secret_mount,
path=secret_path,
)
return secret["data"]["data"]
Best practices
Where to store credentials
Are you setting a credential only for your app as human? Store it to apps/your-app/somepath.
Is the credential to be deployed by Salt? Store it in ops/secrets/nasqueron/your-app/somepath. If you use AppRole, the secret id and role id to provision automatically to your configuration file will be stored there.
To avoid to manage two paths with the same credential and facilitate rotation, it's encouraged to reuse credentials from a database directly from an ops/secrets/dbserver/ path.
Query Vault from Jenkins
If your CI job needs a credential stored in Vault, we've several possibilities:
- get a token from Vault through JWT, an activity to explore as part of T2126 - store a AppRole role-id/secret-id in Jenkins - store a AppRole role-id/secret-id on the server where the test deployment occurs - store directly the secret - refrain to use anything with credentials in CI
From the possibilities, the first is the more promising and should be explored in priority.
Troubleshoot
Access denied
The access can be denied when:
- you're not authenticated
- the resource doesn't exist, or the URL to it has an error
- the policy attached to your Vault credentials don't allow you to query the secret
Check if the URL is correct, try to identify by reading other troubleshoot sections in what case you are, and ask operations assistance to check if the policy is correct and has correctly been deployed.
Access denied when query a full path, without mounting point
Symptom: Permission denied on an URL starting by https://172.27.27.7:8200/v1/secret/data/
Cause: Your library use the default Vault secret engine mount point "secret" instead of "ops" or "apps"
Solution: separate explicitly "ops" or "apps" mount point from the rest of the secret path
____
Vault have secrets engines, each configured to a specific mounting point. We use two points for secrets, ops/, for secrets deployed by Salt, and apps/, for applications.
If you don't specify the mounting point, client libraries like hvac will try to use the default secret/ mount point:
Traceback (most recent call last):
[...]
File "[...]/lib/python3.11/site-packages/hvac/utils.py", line 41, in raise_for_error
raise exceptions.VaultError.from_status(
hvac.exceptions.Forbidden: 1 error occurred:
* permission denied
, on get https://172.27.27.7:8200/v1/secret/data/ops/secrets/dbserver/[...]
That URL has been built from the "secret" default mount point and your ops/secrets/dbserver/... path. Instead, URL to query should have been here /v1/ops/data/secrets/dbserver/...
Example of code above for hvac solve that issue.
TLS - certificate is missing
If you use Vault from your computer through a SSH tunnel the first time, you'll have this TLS error:
Error checking seal status: Get "https://127.0.0.1:8200/v1/sys/seal-status": tls: failed to verify certificate: x509: certificate signed by unknown authority
You need to install the certificate we use for private resources, see Devserver reference#SSH tunnels.
Note: if you see that error on a Nasqueron server, it's a bug to immediately report on DevCentral, rOPS: roles/core/certificates is deployed everywhere with the authority certificate.
TLS - certificate is valid for ..., not localhost
If you set VAULT_ADDR to https://localhost:8200/, you'll get a certificate error as "localhost" isn't added to the certificate:
Error checking seal status: Get "https://localhost:8200/v1/sys/seal-status": tls: failed to verify certificate: x509: certificate is valid for complector.nasqueron.drake, not localhost
You can set VAULT_ADDR at the default value https://127.0.0.1:8200 if you use the tunnel on port 8200, the certificate is valid for both 127.0.0.1 and 172.27.27.7.
TLS - invalid path for certificate
Examples here with hvac target deployment on our FreeBSD servers. If you run the code elsewhere, on your local machine for example, you can encounter this:
OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /usr/local/share/certs/nasqueron-vault-ca.crt
With Python code, it's a bad idea to remove the verify line, as in some distributions, it uses a bundle instead of the system certificates. Probably best to copy the certificate there too on your machine.
For example for Fedora, as root:
$ mkdir -p /usr/local/share/certs
$ cp /etc/pki/ca-trust/source/anchors/nasqueron-vault-ca.crt /usr/local/share/certs/
If you don't have it installed on your system, you can fetch it on operations repository: rOPS: roles/core/certificates/files/nasqueron-vault-ca.crt
Open a task on DevCentral if you need the certificate to be provisioned in that directory on a Nasqueron server. If needed, probably best to put it in /usr/local/share/certs/ everywhere and for the Linux servers, to symlink it to the relevant distribution path.
