Nasqueron Agora - User contributions [en]

Operations grimoire/TLS certificates

2025-05-20T19:38:56Z

DorianWinty: /* improved command */

TLS certificates should be used for every service we provide, so we can encrypt the communications.

== Let's Encrypt ==
=== Certbot commands ===

'''acme-v02 migration.''' If you've a complaint acme-v01.api. isn't available, add <code>--server https://acme-v02.api.letsencrypt.org/directory</code>.

'''T1505 alignment.''' We don't use anymore Certbot as a Docker container on paas-docker role. As such, all paths are now unified. Symlinks have been added to old /srv paths to help transition.

==== Generate a certificate ====
<code>certbot certonly -a webroot --webroot-path=/var/letsencrypt-auto --deploy-hook "service nginx reload" -d foo.nasqueron.org</code>

<code>certbot certonly -a webroot --webroot-path=/var/letsencrypt-auto --deploy-hook "systemctl reload nginx" -d foo.nasqueron.org</code>

==== Generate a certificate for several sites ====
<code>-d foo.nasqueron.org -d bar.nasqueron.org</code>

If a certificate for foo already existed, it will offer to extend it to a new alternative name, which is probably a good idea.

==== Generate a certificate through DNS ====
DNS can be used to generate certificates for domains. For example, the Openfire XMPP certificate is generated like this:

certbot certonly --server https://acme-v02.api.letsencrypt.org/directory --manual --manual-auth-hook /etc/letsencrypt/acme-dns-auth --preferred-challenges dns --debug-challenges -d xmpp.nasqueron.org -d nasqueron.org -d conference.nasqueron.org

From December 2023, you can use this command on FreeBSD servers where {{T|1505}} Let's Encrypt standard installation have been deployed:

certbot certonly --manual --manual-auth-hook /usr/local/etc/letsencrypt/acme-dns-auth --preferred-challenges dns --debug-challenges -d subdomain.nasqueron.org

This uses a specialized DNS server deployed on our Docker PaaS to serve dynamic TLS records under .acme.nasqueron.org, {{Ops file|roles/paas-docker/containers/acme_dns.sls}}.

Support files were previously '''only deployed to paas-docker role''' through {{Ops file|roles/paas-docker/letsencrypt/files}}.

==== Renew all certificates ====
<code>certbot renew</code>

==== Installation on nginx ====
===== Allow Let's encrypt validation =====

include includes/letsencrypt;

This will use {{Ops file|roles/webserver-core/nginx/files/includes/letsencrypt}} nginx configuration.

===== Serve TLS certificate =====

include includes/tls;
ssl_certificate /srv/letsencrypt/etc/live/xmpp.nasqueron.org/fullchain.pem;
ssl_certificate_key /srv/letsencrypt/etc/live/xmpp.nasqueron.org/privkey.pem;

This will configure a compromise between security and compatibility, based on Intermediate Mozilla SSL config<ref>https://ssl-config.mozilla.org/</ref>. The current configuration is served by {{Ops file|roles/webserver-core/nginx/files/includes/tls}}.

If you prefer to restrict the resource for TLS 1.3, and accepts to block legacy clients, you can also use with {{D|3251}}:

include includes/tls-modern-only;
ssl_certificate /srv/letsencrypt/etc/live/xmpp.nasqueron.org/fullchain.pem;
ssl_certificate_key /srv/letsencrypt/etc/live/xmpp.nasqueron.org/privkey.pem;

==== Edit renewal hook ====
In /etc/letsencrypt/renewal or /usr/local/etc/letsencrypt/renewal you can edit this section to customize the command to run after renewal:

[renewalparams]
renew_hook = systemctl reload nginx

That can be applied in batch when nothing is set: <code><nowiki>gsed -i "s/\[\[webroot_map]]/renew_hook = systemctl reload nginx\n\n[[webroot_map]]/g" *.conf</nowiki></code>

=== Install certbot on CentOS 10 stream ===

Packages for Python 3.12 currently used as of 2024-08-04 on CentOS 10 stream can be fetched from Fedora 40 repository.

Installation on Dwellers was done like this:

mkdir /tmp/certbot && cd /tmp/certbot
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/c/certbot-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-acme-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-certbot-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-configargparse-1.7-3.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-josepy-1.13.0-8.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-parsedatetime-2.6-12.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pyOpenSSL-23.2.0-3.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pyrfc3339-1.1-18.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pytz-2024.1-1.fc40.noarch.rpm
dnf install ./*.rpm

=== acme.sh ===

acme.sh should be run with the called "acme" with a
sudo su - acme

==== Generate a certificate ====

==== Generate a certificate for several sites ====

==== Generate a certificate through DNS ====

For DNS certificate, we need to start the certificate generation, edit the DNS records to add the certificate verification. And launch the end of the certificate generation
export ACMEDNS_BASE_URL="https://acme.nasqueron.org"
acme.sh --issue --dns dns_acmedns -d example.com --server letsencrypt

==== Regenerate certificate how are from certbot ====

export ACMEDNS_BASE_URL="https://acme.nasqueron.org"
export ACMEDNS_USERNAME="<username>"
export ACMEDNS_PASSWORD="<password>"
export ACMEDNS_SUBDOMAIN="<subdomain>"

acme.sh --issue --dns dns_acmedns -d example.com --server letsencrypt

==== Renew all certificates ====

For the certificates renewal, anything is needed the cron tab will try to renew them every day "x" day before the expiracy

==== Deploying of the certificates ====

acme.sh --install-cert -d exemple.nasqueron.org \
--cert-file /var/certificates/exemple.nasqueron.org/cert.pem \
--key-file /var/certificates/exemple.nasqueron.org/key.pem \
--fullchain-file /var/certificates/exemple.nasqueron.org/fullchain.pem \
--ca-file /var/certificates/exemple.nasqueron.org/chain.pem

===== improved command =====

sudo su acme

export domain="exemple.nasqueron.org"

mkdir /var/certificates/$domain

acme.sh --install-cert -d $domain \
--cert-file /var/certificates/$domain/cert.pem \
--key-file /var/certificates/$domain/key.pem \
--ca-file /var/certificates/$domain/chain.pem \
--fullchain-file /var/certificates/$domain/fullchain.pem

==== on nginx ====

Le_ReloadCmd='sudo acmesh-nginxCheck'

doit être mis dans le fichier de configuration pour le certificat ou ajouter la commande

== Nasqueron PKI ==
=== Principles ===
Internal PKI material are stored in Vault, see [[Operations grimoire/Vault]] for information about how to generate, renew, etc.

In the operations repository, the {{Ops file|roles/core/certificates}} unit is responsible to deploy <code>nasqueron-vault-ca.crt</code>, used to authenticate for *.nasqueron.drake or IP services. For exemple, Vault clients require a certificate they can validate with a chain from that intermediate CA. Note as we don't currently use the root CA, a fullchain is probably not needed for those services.

=== Rollover a new certificate ===
If <code>nasqueron-vault-ca.crt</code> is updated or if we want instead to provision the root certificate, the following repository need to be updated and services to be redeployed:

{| class="wikitable sortable"
|+ Where to update certificate?
|-
! Repository !! Path !! Purpose !! Deployment instructions
|-
| operations || {{Ops file|roles/core/certificates/files/}} || Trust internal resources on every server || Salt: <code>salt '*' state.apply roles/core/certificates</code>
|-
| docker-airflow || [https://devcentral.nasqueron.org/source/docker-airflow/browse/main/files files/] || Connect to Vault from Airflow Python code || Docker: build and deploy new image nasqueron/airflow on Dwellers
|}

Deployment instructions are up-to-date when newly added on that table. Afterwards, they should be checked against current procedures.

== Special considerations ==
=== New server ===
Let's encrypt client is available on [[Ysul]] (natively) and [[Dwellers]] (as a wrapper script for a Docker container).

Fill a task in Servers component, subscribe Sandlayth and Dereckson to deploy it on a new server.

A salt state would be nice for such purpose. There is work-in-progress on that matter through {{D|3248}}.

=== Internationalized domain names ===
==== Punycode conversion ====
Both for web server configuration and certificate authority, name must be converted to Punycode ([https://www.ietf.org/rfc/rfc3492.txt RFC 3492]):
https://www.punycoder.com/

==== Let's encrypt support ====

Let's encrypt has supported IDN since 2016<ref>https://letsencrypt.org/2016/10/21/introducing-idn-support.html</ref>. We use it for dægrefn.nasqueron.org certificate.

Previously, they were afraid: attackers could register a domain with a Cyrillic character matching a real domains. As some people consider it's the responsibility of the CA to mitigate such risks, the feature has been several times postponed.

==== StartSSL ====

StartSSL is not in activity anymore. It was used at Nasqueron when Let's Encrypt didn't support IDN.

== Troubleshoot ==
=== Acme DNS ===
==== Can't reach ACME DNS API / HTTP Error 403: Forbidden ====
AcmeDNS API access is restricted to IPs addresses set in {{Ops file|roles/webserver-core/nginx/files/includes/geo_nasqueron}}.

If you got a 403 by running acme.sh (only visible in --debug mode):
* If the IP address is missing from geo_nasqueron, add it
* If the IP address is already there, check on the server if /etc/nginx/includes/geo_nasqueron matches
** Yes -> nginx must be reloaded (nginx -t reload)
** No -> Deploy with <code>salt docker-002 state.sls_id /etc/nginx/includes/geo_nasqueron roles/webserver-core/nginx</code>

Another solution is a specific routing to reach the API:
For Hervil, add a route to use router-001 gateway: route add 51.255.124.8/30 172.27.27.1

==== Troubleshoot records in ACME DNS database ====
To check the acme.nasqueron.org subdomain matching your _acme-challenge subdomain: <code>nslookup -type=TXT _acme-challenge.<subdomain>.nasqueron.org</code>

The DNS server to store acme.nasqueron.org records uses a sqlite3 database. To connect and check your parameters:

$ ssh docker-002
$ sqlite3 /srv/acme/lib/acme-dns.db
SELECT Username, Password FROM records WHERE Subdomain = "<DNS answer>"

==== Credential lost for a ACME DNS subdomain ====

Passwords are [https://github.com/joohoi/acme-dns/blob/9c6ca258e1d57e7441b60db4474a68c36356dae2/validation.go hashed with bcrypt], so it's not possible to recover them.

In that case, there are two options:

* if you prefer edit the DNS record, delete acme.sh information for the domain renewal then issue a new certificate, that will register a new account
* if you prefer edit the ACME DNS database, encrypt a new password in bcrypt and update it with sqlite

Certificates requests are limited by Let's Encrypt, updating the password avoids to reach that quota.

You can generate a uuid and pass it to bcrypt, or register a new password (and update DNS):

$ ssh docker-002
$ export SQLITE_HISTORY=/dev/null
$ sqlite3 /srv/acme/lib/acme-dns.db
UPDATE records SET Password = "<new bcrypt hash>" WHERE Subdomain = "<DNS answer>"

You can generate any arbitrary password with any random tool.
It seems UUIDv4 is used by the client code, if you want one, you can use <code>uuidgen -r</code>.

To get password hash with bcrypt, you can use psysh on WindRiver:

$ psysh
password_hash("alpha", PASSWORD_BCRYPT)
$2y$10$N17lnt09YFIoVY4025AmCuGrlmA.ZiThm.RDufcakGq9.3IK4xVcW

Replace $2y$ by $2a$ as x and y are unique for PHP, the rest of the world uses $2a$.

=== Migration to acme.sh ===
==== Paths to update ====

Paths from /usr/local/etc/letsencrypt or /etc/letsencrypt (OS-dependant path) needs to be updated to /var/certificates (cross-platform path). The private key file is only `key.pem` instead of `privkey.pem`.

An example is available on https://devcentral.nasqueron.org/D3624

== Notes & references ==
<references />

Operations grimoire/TLS certificates

2025-05-20T19:38:09Z

DorianWinty: /* improved command */

TLS certificates should be used for every service we provide, so we can encrypt the communications.

== Let's Encrypt ==
=== Certbot commands ===

'''acme-v02 migration.''' If you've a complaint acme-v01.api. isn't available, add <code>--server https://acme-v02.api.letsencrypt.org/directory</code>.

'''T1505 alignment.''' We don't use anymore Certbot as a Docker container on paas-docker role. As such, all paths are now unified. Symlinks have been added to old /srv paths to help transition.

==== Generate a certificate ====
<code>certbot certonly -a webroot --webroot-path=/var/letsencrypt-auto --deploy-hook "service nginx reload" -d foo.nasqueron.org</code>

<code>certbot certonly -a webroot --webroot-path=/var/letsencrypt-auto --deploy-hook "systemctl reload nginx" -d foo.nasqueron.org</code>

==== Generate a certificate for several sites ====
<code>-d foo.nasqueron.org -d bar.nasqueron.org</code>

If a certificate for foo already existed, it will offer to extend it to a new alternative name, which is probably a good idea.

==== Generate a certificate through DNS ====
DNS can be used to generate certificates for domains. For example, the Openfire XMPP certificate is generated like this:

certbot certonly --server https://acme-v02.api.letsencrypt.org/directory --manual --manual-auth-hook /etc/letsencrypt/acme-dns-auth --preferred-challenges dns --debug-challenges -d xmpp.nasqueron.org -d nasqueron.org -d conference.nasqueron.org

From December 2023, you can use this command on FreeBSD servers where {{T|1505}} Let's Encrypt standard installation have been deployed:

certbot certonly --manual --manual-auth-hook /usr/local/etc/letsencrypt/acme-dns-auth --preferred-challenges dns --debug-challenges -d subdomain.nasqueron.org

This uses a specialized DNS server deployed on our Docker PaaS to serve dynamic TLS records under .acme.nasqueron.org, {{Ops file|roles/paas-docker/containers/acme_dns.sls}}.

Support files were previously '''only deployed to paas-docker role''' through {{Ops file|roles/paas-docker/letsencrypt/files}}.

==== Renew all certificates ====
<code>certbot renew</code>

==== Installation on nginx ====
===== Allow Let's encrypt validation =====

include includes/letsencrypt;

This will use {{Ops file|roles/webserver-core/nginx/files/includes/letsencrypt}} nginx configuration.

===== Serve TLS certificate =====

include includes/tls;
ssl_certificate /srv/letsencrypt/etc/live/xmpp.nasqueron.org/fullchain.pem;
ssl_certificate_key /srv/letsencrypt/etc/live/xmpp.nasqueron.org/privkey.pem;

This will configure a compromise between security and compatibility, based on Intermediate Mozilla SSL config<ref>https://ssl-config.mozilla.org/</ref>. The current configuration is served by {{Ops file|roles/webserver-core/nginx/files/includes/tls}}.

If you prefer to restrict the resource for TLS 1.3, and accepts to block legacy clients, you can also use with {{D|3251}}:

include includes/tls-modern-only;
ssl_certificate /srv/letsencrypt/etc/live/xmpp.nasqueron.org/fullchain.pem;
ssl_certificate_key /srv/letsencrypt/etc/live/xmpp.nasqueron.org/privkey.pem;

==== Edit renewal hook ====
In /etc/letsencrypt/renewal or /usr/local/etc/letsencrypt/renewal you can edit this section to customize the command to run after renewal:

[renewalparams]
renew_hook = systemctl reload nginx

That can be applied in batch when nothing is set: <code><nowiki>gsed -i "s/\[\[webroot_map]]/renew_hook = systemctl reload nginx\n\n[[webroot_map]]/g" *.conf</nowiki></code>

=== Install certbot on CentOS 10 stream ===

Packages for Python 3.12 currently used as of 2024-08-04 on CentOS 10 stream can be fetched from Fedora 40 repository.

Installation on Dwellers was done like this:

mkdir /tmp/certbot && cd /tmp/certbot
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/c/certbot-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-acme-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-certbot-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-configargparse-1.7-3.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-josepy-1.13.0-8.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-parsedatetime-2.6-12.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pyOpenSSL-23.2.0-3.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pyrfc3339-1.1-18.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pytz-2024.1-1.fc40.noarch.rpm
dnf install ./*.rpm

=== acme.sh ===

acme.sh should be run with the called "acme" with a
sudo su - acme

==== Generate a certificate ====

==== Generate a certificate for several sites ====

==== Generate a certificate through DNS ====

For DNS certificate, we need to start the certificate generation, edit the DNS records to add the certificate verification. And launch the end of the certificate generation
export ACMEDNS_BASE_URL="https://acme.nasqueron.org"
acme.sh --issue --dns dns_acmedns -d example.com --server letsencrypt

==== Regenerate certificate how are from certbot ====

export ACMEDNS_BASE_URL="https://acme.nasqueron.org"
export ACMEDNS_USERNAME="<username>"
export ACMEDNS_PASSWORD="<password>"
export ACMEDNS_SUBDOMAIN="<subdomain>"

acme.sh --issue --dns dns_acmedns -d example.com --server letsencrypt

==== Renew all certificates ====

For the certificates renewal, anything is needed the cron tab will try to renew them every day "x" day before the expiracy

==== Deploying of the certificates ====

acme.sh --install-cert -d exemple.nasqueron.org \
--cert-file /var/certificates/exemple.nasqueron.org/cert.pem \
--key-file /var/certificates/exemple.nasqueron.org/key.pem \
--fullchain-file /var/certificates/exemple.nasqueron.org/fullchain.pem \
--ca-file /var/certificates/exemple.nasqueron.org/chain.pem

===== improved command =====

sudo su acme

export domain="exemple.nasqueron.org"

mkdir /var/certificates/$domain

acme.sh --install-cert -d $domain \
--cert-file /var/certificates/$domain/cert.pem \
--key-file /var/certificates/$domain/key.pem \
--fullchain-file /var/certificates/$domain/fullchain.pem \
--ca-file /var/certificates/$domain/chain.pem

==== on nginx ====

Le_ReloadCmd='sudo acmesh-nginxCheck'

doit être mis dans le fichier de configuration pour le certificat ou ajouter la commande

== Nasqueron PKI ==
=== Principles ===
Internal PKI material are stored in Vault, see [[Operations grimoire/Vault]] for information about how to generate, renew, etc.

In the operations repository, the {{Ops file|roles/core/certificates}} unit is responsible to deploy <code>nasqueron-vault-ca.crt</code>, used to authenticate for *.nasqueron.drake or IP services. For exemple, Vault clients require a certificate they can validate with a chain from that intermediate CA. Note as we don't currently use the root CA, a fullchain is probably not needed for those services.

=== Rollover a new certificate ===
If <code>nasqueron-vault-ca.crt</code> is updated or if we want instead to provision the root certificate, the following repository need to be updated and services to be redeployed:

{| class="wikitable sortable"
|+ Where to update certificate?
|-
! Repository !! Path !! Purpose !! Deployment instructions
|-
| operations || {{Ops file|roles/core/certificates/files/}} || Trust internal resources on every server || Salt: <code>salt '*' state.apply roles/core/certificates</code>
|-
| docker-airflow || [https://devcentral.nasqueron.org/source/docker-airflow/browse/main/files files/] || Connect to Vault from Airflow Python code || Docker: build and deploy new image nasqueron/airflow on Dwellers
|}

Deployment instructions are up-to-date when newly added on that table. Afterwards, they should be checked against current procedures.

== Special considerations ==
=== New server ===
Let's encrypt client is available on [[Ysul]] (natively) and [[Dwellers]] (as a wrapper script for a Docker container).

Fill a task in Servers component, subscribe Sandlayth and Dereckson to deploy it on a new server.

A salt state would be nice for such purpose. There is work-in-progress on that matter through {{D|3248}}.

=== Internationalized domain names ===
==== Punycode conversion ====
Both for web server configuration and certificate authority, name must be converted to Punycode ([https://www.ietf.org/rfc/rfc3492.txt RFC 3492]):
https://www.punycoder.com/

==== Let's encrypt support ====

Let's encrypt has supported IDN since 2016<ref>https://letsencrypt.org/2016/10/21/introducing-idn-support.html</ref>. We use it for dægrefn.nasqueron.org certificate.

Previously, they were afraid: attackers could register a domain with a Cyrillic character matching a real domains. As some people consider it's the responsibility of the CA to mitigate such risks, the feature has been several times postponed.

==== StartSSL ====

StartSSL is not in activity anymore. It was used at Nasqueron when Let's Encrypt didn't support IDN.

== Troubleshoot ==
=== Acme DNS ===
==== Can't reach ACME DNS API / HTTP Error 403: Forbidden ====
AcmeDNS API access is restricted to IPs addresses set in {{Ops file|roles/webserver-core/nginx/files/includes/geo_nasqueron}}.

If you got a 403 by running acme.sh (only visible in --debug mode):
* If the IP address is missing from geo_nasqueron, add it
* If the IP address is already there, check on the server if /etc/nginx/includes/geo_nasqueron matches
** Yes -> nginx must be reloaded (nginx -t reload)
** No -> Deploy with <code>salt docker-002 state.sls_id /etc/nginx/includes/geo_nasqueron roles/webserver-core/nginx</code>

Another solution is a specific routing to reach the API:
For Hervil, add a route to use router-001 gateway: route add 51.255.124.8/30 172.27.27.1

==== Troubleshoot records in ACME DNS database ====
To check the acme.nasqueron.org subdomain matching your _acme-challenge subdomain: <code>nslookup -type=TXT _acme-challenge.<subdomain>.nasqueron.org</code>

The DNS server to store acme.nasqueron.org records uses a sqlite3 database. To connect and check your parameters:

$ ssh docker-002
$ sqlite3 /srv/acme/lib/acme-dns.db
SELECT Username, Password FROM records WHERE Subdomain = "<DNS answer>"

==== Credential lost for a ACME DNS subdomain ====

Passwords are [https://github.com/joohoi/acme-dns/blob/9c6ca258e1d57e7441b60db4474a68c36356dae2/validation.go hashed with bcrypt], so it's not possible to recover them.

In that case, there are two options:

* if you prefer edit the DNS record, delete acme.sh information for the domain renewal then issue a new certificate, that will register a new account
* if you prefer edit the ACME DNS database, encrypt a new password in bcrypt and update it with sqlite

Certificates requests are limited by Let's Encrypt, updating the password avoids to reach that quota.

You can generate a uuid and pass it to bcrypt, or register a new password (and update DNS):

$ ssh docker-002
$ export SQLITE_HISTORY=/dev/null
$ sqlite3 /srv/acme/lib/acme-dns.db
UPDATE records SET Password = "<new bcrypt hash>" WHERE Subdomain = "<DNS answer>"

You can generate any arbitrary password with any random tool.
It seems UUIDv4 is used by the client code, if you want one, you can use <code>uuidgen -r</code>.

To get password hash with bcrypt, you can use psysh on WindRiver:

$ psysh
password_hash("alpha", PASSWORD_BCRYPT)
$2y$10$N17lnt09YFIoVY4025AmCuGrlmA.ZiThm.RDufcakGq9.3IK4xVcW

Replace $2y$ by $2a$ as x and y are unique for PHP, the rest of the world uses $2a$.

=== Migration to acme.sh ===
==== Paths to update ====

Paths from /usr/local/etc/letsencrypt or /etc/letsencrypt (OS-dependant path) needs to be updated to /var/certificates (cross-platform path). The private key file is only `key.pem` instead of `privkey.pem`.

An example is available on https://devcentral.nasqueron.org/D3624

== Notes & references ==
<references />

Operations grimoire/TLS certificates

2025-05-20T19:35:09Z

DorianWinty: /* Deploying of the certificates */

TLS certificates should be used for every service we provide, so we can encrypt the communications.

== Let's Encrypt ==
=== Certbot commands ===

'''acme-v02 migration.''' If you've a complaint acme-v01.api. isn't available, add <code>--server https://acme-v02.api.letsencrypt.org/directory</code>.

'''T1505 alignment.''' We don't use anymore Certbot as a Docker container on paas-docker role. As such, all paths are now unified. Symlinks have been added to old /srv paths to help transition.

==== Generate a certificate ====
<code>certbot certonly -a webroot --webroot-path=/var/letsencrypt-auto --deploy-hook "service nginx reload" -d foo.nasqueron.org</code>

<code>certbot certonly -a webroot --webroot-path=/var/letsencrypt-auto --deploy-hook "systemctl reload nginx" -d foo.nasqueron.org</code>

==== Generate a certificate for several sites ====
<code>-d foo.nasqueron.org -d bar.nasqueron.org</code>

If a certificate for foo already existed, it will offer to extend it to a new alternative name, which is probably a good idea.

==== Generate a certificate through DNS ====
DNS can be used to generate certificates for domains. For example, the Openfire XMPP certificate is generated like this:

certbot certonly --server https://acme-v02.api.letsencrypt.org/directory --manual --manual-auth-hook /etc/letsencrypt/acme-dns-auth --preferred-challenges dns --debug-challenges -d xmpp.nasqueron.org -d nasqueron.org -d conference.nasqueron.org

From December 2023, you can use this command on FreeBSD servers where {{T|1505}} Let's Encrypt standard installation have been deployed:

certbot certonly --manual --manual-auth-hook /usr/local/etc/letsencrypt/acme-dns-auth --preferred-challenges dns --debug-challenges -d subdomain.nasqueron.org

This uses a specialized DNS server deployed on our Docker PaaS to serve dynamic TLS records under .acme.nasqueron.org, {{Ops file|roles/paas-docker/containers/acme_dns.sls}}.

Support files were previously '''only deployed to paas-docker role''' through {{Ops file|roles/paas-docker/letsencrypt/files}}.

==== Renew all certificates ====
<code>certbot renew</code>

==== Installation on nginx ====
===== Allow Let's encrypt validation =====

include includes/letsencrypt;

This will use {{Ops file|roles/webserver-core/nginx/files/includes/letsencrypt}} nginx configuration.

===== Serve TLS certificate =====

include includes/tls;
ssl_certificate /srv/letsencrypt/etc/live/xmpp.nasqueron.org/fullchain.pem;
ssl_certificate_key /srv/letsencrypt/etc/live/xmpp.nasqueron.org/privkey.pem;

This will configure a compromise between security and compatibility, based on Intermediate Mozilla SSL config<ref>https://ssl-config.mozilla.org/</ref>. The current configuration is served by {{Ops file|roles/webserver-core/nginx/files/includes/tls}}.

If you prefer to restrict the resource for TLS 1.3, and accepts to block legacy clients, you can also use with {{D|3251}}:

include includes/tls-modern-only;
ssl_certificate /srv/letsencrypt/etc/live/xmpp.nasqueron.org/fullchain.pem;
ssl_certificate_key /srv/letsencrypt/etc/live/xmpp.nasqueron.org/privkey.pem;

==== Edit renewal hook ====
In /etc/letsencrypt/renewal or /usr/local/etc/letsencrypt/renewal you can edit this section to customize the command to run after renewal:

[renewalparams]
renew_hook = systemctl reload nginx

That can be applied in batch when nothing is set: <code><nowiki>gsed -i "s/\[\[webroot_map]]/renew_hook = systemctl reload nginx\n\n[[webroot_map]]/g" *.conf</nowiki></code>

=== Install certbot on CentOS 10 stream ===

Packages for Python 3.12 currently used as of 2024-08-04 on CentOS 10 stream can be fetched from Fedora 40 repository.

Installation on Dwellers was done like this:

mkdir /tmp/certbot && cd /tmp/certbot
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/c/certbot-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-acme-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-certbot-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-configargparse-1.7-3.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-josepy-1.13.0-8.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-parsedatetime-2.6-12.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pyOpenSSL-23.2.0-3.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pyrfc3339-1.1-18.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pytz-2024.1-1.fc40.noarch.rpm
dnf install ./*.rpm

=== acme.sh ===

acme.sh should be run with the called "acme" with a
sudo su - acme

==== Generate a certificate ====

==== Generate a certificate for several sites ====

==== Generate a certificate through DNS ====

For DNS certificate, we need to start the certificate generation, edit the DNS records to add the certificate verification. And launch the end of the certificate generation
export ACMEDNS_BASE_URL="https://acme.nasqueron.org"
acme.sh --issue --dns dns_acmedns -d example.com --server letsencrypt

==== Regenerate certificate how are from certbot ====

export ACMEDNS_BASE_URL="https://acme.nasqueron.org"
export ACMEDNS_USERNAME="<username>"
export ACMEDNS_PASSWORD="<password>"
export ACMEDNS_SUBDOMAIN="<subdomain>"

acme.sh --issue --dns dns_acmedns -d example.com --server letsencrypt

==== Renew all certificates ====

For the certificates renewal, anything is needed the cron tab will try to renew them every day "x" day before the expiracy

==== Deploying of the certificates ====

acme.sh --install-cert -d exemple.nasqueron.org \
--cert-file /var/certificates/exemple.nasqueron.org/cert.pem \
--key-file /var/certificates/exemple.nasqueron.org/key.pem \
--fullchain-file /var/certificates/exemple.nasqueron.org/fullchain.pem \
--ca-file /var/certificates/exemple.nasqueron.org/chain.pem

===== improved command =====

sudo su acme

export domain="exemple.nasqueron.org"

mkdir /var/certificates/$domain

acme.sh --install-cert -d $domain \
--cert-file /var/certificates/$domain/cert.pem \
--key-file /var/certificates/$domain/key.pem \
--fullchain-file /var/certificates/$domain/fullchain.pem
--ca-file /var/certificates/$domain/chain.pem

==== on nginx ====

Le_ReloadCmd='sudo acmesh-nginxCheck'

doit être mis dans le fichier de configuration pour le certificat ou ajouter la commande

== Nasqueron PKI ==
=== Principles ===
Internal PKI material are stored in Vault, see [[Operations grimoire/Vault]] for information about how to generate, renew, etc.

In the operations repository, the {{Ops file|roles/core/certificates}} unit is responsible to deploy <code>nasqueron-vault-ca.crt</code>, used to authenticate for *.nasqueron.drake or IP services. For exemple, Vault clients require a certificate they can validate with a chain from that intermediate CA. Note as we don't currently use the root CA, a fullchain is probably not needed for those services.

=== Rollover a new certificate ===
If <code>nasqueron-vault-ca.crt</code> is updated or if we want instead to provision the root certificate, the following repository need to be updated and services to be redeployed:

{| class="wikitable sortable"
|+ Where to update certificate?
|-
! Repository !! Path !! Purpose !! Deployment instructions
|-
| operations || {{Ops file|roles/core/certificates/files/}} || Trust internal resources on every server || Salt: <code>salt '*' state.apply roles/core/certificates</code>
|-
| docker-airflow || [https://devcentral.nasqueron.org/source/docker-airflow/browse/main/files files/] || Connect to Vault from Airflow Python code || Docker: build and deploy new image nasqueron/airflow on Dwellers
|}

Deployment instructions are up-to-date when newly added on that table. Afterwards, they should be checked against current procedures.

== Special considerations ==
=== New server ===
Let's encrypt client is available on [[Ysul]] (natively) and [[Dwellers]] (as a wrapper script for a Docker container).

Fill a task in Servers component, subscribe Sandlayth and Dereckson to deploy it on a new server.

A salt state would be nice for such purpose. There is work-in-progress on that matter through {{D|3248}}.

=== Internationalized domain names ===
==== Punycode conversion ====
Both for web server configuration and certificate authority, name must be converted to Punycode ([https://www.ietf.org/rfc/rfc3492.txt RFC 3492]):
https://www.punycoder.com/

==== Let's encrypt support ====

Let's encrypt has supported IDN since 2016<ref>https://letsencrypt.org/2016/10/21/introducing-idn-support.html</ref>. We use it for dægrefn.nasqueron.org certificate.

Previously, they were afraid: attackers could register a domain with a Cyrillic character matching a real domains. As some people consider it's the responsibility of the CA to mitigate such risks, the feature has been several times postponed.

==== StartSSL ====

StartSSL is not in activity anymore. It was used at Nasqueron when Let's Encrypt didn't support IDN.

== Troubleshoot ==
=== Acme DNS ===
==== Can't reach ACME DNS API / HTTP Error 403: Forbidden ====
AcmeDNS API access is restricted to IPs addresses set in {{Ops file|roles/webserver-core/nginx/files/includes/geo_nasqueron}}.

If you got a 403 by running acme.sh (only visible in --debug mode):
* If the IP address is missing from geo_nasqueron, add it
* If the IP address is already there, check on the server if /etc/nginx/includes/geo_nasqueron matches
** Yes -> nginx must be reloaded (nginx -t reload)
** No -> Deploy with <code>salt docker-002 state.sls_id /etc/nginx/includes/geo_nasqueron roles/webserver-core/nginx</code>

Another solution is a specific routing to reach the API:
For Hervil, add a route to use router-001 gateway: route add 51.255.124.8/30 172.27.27.1

==== Troubleshoot records in ACME DNS database ====
To check the acme.nasqueron.org subdomain matching your _acme-challenge subdomain: <code>nslookup -type=TXT _acme-challenge.<subdomain>.nasqueron.org</code>

The DNS server to store acme.nasqueron.org records uses a sqlite3 database. To connect and check your parameters:

$ ssh docker-002
$ sqlite3 /srv/acme/lib/acme-dns.db
SELECT Username, Password FROM records WHERE Subdomain = "<DNS answer>"

==== Credential lost for a ACME DNS subdomain ====

Passwords are [https://github.com/joohoi/acme-dns/blob/9c6ca258e1d57e7441b60db4474a68c36356dae2/validation.go hashed with bcrypt], so it's not possible to recover them.

In that case, there are two options:

* if you prefer edit the DNS record, delete acme.sh information for the domain renewal then issue a new certificate, that will register a new account
* if you prefer edit the ACME DNS database, encrypt a new password in bcrypt and update it with sqlite

Certificates requests are limited by Let's Encrypt, updating the password avoids to reach that quota.

You can generate a uuid and pass it to bcrypt, or register a new password (and update DNS):

$ ssh docker-002
$ export SQLITE_HISTORY=/dev/null
$ sqlite3 /srv/acme/lib/acme-dns.db
UPDATE records SET Password = "<new bcrypt hash>" WHERE Subdomain = "<DNS answer>"

You can generate any arbitrary password with any random tool.
It seems UUIDv4 is used by the client code, if you want one, you can use <code>uuidgen -r</code>.

To get password hash with bcrypt, you can use psysh on WindRiver:

$ psysh
password_hash("alpha", PASSWORD_BCRYPT)
$2y$10$N17lnt09YFIoVY4025AmCuGrlmA.ZiThm.RDufcakGq9.3IK4xVcW

Replace $2y$ by $2a$ as x and y are unique for PHP, the rest of the world uses $2a$.

=== Migration to acme.sh ===
==== Paths to update ====

Paths from /usr/local/etc/letsencrypt or /etc/letsencrypt (OS-dependant path) needs to be updated to /var/certificates (cross-platform path). The private key file is only `key.pem` instead of `privkey.pem`.

An example is available on https://devcentral.nasqueron.org/D3624

== Notes & references ==
<references />

Operations grimoire/TLS certificates

2025-03-12T21:25:27Z

DorianWinty: /* on nginx */

TLS certificates should be used for every service we provide, so we can encrypt the communications.

== Let's Encrypt ==
=== Certbot commands ===

'''acme-v02 migration.''' If you've a complaint acme-v01.api. isn't available, add <code>--server https://acme-v02.api.letsencrypt.org/directory</code>.

'''T1505 alignment.''' We don't use anymore Certbot as a Docker container on paas-docker role. As such, all paths are now unified. Symlinks have been added to old /srv paths to help transition.

==== Generate a certificate ====
<code>certbot certonly -a webroot --webroot-path=/var/letsencrypt-auto --deploy-hook "service nginx reload" -d foo.nasqueron.org</code>

<code>certbot certonly -a webroot --webroot-path=/var/letsencrypt-auto --deploy-hook "systemctl reload nginx" -d foo.nasqueron.org</code>

==== Generate a certificate for several sites ====
<code>-d foo.nasqueron.org -d bar.nasqueron.org</code>

If a certificate for foo already existed, it will offer to extend it to a new alternative name, which is probably a good idea.

==== Generate a certificate through DNS ====
DNS can be used to generate certificates for domains. For example, the Openfire XMPP certificate is generated like this:

certbot certonly --server https://acme-v02.api.letsencrypt.org/directory --manual --manual-auth-hook /etc/letsencrypt/acme-dns-auth --preferred-challenges dns --debug-challenges -d xmpp.nasqueron.org -d nasqueron.org -d conference.nasqueron.org

From December 2023, you can use this command on FreeBSD servers where {{T|1505}} Let's Encrypt standard installation have been deployed:

certbot certonly --manual --manual-auth-hook /usr/local/etc/letsencrypt/acme-dns-auth --preferred-challenges dns --debug-challenges -d subdomain.nasqueron.org

This uses a specialized DNS server deployed on our Docker PaaS to serve dynamic TLS records under .acme.nasqueron.org, {{Ops file|roles/paas-docker/containers/acme_dns.sls}}.

Support files were previously '''only deployed to paas-docker role''' through {{Ops file|roles/paas-docker/letsencrypt/files}}.

==== Renew all certificates ====
<code>certbot renew</code>

==== Installation on nginx ====
===== Allow Let's encrypt validation =====

include includes/letsencrypt;

This will use {{Ops file|roles/webserver-core/nginx/files/includes/letsencrypt}} nginx configuration.

===== Serve TLS certificate =====

include includes/tls;
ssl_certificate /srv/letsencrypt/etc/live/xmpp.nasqueron.org/fullchain.pem;
ssl_certificate_key /srv/letsencrypt/etc/live/xmpp.nasqueron.org/privkey.pem;

This will configure a compromise between security and compatibility, based on Intermediate Mozilla SSL config<ref>https://ssl-config.mozilla.org/</ref>. The current configuration is served by {{Ops file|roles/webserver-core/nginx/files/includes/tls}}.

If you prefer to restrict the resource for TLS 1.3, and accepts to block legacy clients, you can also use with {{D|3251}}:

include includes/tls-modern-only;
ssl_certificate /srv/letsencrypt/etc/live/xmpp.nasqueron.org/fullchain.pem;
ssl_certificate_key /srv/letsencrypt/etc/live/xmpp.nasqueron.org/privkey.pem;

==== Edit renewal hook ====
In /etc/letsencrypt/renewal or /usr/local/etc/letsencrypt/renewal you can edit this section to customize the command to run after renewal:

[renewalparams]
renew_hook = systemctl reload nginx

That can be applied in batch when nothing is set: <code><nowiki>gsed -i "s/\[\[webroot_map]]/renew_hook = systemctl reload nginx\n\n[[webroot_map]]/g" *.conf</nowiki></code>

=== Install certbot on CentOS 10 stream ===

Packages for Python 3.12 currently used as of 2024-08-04 on CentOS 10 stream can be fetched from Fedora 40 repository.

Installation on Dwellers was done like this:

mkdir /tmp/certbot && cd /tmp/certbot
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/c/certbot-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-acme-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-certbot-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-configargparse-1.7-3.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-josepy-1.13.0-8.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-parsedatetime-2.6-12.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pyOpenSSL-23.2.0-3.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pyrfc3339-1.1-18.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pytz-2024.1-1.fc40.noarch.rpm
dnf install ./*.rpm

=== acme.sh ===

acme.sh should be run with the called "acme" with a
sudo su - acme

==== Generate a certificate ====

==== Generate a certificate for several sites ====

==== Generate a certificate through DNS ====

For DNS certificate, we need to start the certificate generation, edit the DNS records to add the certificate verification. And launch the end of the certificate generation
export ACMEDNS_BASE_URL="https://acme.nasqueron.org"
acme.sh --issue --dns dns_acmedns -d example.com --server letsencrypt

==== Regenerate certificate how are from certbot ====

export ACMEDNS_BASE_URL="https://acme.nasqueron.org"
export ACMEDNS_USERNAME="<username>"
export ACMEDNS_PASSWORD="<password>"
export ACMEDNS_SUBDOMAIN="<subdomain>"

acme.sh --issue --dns dns_acmedns -d example.com --server letsencrypt

==== Renew all certificates ====

For the certificates renewal, anything is needed the cron tab will try to renew them every day "x" day before the expiracy

==== Deploying of the certificates ====

acme.sh --install-cert -d exemple.nasqueron.org \
--cert-file /var/certificates/exemple.nasqueron.org/cert.pem \
--key-file /var/certificates/exemple.nasqueron.org/key.pem \
--fullchain-file /var/certificates/exemple.nasqueron.org/fullchain.pem

===== improved command =====

sudo su acme

export domain="exemple.nasqueron.org"

mkdir /var/certificates/$domain

acme.sh --install-cert -d $domain \
--cert-file /var/certificates/$domain/cert.pem \
--key-file /var/certificates/$domain/key.pem \
--fullchain-file /var/certificates/$domain/fullchain.pem

==== on nginx ====

Le_ReloadCmd='sudo acmesh-nginxCheck'

doit être mis dans le fichier de configuration pour le certificat ou ajouter la commande

== Nasqueron PKI ==
=== Principles ===
Internal PKI material are stored in Vault, see [[Operations grimoire/Vault]] for information about how to generate, renew, etc.

In the operations repository, the {{Ops file|roles/core/certificates}} unit is responsible to deploy <code>nasqueron-vault-ca.crt</code>, used to authenticate for *.nasqueron.drake or IP services. For exemple, Vault clients require a certificate they can validate with a chain from that intermediate CA. Note as we don't currently use the root CA, a fullchain is probably not needed for those services.

=== Rollover a new certificate ===
If <code>nasqueron-vault-ca.crt</code> is updated or if we want instead to provision the root certificate, the following repository need to be updated and services to be redeployed:

{| class="wikitable sortable"
|+ Where to update certificate?
|-
! Repository !! Path !! Purpose !! Deployment instructions
|-
| operations || {{Ops file|roles/core/certificates/files/}} || Trust internal resources on every server || Salt: <code>salt '*' state.apply roles/core/certificates</code>
|-
| docker-airflow || [https://devcentral.nasqueron.org/source/docker-airflow/browse/main/files files/] || Connect to Vault from Airflow Python code || Docker: build and deploy new image nasqueron/airflow on Dwellers
|}

Deployment instructions are up-to-date when newly added on that table. Afterwards, they should be checked against current procedures.

== Special considerations ==
=== New server ===
Let's encrypt client is available on [[Ysul]] (natively) and [[Dwellers]] (as a wrapper script for a Docker container).

Fill a task in Servers component, subscribe Sandlayth and Dereckson to deploy it on a new server.

A salt state would be nice for such purpose. There is work-in-progress on that matter through {{D|3248}}.

=== Internationalized domain names ===
==== Punycode conversion ====
Both for web server configuration and certificate authority, name must be converted to Punycode ([https://www.ietf.org/rfc/rfc3492.txt RFC 3492]):
https://www.punycoder.com/

==== Let's encrypt support ====

Let's encrypt has supported IDN since 2016<ref>https://letsencrypt.org/2016/10/21/introducing-idn-support.html</ref>. We use it for dægrefn.nasqueron.org certificate.

Previously, they were afraid: attackers could register a domain with a Cyrillic character matching a real domains. As some people consider it's the responsibility of the CA to mitigate such risks, the feature has been several times postponed.

==== StartSSL ====

StartSSL is not in activity anymore. It was used at Nasqueron when Let's Encrypt didn't support IDN.

== Troubleshoot ==
=== Acme DNS ===
==== Can't reach ACME DNS API / HTTP Error 403: Forbidden ====
AcmeDNS API access is restricted to IPs addresses set in {{Ops file|roles/webserver-core/nginx/files/includes/geo_nasqueron}}.

If you got a 403 by running acme.sh (only visible in --debug mode):
* If the IP address is missing from geo_nasqueron, add it
* If the IP address is already there, check on the server if /etc/nginx/includes/geo_nasqueron matches
** Yes -> nginx must be reloaded (nginx -t reload)
** No -> Deploy with <code>salt docker-002 state.sls_id /etc/nginx/includes/geo_nasqueron roles/webserver-core/nginx</code>

Another solution is a specific routing to reach the API:
For Hervil, add a route to use router-001 gateway: route add 51.255.124.8/30 172.27.27.1

==== Troubleshoot records in ACME DNS database ====
To check the acme.nasqueron.org subdomain matching your _acme-challenge subdomain: <code>nslookup -type=TXT _acme-challenge.<subdomain>.nasqueron.org</code>

The DNS server to store acme.nasqueron.org records uses a sqlite3 database. To connect and check your parameters:

$ ssh docker-002
$ sqlite3 /srv/acme/lib/acme-dns.db
SELECT Username, Password FROM records WHERE Subdomain = "<DNS answer>"

==== Credential lost for a ACME DNS subdomain ====

Passwords are [https://github.com/joohoi/acme-dns/blob/9c6ca258e1d57e7441b60db4474a68c36356dae2/validation.go hashed with bcrypt], so it's not possible to recover them.

In that case, there are two options:

* if you prefer edit the DNS record, delete acme.sh information for the domain renewal then issue a new certificate, that will register a new account
* if you prefer edit the ACME DNS database, encrypt a new password in bcrypt and update it with sqlite

Certificates requests are limited by Let's Encrypt, updating the password avoids to reach that quota.

You can generate a uuid and pass it to bcrypt, or register a new password (and update DNS):

$ ssh docker-002
$ export SQLITE_HISTORY=/dev/null
$ sqlite3 /srv/acme/lib/acme-dns.db
UPDATE records SET Password = "<new bcrypt hash>" WHERE Subdomain = "<DNS answer>"

You can generate any arbitrary password with any random tool.
It seems UUIDv4 is used by the client code, if you want one, you can use <code>uuidgen -r</code>.

To get password hash with bcrypt, you can use psysh on WindRiver:

$ psysh
password_hash("alpha", PASSWORD_BCRYPT)
$2y$10$N17lnt09YFIoVY4025AmCuGrlmA.ZiThm.RDufcakGq9.3IK4xVcW

Replace $2y$ by $2a$ as x and y are unique for PHP, the rest of the world uses $2a$.

=== Migration to acme.sh ===
==== Paths to update ====

Paths from /usr/local/etc/letsencrypt or /etc/letsencrypt (OS-dependant path) needs to be updated to /var/certificates (cross-platform path). The private key file is only `key.pem` instead of `privkey.pem`.

An example is available on https://devcentral.nasqueron.org/D3624

== Notes & references ==
<references />

Operations grimoire/TLS certificates

2025-02-10T20:51:54Z

DorianWinty: /* Credential lost for a ACME DNS subdomain */

TLS certificates should be used for every service we provide, so we can encrypt the communications.

== Let's Encrypt ==
=== Certbot commands ===

'''acme-v02 migration.''' If you've a complaint acme-v01.api. isn't available, add <code>--server https://acme-v02.api.letsencrypt.org/directory</code>.

'''T1505 alignment.''' We don't use anymore Certbot as a Docker container on paas-docker role. As such, all paths are now unified. Symlinks have been added to old /srv paths to help transition.

==== Generate a certificate ====
<code>certbot certonly -a webroot --webroot-path=/var/letsencrypt-auto --deploy-hook "service nginx reload" -d foo.nasqueron.org</code>

<code>certbot certonly -a webroot --webroot-path=/var/letsencrypt-auto --deploy-hook "systemctl reload nginx" -d foo.nasqueron.org</code>

==== Generate a certificate for several sites ====
<code>-d foo.nasqueron.org -d bar.nasqueron.org</code>

If a certificate for foo already existed, it will offer to extend it to a new alternative name, which is probably a good idea.

==== Generate a certificate through DNS ====
DNS can be used to generate certificates for domains. For example, the Openfire XMPP certificate is generated like this:

certbot certonly --server https://acme-v02.api.letsencrypt.org/directory --manual --manual-auth-hook /etc/letsencrypt/acme-dns-auth --preferred-challenges dns --debug-challenges -d xmpp.nasqueron.org -d nasqueron.org -d conference.nasqueron.org

From December 2023, you can use this command on FreeBSD servers where {{T|1505}} Let's Encrypt standard installation have been deployed:

certbot certonly --manual --manual-auth-hook /usr/local/etc/letsencrypt/acme-dns-auth --preferred-challenges dns --debug-challenges -d subdomain.nasqueron.org

This uses a specialized DNS server deployed on our Docker PaaS to serve dynamic TLS records under .acme.nasqueron.org, {{Ops file|roles/paas-docker/containers/acme_dns.sls}}.

Support files were previously '''only deployed to paas-docker role''' through {{Ops file|roles/paas-docker/letsencrypt/files}}.

==== Renew all certificates ====
<code>certbot renew</code>

==== Installation on nginx ====
===== Allow Let's encrypt validation =====

include includes/letsencrypt;

This will use {{Ops file|roles/webserver-core/nginx/files/includes/letsencrypt}} nginx configuration.

===== Serve TLS certificate =====

include includes/tls;
ssl_certificate /srv/letsencrypt/etc/live/xmpp.nasqueron.org/fullchain.pem;
ssl_certificate_key /srv/letsencrypt/etc/live/xmpp.nasqueron.org/privkey.pem;

This will configure a compromise between security and compatibility, based on Intermediate Mozilla SSL config<ref>https://ssl-config.mozilla.org/</ref>. The current configuration is served by {{Ops file|roles/webserver-core/nginx/files/includes/tls}}.

If you prefer to restrict the resource for TLS 1.3, and accepts to block legacy clients, you can also use with {{D|3251}}:

include includes/tls-modern-only;
ssl_certificate /srv/letsencrypt/etc/live/xmpp.nasqueron.org/fullchain.pem;
ssl_certificate_key /srv/letsencrypt/etc/live/xmpp.nasqueron.org/privkey.pem;

==== Edit renewal hook ====
In /etc/letsencrypt/renewal or /usr/local/etc/letsencrypt/renewal you can edit this section to customize the command to run after renewal:

[renewalparams]
renew_hook = systemctl reload nginx

That can be applied in batch when nothing is set: <code><nowiki>gsed -i "s/\[\[webroot_map]]/renew_hook = systemctl reload nginx\n\n[[webroot_map]]/g" *.conf</nowiki></code>

=== Install certbot on CentOS 10 stream ===

Packages for Python 3.12 currently used as of 2024-08-04 on CentOS 10 stream can be fetched from Fedora 40 repository.

Installation on Dwellers was done like this:

mkdir /tmp/certbot && cd /tmp/certbot
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/c/certbot-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-acme-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-certbot-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-configargparse-1.7-3.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-josepy-1.13.0-8.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-parsedatetime-2.6-12.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pyOpenSSL-23.2.0-3.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pyrfc3339-1.1-18.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pytz-2024.1-1.fc40.noarch.rpm
dnf install ./*.rpm

=== acme.sh ===

acme.sh should be run with the called "acme" with a
sudo su - acme

==== Generate a certificate ====

==== Generate a certificate for several sites ====

==== Generate a certificate through DNS ====

For DNS certificate, we need to start the certificate generation, edit the DNS records to add the certificate verification. And launch the end of the certificate generation
export ACMEDNS_BASE_URL="https://acme.nasqueron.org"
acme.sh --issue --dns dns_acmedns -d example.com --server letsencrypt

==== Regenerate certificate how are from certbot ====

export ACMEDNS_BASE_URL="https://acme.nasqueron.org"
export ACMEDNS_USERNAME="<username>"
export ACMEDNS_PASSWORD="<password>"
export ACMEDNS_SUBDOMAIN="<subdomain>"

acme.sh --issue --dns dns_acmedns -d example.com --server letsencrypt

==== Renew all certificates ====

For the certificates renewal, anything is needed the cron tab will try to renew them every day "x" day before the expiracy

==== Deploying of the certificates ====

acme.sh --install-cert -d exemple.nasqueron.org \
--cert-file /var/certificates/exemple.nasqueron.org/cert.pem \
--key-file /var/certificates/exemple.nasqueron.org/key.pem \
--fullchain-file /var/certificates/exemple.nasqueron.org/fullchain.pem

===== improved command =====

sudo su acme

export domain="exemple.nasqueron.org"

mkdir /var/certificates/$domain

acme.sh --install-cert -d $domain \
--cert-file /var/certificates/$domain/cert.pem \
--key-file /var/certificates/$domain/key.pem \
--fullchain-file /var/certificates/$domain/fullchain.pem

==== on nginx ====

== Nasqueron PKI ==
=== Principles ===
Internal PKI material are stored in Vault, see [[Operations grimoire/Vault]] for information about how to generate, renew, etc.

In the operations repository, the {{Ops file|roles/core/certificates}} unit is responsible to deploy <code>nasqueron-vault-ca.crt</code>, used to authenticate for *.nasqueron.drake or IP services. For exemple, Vault clients require a certificate they can validate with a chain from that intermediate CA. Note as we don't currently use the root CA, a fullchain is probably not needed for those services.

=== Rollover a new certificate ===
If <code>nasqueron-vault-ca.crt</code> is updated or if we want instead to provision the root certificate, the following repository need to be updated and services to be redeployed:

{| class="wikitable sortable"
|+ Where to update certificate?
|-
! Repository !! Path !! Purpose !! Deployment instructions
|-
| operations || {{Ops file|roles/core/certificates/files/}} || Trust internal resources on every server || Salt: <code>salt '*' state.apply roles/core/certificates</code>
|-
| docker-airflow || [https://devcentral.nasqueron.org/source/docker-airflow/browse/main/files files/] || Connect to Vault from Airflow Python code || Docker: build and deploy new image nasqueron/airflow on Dwellers
|}

Deployment instructions are up-to-date when newly added on that table. Afterwards, they should be checked against current procedures.

== Special considerations ==
=== New server ===
Let's encrypt client is available on [[Ysul]] (natively) and [[Dwellers]] (as a wrapper script for a Docker container).

Fill a task in Servers component, subscribe Sandlayth and Dereckson to deploy it on a new server.

A salt state would be nice for such purpose. There is work-in-progress on that matter through {{D|3248}}.

=== Internationalized domain names ===
==== Punycode conversion ====
Both for web server configuration and certificate authority, name must be converted to Punycode ([https://www.ietf.org/rfc/rfc3492.txt RFC 3492]):
https://www.punycoder.com/

==== Let's encrypt support ====

Let's encrypt has supported IDN since 2016<ref>https://letsencrypt.org/2016/10/21/introducing-idn-support.html</ref>. We use it for dægrefn.nasqueron.org certificate.

Previously, they were afraid: attackers could register a domain with a Cyrillic character matching a real domains. As some people consider it's the responsibility of the CA to mitigate such risks, the feature has been several times postponed.

==== StartSSL ====

StartSSL is not in activity anymore. It was used at Nasqueron when Let's Encrypt didn't support IDN.

== Troubleshoot ==
=== Acme DNS ===
==== Can't reach ACME DNS API / HTTP Error 403: Forbidden ====
AcmeDNS API access is restricted to IPs addresses set in {{Ops file|roles/webserver-core/nginx/files/includes/geo_nasqueron}}.

If you got a 403 by running acme.sh (only visible in --debug mode):
* If the IP address is missing from geo_nasqueron, add it
* If the IP address is already there, check on the server if /etc/nginx/includes/geo_nasqueron matches
** Yes -> nginx must be reloaded (nginx -t reload)
** No -> Deploy with <code>salt docker-002 state.sls_id /etc/nginx/includes/geo_nasqueron roles/webserver-core/nginx</code>

Another solution is a specific routing to reach the API:
For Hervil, add a route to use router-001 gateway: route add 51.255.124.8/30 172.27.27.1

==== Troubleshoot records in ACME DNS database ====
To check the acme.nasqueron.org subdomain matching your _acme-challenge subdomain: <code>nslookup -type=TXT _acme-challenge.<subdomain>.nasqueron.org</code>

The DNS server to store acme.nasqueron.org records uses a sqlite3 database. To connect and check your parameters:

$ ssh docker-002
$ sqlite3 /srv/acme/lib/acme-dns.db
SELECT Username, Password FROM records WHERE Subdomain = "<DNS answer>"

==== Credential lost for a ACME DNS subdomain ====

Passwords are [https://github.com/joohoi/acme-dns/blob/9c6ca258e1d57e7441b60db4474a68c36356dae2/validation.go hashed with bcrypt], so it's not possible to recover them.

In that case, there are two options:

* if you prefer edit the DNS record, delete acme.sh information for the domain renewal then issue a new certificate, that will register a new account
* if you prefer edit the ACME DNS database, encrypt a new password in bcrypt and update it with sqlite

Certificates requests are limited by Let's Encrypt, updating the password avoids to reach that quota.

You can generate a uuid and pass it to bcrypt, or register a new password (and update DNS):

$ ssh docker-002
$ export SQLITE_HISTORY=/dev/null
$ sqlite3 /srv/acme/lib/acme-dns.db
UPDATE records SET Password = "<new bcrypt hash>" WHERE Subdomain = "<DNS answer>"

You can generate any arbitrary password with any random tool.
It seems UUIDv4 is used by the client code, if you want one, you can use <code>uuidgen -r</code>.

To get password hash with bcrypt, you can use psysh on WindRiver:

$ psysh
password_hash("alpha", PASSWORD_BCRYPT)
$2y$10$N17lnt09YFIoVY4025AmCuGrlmA.ZiThm.RDufcakGq9.3IK4xVcW

Replace $2y$ by $2a$ as x and y are unique for PHP, the rest of the world uses $2a$.

=== Migration to acme.sh ===
==== Paths to update ====

Paths from /usr/local/etc/letsencrypt or /etc/letsencrypt (OS-dependant path) needs to be updated to /var/certificates (cross-platform path). The private key file is only `key.pem` instead of `privkey.pem`.

An example is available on https://devcentral.nasqueron.org/D3624

== Notes & references ==
<references />

Operations grimoire/TLS certificates

2024-11-05T20:34:57Z

DorianWinty: /* Deploying of the certificates */

TLS certificates should be used for every service we provide, so we can encrypt the communications.

== Let's Encrypt ==
=== Certbot commands ===

'''acme-v02 migration.''' If you've a complaint acme-v01.api. isn't available, add <code>--server https://acme-v02.api.letsencrypt.org/directory</code>.

'''T1505 alignment.''' We don't use anymore Certbot as a Docker container on paas-docker role. As such, all paths are now unified. Symlinks have been added to old /srv paths to help transition.

==== Generate a certificate ====
<code>certbot certonly -a webroot --webroot-path=/var/letsencrypt-auto --deploy-hook "service nginx reload" -d foo.nasqueron.org</code>

<code>certbot certonly -a webroot --webroot-path=/var/letsencrypt-auto --deploy-hook "systemctl reload nginx" -d foo.nasqueron.org</code>

==== Generate a certificate for several sites ====
<code>-d foo.nasqueron.org -d bar.nasqueron.org</code>

If a certificate for foo already existed, it will offer to extend it to a new alternative name, which is probably a good idea.

==== Generate a certificate through DNS ====
DNS can be used to generate certificates for domains. For example, the Openfire XMPP certificate is generated like this:

certbot certonly --server https://acme-v02.api.letsencrypt.org/directory --manual --manual-auth-hook /etc/letsencrypt/acme-dns-auth --preferred-challenges dns --debug-challenges -d xmpp.nasqueron.org -d nasqueron.org -d conference.nasqueron.org

From December 2023, you can use this command on FreeBSD servers where {{T|1505}} Let's Encrypt standard installation have been deployed:

certbot certonly --manual --manual-auth-hook /usr/local/etc/letsencrypt/acme-dns-auth --preferred-challenges dns --debug-challenges -d subdomain.nasqueron.org

This uses a specialized DNS server deployed on our Docker PaaS to serve dynamic TLS records under .acme.nasqueron.org, {{Ops file|roles/paas-docker/containers/acme_dns.sls}}.

Support files were previously '''only deployed to paas-docker role''' through {{Ops file|roles/paas-docker/letsencrypt/files}}.

==== Renew all certificates ====
<code>certbot renew</code>

==== Installation on nginx ====
===== Allow Let's encrypt validation =====

include includes/letsencrypt;

This will use {{Ops file|roles/webserver-core/nginx/files/includes/letsencrypt}} nginx configuration.

===== Serve TLS certificate =====

include includes/tls;
ssl_certificate /srv/letsencrypt/etc/live/xmpp.nasqueron.org/fullchain.pem;
ssl_certificate_key /srv/letsencrypt/etc/live/xmpp.nasqueron.org/privkey.pem;

This will configure a compromise between security and compatibility, based on Intermediate Mozilla SSL config<ref>https://ssl-config.mozilla.org/</ref>. The current configuration is served by {{Ops file|roles/webserver-core/nginx/files/includes/tls}}.

If you prefer to restrict the resource for TLS 1.3, and accepts to block legacy clients, you can also use with {{D|3251}}:

include includes/tls-modern-only;
ssl_certificate /srv/letsencrypt/etc/live/xmpp.nasqueron.org/fullchain.pem;
ssl_certificate_key /srv/letsencrypt/etc/live/xmpp.nasqueron.org/privkey.pem;

==== Edit renewal hook ====
In /etc/letsencrypt/renewal or /usr/local/etc/letsencrypt/renewal you can edit this section to customize the command to run after renewal:

[renewalparams]
renew_hook = systemctl reload nginx

That can be applied in batch when nothing is set: <code><nowiki>gsed -i "s/\[\[webroot_map]]/renew_hook = systemctl reload nginx\n\n[[webroot_map]]/g" *.conf</nowiki></code>

=== Install certbot on CentOS 10 stream ===

Packages for Python 3.12 currently used as of 2024-08-04 on CentOS 10 stream can be fetched from Fedora 40 repository.

Installation on Dwellers was done like this:

mkdir /tmp/certbot && cd /tmp/certbot
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/c/certbot-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-acme-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-certbot-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-configargparse-1.7-3.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-josepy-1.13.0-8.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-parsedatetime-2.6-12.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pyOpenSSL-23.2.0-3.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pyrfc3339-1.1-18.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pytz-2024.1-1.fc40.noarch.rpm
dnf install ./*.rpm

=== acme.sh ===

acme.sh should be run with the called "acme" with a
sudo su - acme

==== Generate a certificate ====

==== Generate a certificate for several sites ====

==== Generate a certificate through DNS ====

For DNS certificate, we need to start the certificate generation, edit the DNS records to add the certificate verification. And launch the end of the certificate generation
export ACMEDNS_BASE_URL="https://acme.nasqueron.org"
acme.sh --issue --dns dns_acmedns -d example.com --server letsencrypt

==== Regenerate certificate how are from certbot ====

export ACMEDNS_BASE_URL="https://acme.nasqueron.org"
export ACMEDNS_USERNAME="<username>"
export ACMEDNS_PASSWORD="<password>"
export ACMEDNS_SUBDOMAIN="<subdomain>"

acme.sh --issue --dns dns_acmedns -d example.com --server letsencrypt

==== Renew all certificates ====

For the certificates renewal, anything is needed the cron tab will try to renew them every day "x" day before the expiracy

==== Deploying of the certificates ====

acme.sh --install-cert -d exemple.nasqueron.org \
--cert-file /var/certificates/exemple.nasqueron.org/cert.pem \
--key-file /var/certificates/exemple.nasqueron.org/key.pem \
--fullchain-file /var/certificates/exemple.nasqueron.org/fullchain.pem

===== improved command =====

sudo su acme

export domain="exemple.nasqueron.org"

mkdir /var/certificates/$domain

acme.sh --install-cert -d $domain \
--cert-file /var/certificates/$domain/cert.pem \
--key-file /var/certificates/$domain/key.pem \
--fullchain-file /var/certificates/$domain/fullchain.pem

==== on nginx ====

== Nasqueron PKI ==
=== Principles ===
Internal PKI material are stored in Vault, see [[Operations grimoire/Vault]] for information about how to generate, renew, etc.

In the operations repository, the {{Ops file|roles/core/certificates}} unit is responsible to deploy <code>nasqueron-vault-ca.crt</code>, used to authenticate for *.nasqueron.drake or IP services. For exemple, Vault clients require a certificate they can validate with a chain from that intermediate CA. Note as we don't currently use the root CA, a fullchain is probably not needed for those services.

=== Rollover a new certificate ===
If <code>nasqueron-vault-ca.crt</code> is updated or if we want instead to provision the root certificate, the following repository need to be updated and services to be redeployed:

{| class="wikitable sortable"
|+ Where to update certificate?
|-
! Repository !! Path !! Purpose !! Deployment instructions
|-
| operations || {{Ops file|roles/core/certificates/files/}} || Trust internal resources on every server || Salt: <code>salt '*' state.apply roles/core/certificates</code>
|-
| docker-airflow || [https://devcentral.nasqueron.org/source/docker-airflow/browse/main/files files/] || Connect to Vault from Airflow Python code || Docker: build and deploy new image nasqueron/airflow on Dwellers
|}

Deployment instructions are up-to-date when newly added on that table. Afterwards, they should be checked against current procedures.

== Special considerations ==
=== New server ===
Let's encrypt client is available on [[Ysul]] (natively) and [[Dwellers]] (as a wrapper script for a Docker container).

Fill a task in Servers component, subscribe Sandlayth and Dereckson to deploy it on a new server.

A salt state would be nice for such purpose. There is work-in-progress on that matter through {{D|3248}}.

=== Internationalized domain names ===
==== Punycode conversion ====
Both for web server configuration and certificate authority, name must be converted to Punycode ([https://www.ietf.org/rfc/rfc3492.txt RFC 3492]):
https://www.punycoder.com/

==== Let's encrypt support ====

Let's encrypt has supported IDN since 2016<ref>https://letsencrypt.org/2016/10/21/introducing-idn-support.html</ref>. We use it for dægrefn.nasqueron.org certificate.

Previously, they were afraid: attackers could register a domain with a Cyrillic character matching a real domains. As some people consider it's the responsibility of the CA to mitigate such risks, the feature has been several times postponed.

==== StartSSL ====

StartSSL is not in activity anymore. It was used at Nasqueron when Let's Encrypt didn't support IDN.

== Troubleshoot ==
=== Acme DNS ===
==== Can't reach ACME DNS API / HTTP Error 403: Forbidden ====
AcmeDNS API access is restricted to IPs addresses set in {{Ops file|roles/webserver-core/nginx/files/includes/geo_nasqueron}}.

If you got a 403 by running acme.sh (only visible in --debug mode):
* If the IP address is missing from geo_nasqueron, add it
* If the IP address is already there, check on the server if /etc/nginx/includes/geo_nasqueron matches
** Yes -> nginx must be reloaded (nginx -t reload)
** No -> Deploy with <code>salt docker-002 state.sls_id /etc/nginx/includes/geo_nasqueron roles/webserver-core/nginx</code>

Another solution is a specific routing to reach the API:
For Hervil, add a route to use router-001 gateway: route add 51.255.124.8/30 172.27.27.1

==== Troubleshoot records in ACME DNS database ====
To check the acme.nasqueron.org subdomain matching your _acme-challenge subdomain: <code>nslookup -type=TXT _acme-challenge.<subdomain>.nasqueron.org</code>

The DNS server to store acme.nasqueron.org records uses a sqlite3 database. To connect and check your parameters:

$ ssh docker-002
$ sqlite3 /srv/acme/lib/acme-dns.db
SELECT Username, Password FROM records WHERE Subdomain = "<DNS answer>"

==== Credential lost for a ACME DNS subdomain ====

Passwords are [https://github.com/joohoi/acme-dns/blob/9c6ca258e1d57e7441b60db4474a68c36356dae2/validation.go hashed with bcrypt], so it's not possible to recover them.

In that case, there are two options:

* if you prefer edit the DNS record, delete acme.sh information for the domain renewal then issue a new certificate, that will register a new account
* if you prefer edit the ACME DNS database, encrypt a new password in bcrypt and update it with sqlite

Certificates requests are limited by Let's Encrypt, updating the password avoids to reach that quota.

You can generate a uuid and pass it to bcrypt, or register a new password (and update DNS):

$ ssh docker-002
$ export SQLITE_HISTORY=/dev/null
$ sqlite3 /srv/acme/. lib/acme-dns.db
UPDATE records SET Password = "<new bcrypt hash>" WHERE Subdomain = "<DNS answer>"

You can generate any arbitrary password with any random tool.
It seems UUIDv4 is used by the client code, if you want one, you can use <code>uuidgen -r</code>.

To get password hash with bcrypt, you can use psysh on WindRiver:

$ psysh
password_hash("alpha", PASSWORD_BCRYPT)
$2y$10$N17lnt09YFIoVY4025AmCuGrlmA.ZiThm.RDufcakGq9.3IK4xVcW

Replace $2y$ by $2a$ as x and y are unique for PHP, the rest of the world uses $2a$.

== Notes & references ==
<references />

Operations grimoire/TLS certificates

2024-11-05T20:16:48Z

DorianWinty: /* Deploying of the certificates */

TLS certificates should be used for every service we provide, so we can encrypt the communications.

== Let's Encrypt ==
=== Certbot commands ===

'''acme-v02 migration.''' If you've a complaint acme-v01.api. isn't available, add <code>--server https://acme-v02.api.letsencrypt.org/directory</code>.

'''T1505 alignment.''' We don't use anymore Certbot as a Docker container on paas-docker role. As such, all paths are now unified. Symlinks have been added to old /srv paths to help transition.

==== Generate a certificate ====
<code>certbot certonly -a webroot --webroot-path=/var/letsencrypt-auto --deploy-hook "service nginx reload" -d foo.nasqueron.org</code>

<code>certbot certonly -a webroot --webroot-path=/var/letsencrypt-auto --deploy-hook "systemctl reload nginx" -d foo.nasqueron.org</code>

==== Generate a certificate for several sites ====
<code>-d foo.nasqueron.org -d bar.nasqueron.org</code>

If a certificate for foo already existed, it will offer to extend it to a new alternative name, which is probably a good idea.

==== Generate a certificate through DNS ====
DNS can be used to generate certificates for domains. For example, the Openfire XMPP certificate is generated like this:

certbot certonly --server https://acme-v02.api.letsencrypt.org/directory --manual --manual-auth-hook /etc/letsencrypt/acme-dns-auth --preferred-challenges dns --debug-challenges -d xmpp.nasqueron.org -d nasqueron.org -d conference.nasqueron.org

From December 2023, you can use this command on FreeBSD servers where {{T|1505}} Let's Encrypt standard installation have been deployed:

certbot certonly --manual --manual-auth-hook /usr/local/etc/letsencrypt/acme-dns-auth --preferred-challenges dns --debug-challenges -d subdomain.nasqueron.org

This uses a specialized DNS server deployed on our Docker PaaS to serve dynamic TLS records under .acme.nasqueron.org, {{Ops file|roles/paas-docker/containers/acme_dns.sls}}.

Support files were previously '''only deployed to paas-docker role''' through {{Ops file|roles/paas-docker/letsencrypt/files}}.

==== Renew all certificates ====
<code>certbot renew</code>

==== Installation on nginx ====
===== Allow Let's encrypt validation =====

include includes/letsencrypt;

This will use {{Ops file|roles/webserver-core/nginx/files/includes/letsencrypt}} nginx configuration.

===== Serve TLS certificate =====

include includes/tls;
ssl_certificate /srv/letsencrypt/etc/live/xmpp.nasqueron.org/fullchain.pem;
ssl_certificate_key /srv/letsencrypt/etc/live/xmpp.nasqueron.org/privkey.pem;

This will configure a compromise between security and compatibility, based on Intermediate Mozilla SSL config<ref>https://ssl-config.mozilla.org/</ref>. The current configuration is served by {{Ops file|roles/webserver-core/nginx/files/includes/tls}}.

If you prefer to restrict the resource for TLS 1.3, and accepts to block legacy clients, you can also use with {{D|3251}}:

include includes/tls-modern-only;
ssl_certificate /srv/letsencrypt/etc/live/xmpp.nasqueron.org/fullchain.pem;
ssl_certificate_key /srv/letsencrypt/etc/live/xmpp.nasqueron.org/privkey.pem;

==== Edit renewal hook ====
In /etc/letsencrypt/renewal or /usr/local/etc/letsencrypt/renewal you can edit this section to customize the command to run after renewal:

[renewalparams]
renew_hook = systemctl reload nginx

That can be applied in batch when nothing is set: <code><nowiki>gsed -i "s/\[\[webroot_map]]/renew_hook = systemctl reload nginx\n\n[[webroot_map]]/g" *.conf</nowiki></code>

=== Install certbot on CentOS 10 stream ===

Packages for Python 3.12 currently used as of 2024-08-04 on CentOS 10 stream can be fetched from Fedora 40 repository.

Installation on Dwellers was done like this:

mkdir /tmp/certbot && cd /tmp/certbot
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/c/certbot-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-acme-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-certbot-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-configargparse-1.7-3.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-josepy-1.13.0-8.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-parsedatetime-2.6-12.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pyOpenSSL-23.2.0-3.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pyrfc3339-1.1-18.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pytz-2024.1-1.fc40.noarch.rpm
dnf install ./*.rpm

=== acme.sh ===

acme.sh should be run with the called "acme" with a
sudo su - acme

==== Generate a certificate ====

==== Generate a certificate for several sites ====

==== Generate a certificate through DNS ====

For DNS certificate, we need to start the certificate generation, edit the DNS records to add the certificate verification. And launch the end of the certificate generation
export ACMEDNS_BASE_URL="https://acme.nasqueron.org"
acme.sh --issue --dns dns_acmedns -d example.com --server letsencrypt

==== Regenerate certificate how are from certbot ====

export ACMEDNS_BASE_URL="https://acme.nasqueron.org"
export ACMEDNS_USERNAME="<username>"
export ACMEDNS_PASSWORD="<password>"
export ACMEDNS_SUBDOMAIN="<subdomain>"

acme.sh --issue --dns dns_acmedns -d example.com --server letsencrypt

==== Renew all certificates ====

For the certificates renewal, anything is needed the cron tab will try to renew them every day "x" day before the expiracy

==== Deploying of the certificates ====

acme.sh --install-cert -d exemple.nasqueron.org \
--cert-file /var/certificates/exemple.nasqueron.org/cert.pem \
--key-file /var/certificates/exemple.nasqueron.org/key.pem \
--fullchain-file /var/certificates/exemple.nasqueron.org/fullchain.pem \

==== on nginx ====

== Nasqueron PKI ==
=== Principles ===
Internal PKI material are stored in Vault, see [[Operations grimoire/Vault]] for information about how to generate, renew, etc.

In the operations repository, the {{Ops file|roles/core/certificates}} unit is responsible to deploy <code>nasqueron-vault-ca.crt</code>, used to authenticate for *.nasqueron.drake or IP services. For exemple, Vault clients require a certificate they can validate with a chain from that intermediate CA. Note as we don't currently use the root CA, a fullchain is probably not needed for those services.

=== Rollover a new certificate ===
If <code>nasqueron-vault-ca.crt</code> is updated or if we want instead to provision the root certificate, the following repository need to be updated and services to be redeployed:

{| class="wikitable sortable"
|+ Where to update certificate?
|-
! Repository !! Path !! Purpose !! Deployment instructions
|-
| operations || {{Ops file|roles/core/certificates/files/}} || Trust internal resources on every server || Salt: <code>salt '*' state.apply roles/core/certificates</code>
|-
| docker-airflow || [https://devcentral.nasqueron.org/source/docker-airflow/browse/main/files files/] || Connect to Vault from Airflow Python code || Docker: build and deploy new image nasqueron/airflow on Dwellers
|}

Deployment instructions are up-to-date when newly added on that table. Afterwards, they should be checked against current procedures.

== Special considerations ==
=== New server ===
Let's encrypt client is available on [[Ysul]] (natively) and [[Dwellers]] (as a wrapper script for a Docker container).

Fill a task in Servers component, subscribe Sandlayth and Dereckson to deploy it on a new server.

A salt state would be nice for such purpose. There is work-in-progress on that matter through {{D|3248}}.

=== Internationalized domain names ===
==== Punycode conversion ====
Both for web server configuration and certificate authority, name must be converted to Punycode ([https://www.ietf.org/rfc/rfc3492.txt RFC 3492]):
https://www.punycoder.com/

==== Let's encrypt support ====

Let's encrypt has supported IDN since 2016<ref>https://letsencrypt.org/2016/10/21/introducing-idn-support.html</ref>. We use it for dægrefn.nasqueron.org certificate.

Previously, they were afraid: attackers could register a domain with a Cyrillic character matching a real domains. As some people consider it's the responsibility of the CA to mitigate such risks, the feature has been several times postponed.

==== StartSSL ====

StartSSL is not in activity anymore. It was used at Nasqueron when Let's Encrypt didn't support IDN.

== Troubleshoot ==
=== Acme DNS ===
==== Can't reach ACME DNS API / HTTP Error 403: Forbidden ====
AcmeDNS API access is restricted to IPs addresses set in {{Ops file|roles/webserver-core/nginx/files/includes/geo_nasqueron}}.

If you got a 403 by running acme.sh (only visible in --debug mode):
* If the IP address is missing from geo_nasqueron, add it
* If the IP address is already there, check on the server if /etc/nginx/includes/geo_nasqueron matches
** Yes -> nginx must be reloaded (nginx -t reload)
** No -> Deploy with <code>salt docker-002 state.sls_id /etc/nginx/includes/geo_nasqueron roles/webserver-core/nginx</code>

Another solution is a specific routing to reach the API:
For Hervil, add a route to use router-001 gateway: route add 51.255.124.8/30 172.27.27.1

==== Troubleshoot records in ACME DNS database ====
To check the acme.nasqueron.org subdomain matching your _acme-challenge subdomain: <code>nslookup -type=TXT _acme-challenge.<subdomain>.nasqueron.org</code>

The DNS server to store acme.nasqueron.org records uses a sqlite3 database. To connect and check your parameters:

$ ssh docker-002
$ sqlite3 /srv/acme/lib/acme-dns.db
SELECT Username, Password FROM records WHERE Subdomain = "<DNS answer>"

==== Credential lost for a ACME DNS subdomain ====

Passwords are [https://github.com/joohoi/acme-dns/blob/9c6ca258e1d57e7441b60db4474a68c36356dae2/validation.go hashed with bcrypt], so it's not possible to recover them.

In that case, there are two options:

* if you prefer edit the DNS record, delete acme.sh information for the domain renewal then issue a new certificate, that will register a new account
* if you prefer edit the ACME DNS database, encrypt a new password in bcrypt and update it with sqlite

Certificates requests are limited by Let's Encrypt, updating the password avoids to reach that quota.

You can generate a uuid and pass it to bcrypt, or register a new password (and update DNS):

$ ssh docker-002
$ export SQLITE_HISTORY=/dev/null
$ sqlite3 /srv/acme/. lib/acme-dns.db
UPDATE records SET Password = "<new bcrypt hash>" WHERE Subdomain = "<DNS answer>"

You can generate any arbitrary password with any random tool.
It seems UUIDv4 is used by the client code, if you want one, you can use <code>uuidgen -r</code>.

To get password hash with bcrypt, you can use psysh on WindRiver:

$ psysh
password_hash("alpha", PASSWORD_BCRYPT)
$2y$10$N17lnt09YFIoVY4025AmCuGrlmA.ZiThm.RDufcakGq9.3IK4xVcW

Replace $2y$ by $2a$ as x and y are unique for PHP, the rest of the world uses $2a$.

== Notes & references ==
<references />

Operations grimoire/TLS certificates

2024-11-05T20:01:19Z

DorianWinty: /* Deploying of the certificates */

TLS certificates should be used for every service we provide, so we can encrypt the communications.

== Let's Encrypt ==
=== Certbot commands ===

'''acme-v02 migration.''' If you've a complaint acme-v01.api. isn't available, add <code>--server https://acme-v02.api.letsencrypt.org/directory</code>.

'''T1505 alignment.''' We don't use anymore Certbot as a Docker container on paas-docker role. As such, all paths are now unified. Symlinks have been added to old /srv paths to help transition.

==== Generate a certificate ====
<code>certbot certonly -a webroot --webroot-path=/var/letsencrypt-auto --deploy-hook "service nginx reload" -d foo.nasqueron.org</code>

<code>certbot certonly -a webroot --webroot-path=/var/letsencrypt-auto --deploy-hook "systemctl reload nginx" -d foo.nasqueron.org</code>

==== Generate a certificate for several sites ====
<code>-d foo.nasqueron.org -d bar.nasqueron.org</code>

If a certificate for foo already existed, it will offer to extend it to a new alternative name, which is probably a good idea.

==== Generate a certificate through DNS ====
DNS can be used to generate certificates for domains. For example, the Openfire XMPP certificate is generated like this:

certbot certonly --server https://acme-v02.api.letsencrypt.org/directory --manual --manual-auth-hook /etc/letsencrypt/acme-dns-auth --preferred-challenges dns --debug-challenges -d xmpp.nasqueron.org -d nasqueron.org -d conference.nasqueron.org

From December 2023, you can use this command on FreeBSD servers where {{T|1505}} Let's Encrypt standard installation have been deployed:

certbot certonly --manual --manual-auth-hook /usr/local/etc/letsencrypt/acme-dns-auth --preferred-challenges dns --debug-challenges -d subdomain.nasqueron.org

This uses a specialized DNS server deployed on our Docker PaaS to serve dynamic TLS records under .acme.nasqueron.org, {{Ops file|roles/paas-docker/containers/acme_dns.sls}}.

Support files were previously '''only deployed to paas-docker role''' through {{Ops file|roles/paas-docker/letsencrypt/files}}.

==== Renew all certificates ====
<code>certbot renew</code>

==== Installation on nginx ====
===== Allow Let's encrypt validation =====

include includes/letsencrypt;

This will use {{Ops file|roles/webserver-core/nginx/files/includes/letsencrypt}} nginx configuration.

===== Serve TLS certificate =====

include includes/tls;
ssl_certificate /srv/letsencrypt/etc/live/xmpp.nasqueron.org/fullchain.pem;
ssl_certificate_key /srv/letsencrypt/etc/live/xmpp.nasqueron.org/privkey.pem;

This will configure a compromise between security and compatibility, based on Intermediate Mozilla SSL config<ref>https://ssl-config.mozilla.org/</ref>. The current configuration is served by {{Ops file|roles/webserver-core/nginx/files/includes/tls}}.

If you prefer to restrict the resource for TLS 1.3, and accepts to block legacy clients, you can also use with {{D|3251}}:

include includes/tls-modern-only;
ssl_certificate /srv/letsencrypt/etc/live/xmpp.nasqueron.org/fullchain.pem;
ssl_certificate_key /srv/letsencrypt/etc/live/xmpp.nasqueron.org/privkey.pem;

==== Edit renewal hook ====
In /etc/letsencrypt/renewal or /usr/local/etc/letsencrypt/renewal you can edit this section to customize the command to run after renewal:

[renewalparams]
renew_hook = systemctl reload nginx

That can be applied in batch when nothing is set: <code><nowiki>gsed -i "s/\[\[webroot_map]]/renew_hook = systemctl reload nginx\n\n[[webroot_map]]/g" *.conf</nowiki></code>

=== Install certbot on CentOS 10 stream ===

Packages for Python 3.12 currently used as of 2024-08-04 on CentOS 10 stream can be fetched from Fedora 40 repository.

Installation on Dwellers was done like this:

mkdir /tmp/certbot && cd /tmp/certbot
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/c/certbot-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-acme-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-certbot-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-configargparse-1.7-3.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-josepy-1.13.0-8.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-parsedatetime-2.6-12.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pyOpenSSL-23.2.0-3.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pyrfc3339-1.1-18.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pytz-2024.1-1.fc40.noarch.rpm
dnf install ./*.rpm

=== acme.sh ===

acme.sh should be run with the called "acme" with a
sudo su - acme

==== Generate a certificate ====

==== Generate a certificate for several sites ====

==== Generate a certificate through DNS ====

For DNS certificate, we need to start the certificate generation, edit the DNS records to add the certificate verification. And launch the end of the certificate generation
export ACMEDNS_BASE_URL="https://acme.nasqueron.org"
acme.sh --issue --dns dns_acmedns -d example.com --server letsencrypt

==== Regenerate certificate how are from certbot ====

export ACMEDNS_BASE_URL="https://acme.nasqueron.org"
export ACMEDNS_USERNAME="<username>"
export ACMEDNS_PASSWORD="<password>"
export ACMEDNS_SUBDOMAIN="<subdomain>"

acme.sh --issue --dns dns_acmedns -d example.com --server letsencrypt

==== Renew all certificates ====

For the certificates renewal, anything is needed the cron tab will try to renew them every day "x" day before the expiracy

==== Deploying of the certificates ====

acme.sh --install-cert -d exemple.nasqueron.org \
--cert-file /var/certificates/exemple.nasqueron.org/cert.pem
--key-file /var/certificates/exemple.nasqueron.org/key.pem \
--fullchain-file /var/certificates/exemple.nasqueron.org/fullchain.pem \

==== on nginx ====

== Nasqueron PKI ==
=== Principles ===
Internal PKI material are stored in Vault, see [[Operations grimoire/Vault]] for information about how to generate, renew, etc.

In the operations repository, the {{Ops file|roles/core/certificates}} unit is responsible to deploy <code>nasqueron-vault-ca.crt</code>, used to authenticate for *.nasqueron.drake or IP services. For exemple, Vault clients require a certificate they can validate with a chain from that intermediate CA. Note as we don't currently use the root CA, a fullchain is probably not needed for those services.

=== Rollover a new certificate ===
If <code>nasqueron-vault-ca.crt</code> is updated or if we want instead to provision the root certificate, the following repository need to be updated and services to be redeployed:

{| class="wikitable sortable"
|+ Where to update certificate?
|-
! Repository !! Path !! Purpose !! Deployment instructions
|-
| operations || {{Ops file|roles/core/certificates/files/}} || Trust internal resources on every server || Salt: <code>salt '*' state.apply roles/core/certificates</code>
|-
| docker-airflow || [https://devcentral.nasqueron.org/source/docker-airflow/browse/main/files files/] || Connect to Vault from Airflow Python code || Docker: build and deploy new image nasqueron/airflow on Dwellers
|}

Deployment instructions are up-to-date when newly added on that table. Afterwards, they should be checked against current procedures.

== Special considerations ==
=== New server ===
Let's encrypt client is available on [[Ysul]] (natively) and [[Dwellers]] (as a wrapper script for a Docker container).

Fill a task in Servers component, subscribe Sandlayth and Dereckson to deploy it on a new server.

A salt state would be nice for such purpose. There is work-in-progress on that matter through {{D|3248}}.

=== Internationalized domain names ===
==== Punycode conversion ====
Both for web server configuration and certificate authority, name must be converted to Punycode ([https://www.ietf.org/rfc/rfc3492.txt RFC 3492]):
https://www.punycoder.com/

==== Let's encrypt support ====

Let's encrypt has supported IDN since 2016<ref>https://letsencrypt.org/2016/10/21/introducing-idn-support.html</ref>. We use it for dægrefn.nasqueron.org certificate.

Previously, they were afraid: attackers could register a domain with a Cyrillic character matching a real domains. As some people consider it's the responsibility of the CA to mitigate such risks, the feature has been several times postponed.

==== StartSSL ====

StartSSL is not in activity anymore. It was used at Nasqueron when Let's Encrypt didn't support IDN.

== Troubleshoot ==
=== Acme DNS ===
==== Can't reach ACME DNS API / HTTP Error 403: Forbidden ====
AcmeDNS API access is restricted to IPs addresses set in {{Ops file|roles/webserver-core/nginx/files/includes/geo_nasqueron}}.

If you got a 403 by running acme.sh (only visible in --debug mode):
* If the IP address is missing from geo_nasqueron, add it
* If the IP address is already there, check on the server if /etc/nginx/includes/geo_nasqueron matches
** Yes -> nginx must be reloaded (nginx -t reload)
** No -> Deploy with <code>salt docker-002 state.sls_id /etc/nginx/includes/geo_nasqueron roles/webserver-core/nginx</code>

Another solution is a specific routing to reach the API:
For Hervil, add a route to use router-001 gateway: route add 51.255.124.8/30 172.27.27.1

==== Troubleshoot records in ACME DNS database ====
To check the acme.nasqueron.org subdomain matching your _acme-challenge subdomain: <code>nslookup -type=TXT _acme-challenge.<subdomain>.nasqueron.org</code>

The DNS server to store acme.nasqueron.org records uses a sqlite3 database. To connect and check your parameters:

$ ssh docker-002
$ sqlite3 /srv/acme/lib/acme-dns.db
SELECT Username, Password FROM records WHERE Subdomain = "<DNS answer>"

==== Credential lost for a ACME DNS subdomain ====

Passwords are [https://github.com/joohoi/acme-dns/blob/9c6ca258e1d57e7441b60db4474a68c36356dae2/validation.go hashed with bcrypt], so it's not possible to recover them.

In that case, there are two options:

* if you prefer edit the DNS record, delete acme.sh information for the domain renewal then issue a new certificate, that will register a new account
* if you prefer edit the ACME DNS database, encrypt a new password in bcrypt and update it with sqlite

Certificates requests are limited by Let's Encrypt, updating the password avoids to reach that quota.

You can generate a uuid and pass it to bcrypt, or register a new password (and update DNS):

$ ssh docker-002
$ export SQLITE_HISTORY=/dev/null
$ sqlite3 /srv/acme/. lib/acme-dns.db
UPDATE records SET Password = "<new bcrypt hash>" WHERE Subdomain = "<DNS answer>"

You can generate any arbitrary password with any random tool.
It seems UUIDv4 is used by the client code, if you want one, you can use <code>uuidgen -r</code>.

To get password hash with bcrypt, you can use psysh on WindRiver:

$ psysh
password_hash("alpha", PASSWORD_BCRYPT)
$2y$10$N17lnt09YFIoVY4025AmCuGrlmA.ZiThm.RDufcakGq9.3IK4xVcW

Replace $2y$ by $2a$ as x and y are unique for PHP, the rest of the world uses $2a$.

== Notes & references ==
<references />

Operations grimoire/TLS certificates

2024-11-05T20:01:05Z

DorianWinty: /* Deploying of the certificates */

TLS certificates should be used for every service we provide, so we can encrypt the communications.

== Let's Encrypt ==
=== Certbot commands ===

'''acme-v02 migration.''' If you've a complaint acme-v01.api. isn't available, add <code>--server https://acme-v02.api.letsencrypt.org/directory</code>.

'''T1505 alignment.''' We don't use anymore Certbot as a Docker container on paas-docker role. As such, all paths are now unified. Symlinks have been added to old /srv paths to help transition.

==== Generate a certificate ====
<code>certbot certonly -a webroot --webroot-path=/var/letsencrypt-auto --deploy-hook "service nginx reload" -d foo.nasqueron.org</code>

<code>certbot certonly -a webroot --webroot-path=/var/letsencrypt-auto --deploy-hook "systemctl reload nginx" -d foo.nasqueron.org</code>

==== Generate a certificate for several sites ====
<code>-d foo.nasqueron.org -d bar.nasqueron.org</code>

If a certificate for foo already existed, it will offer to extend it to a new alternative name, which is probably a good idea.

==== Generate a certificate through DNS ====
DNS can be used to generate certificates for domains. For example, the Openfire XMPP certificate is generated like this:

certbot certonly --server https://acme-v02.api.letsencrypt.org/directory --manual --manual-auth-hook /etc/letsencrypt/acme-dns-auth --preferred-challenges dns --debug-challenges -d xmpp.nasqueron.org -d nasqueron.org -d conference.nasqueron.org

From December 2023, you can use this command on FreeBSD servers where {{T|1505}} Let's Encrypt standard installation have been deployed:

certbot certonly --manual --manual-auth-hook /usr/local/etc/letsencrypt/acme-dns-auth --preferred-challenges dns --debug-challenges -d subdomain.nasqueron.org

This uses a specialized DNS server deployed on our Docker PaaS to serve dynamic TLS records under .acme.nasqueron.org, {{Ops file|roles/paas-docker/containers/acme_dns.sls}}.

Support files were previously '''only deployed to paas-docker role''' through {{Ops file|roles/paas-docker/letsencrypt/files}}.

==== Renew all certificates ====
<code>certbot renew</code>

==== Installation on nginx ====
===== Allow Let's encrypt validation =====

include includes/letsencrypt;

This will use {{Ops file|roles/webserver-core/nginx/files/includes/letsencrypt}} nginx configuration.

===== Serve TLS certificate =====

include includes/tls;
ssl_certificate /srv/letsencrypt/etc/live/xmpp.nasqueron.org/fullchain.pem;
ssl_certificate_key /srv/letsencrypt/etc/live/xmpp.nasqueron.org/privkey.pem;

This will configure a compromise between security and compatibility, based on Intermediate Mozilla SSL config<ref>https://ssl-config.mozilla.org/</ref>. The current configuration is served by {{Ops file|roles/webserver-core/nginx/files/includes/tls}}.

If you prefer to restrict the resource for TLS 1.3, and accepts to block legacy clients, you can also use with {{D|3251}}:

include includes/tls-modern-only;
ssl_certificate /srv/letsencrypt/etc/live/xmpp.nasqueron.org/fullchain.pem;
ssl_certificate_key /srv/letsencrypt/etc/live/xmpp.nasqueron.org/privkey.pem;

==== Edit renewal hook ====
In /etc/letsencrypt/renewal or /usr/local/etc/letsencrypt/renewal you can edit this section to customize the command to run after renewal:

[renewalparams]
renew_hook = systemctl reload nginx

That can be applied in batch when nothing is set: <code><nowiki>gsed -i "s/\[\[webroot_map]]/renew_hook = systemctl reload nginx\n\n[[webroot_map]]/g" *.conf</nowiki></code>

=== Install certbot on CentOS 10 stream ===

Packages for Python 3.12 currently used as of 2024-08-04 on CentOS 10 stream can be fetched from Fedora 40 repository.

Installation on Dwellers was done like this:

mkdir /tmp/certbot && cd /tmp/certbot
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/c/certbot-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-acme-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-certbot-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-configargparse-1.7-3.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-josepy-1.13.0-8.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-parsedatetime-2.6-12.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pyOpenSSL-23.2.0-3.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pyrfc3339-1.1-18.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pytz-2024.1-1.fc40.noarch.rpm
dnf install ./*.rpm

=== acme.sh ===

acme.sh should be run with the called "acme" with a
sudo su - acme

==== Generate a certificate ====

==== Generate a certificate for several sites ====

==== Generate a certificate through DNS ====

For DNS certificate, we need to start the certificate generation, edit the DNS records to add the certificate verification. And launch the end of the certificate generation
export ACMEDNS_BASE_URL="https://acme.nasqueron.org"
acme.sh --issue --dns dns_acmedns -d example.com --server letsencrypt

==== Regenerate certificate how are from certbot ====

export ACMEDNS_BASE_URL="https://acme.nasqueron.org"
export ACMEDNS_USERNAME="<username>"
export ACMEDNS_PASSWORD="<password>"
export ACMEDNS_SUBDOMAIN="<subdomain>"

acme.sh --issue --dns dns_acmedns -d example.com --server letsencrypt

==== Renew all certificates ====

For the certificates renewal, anything is needed the cron tab will try to renew them every day "x" day before the expiracy

==== Deploying of the certificates ====

acme.sh --install-cert -d exemple.nasqueron.org \
--cert-file /var/certificates/exemple.nasqueron.org/cert.pem
--key-file /var/certificates/exemple.nasqueron.org/key.pem \
--fullchain-file /var/certificates/exemple.nasqueron.org/fullchain.pemt.pem \

==== on nginx ====

== Nasqueron PKI ==
=== Principles ===
Internal PKI material are stored in Vault, see [[Operations grimoire/Vault]] for information about how to generate, renew, etc.

In the operations repository, the {{Ops file|roles/core/certificates}} unit is responsible to deploy <code>nasqueron-vault-ca.crt</code>, used to authenticate for *.nasqueron.drake or IP services. For exemple, Vault clients require a certificate they can validate with a chain from that intermediate CA. Note as we don't currently use the root CA, a fullchain is probably not needed for those services.

=== Rollover a new certificate ===
If <code>nasqueron-vault-ca.crt</code> is updated or if we want instead to provision the root certificate, the following repository need to be updated and services to be redeployed:

{| class="wikitable sortable"
|+ Where to update certificate?
|-
! Repository !! Path !! Purpose !! Deployment instructions
|-
| operations || {{Ops file|roles/core/certificates/files/}} || Trust internal resources on every server || Salt: <code>salt '*' state.apply roles/core/certificates</code>
|-
| docker-airflow || [https://devcentral.nasqueron.org/source/docker-airflow/browse/main/files files/] || Connect to Vault from Airflow Python code || Docker: build and deploy new image nasqueron/airflow on Dwellers
|}

Deployment instructions are up-to-date when newly added on that table. Afterwards, they should be checked against current procedures.

== Special considerations ==
=== New server ===
Let's encrypt client is available on [[Ysul]] (natively) and [[Dwellers]] (as a wrapper script for a Docker container).

Fill a task in Servers component, subscribe Sandlayth and Dereckson to deploy it on a new server.

A salt state would be nice for such purpose. There is work-in-progress on that matter through {{D|3248}}.

=== Internationalized domain names ===
==== Punycode conversion ====
Both for web server configuration and certificate authority, name must be converted to Punycode ([https://www.ietf.org/rfc/rfc3492.txt RFC 3492]):
https://www.punycoder.com/

==== Let's encrypt support ====

Let's encrypt has supported IDN since 2016<ref>https://letsencrypt.org/2016/10/21/introducing-idn-support.html</ref>. We use it for dægrefn.nasqueron.org certificate.

Previously, they were afraid: attackers could register a domain with a Cyrillic character matching a real domains. As some people consider it's the responsibility of the CA to mitigate such risks, the feature has been several times postponed.

==== StartSSL ====

StartSSL is not in activity anymore. It was used at Nasqueron when Let's Encrypt didn't support IDN.

== Troubleshoot ==
=== Acme DNS ===
==== Can't reach ACME DNS API / HTTP Error 403: Forbidden ====
AcmeDNS API access is restricted to IPs addresses set in {{Ops file|roles/webserver-core/nginx/files/includes/geo_nasqueron}}.

If you got a 403 by running acme.sh (only visible in --debug mode):
* If the IP address is missing from geo_nasqueron, add it
* If the IP address is already there, check on the server if /etc/nginx/includes/geo_nasqueron matches
** Yes -> nginx must be reloaded (nginx -t reload)
** No -> Deploy with <code>salt docker-002 state.sls_id /etc/nginx/includes/geo_nasqueron roles/webserver-core/nginx</code>

Another solution is a specific routing to reach the API:
For Hervil, add a route to use router-001 gateway: route add 51.255.124.8/30 172.27.27.1

==== Troubleshoot records in ACME DNS database ====
To check the acme.nasqueron.org subdomain matching your _acme-challenge subdomain: <code>nslookup -type=TXT _acme-challenge.<subdomain>.nasqueron.org</code>

The DNS server to store acme.nasqueron.org records uses a sqlite3 database. To connect and check your parameters:

$ ssh docker-002
$ sqlite3 /srv/acme/lib/acme-dns.db
SELECT Username, Password FROM records WHERE Subdomain = "<DNS answer>"

==== Credential lost for a ACME DNS subdomain ====

Passwords are [https://github.com/joohoi/acme-dns/blob/9c6ca258e1d57e7441b60db4474a68c36356dae2/validation.go hashed with bcrypt], so it's not possible to recover them.

In that case, there are two options:

* if you prefer edit the DNS record, delete acme.sh information for the domain renewal then issue a new certificate, that will register a new account
* if you prefer edit the ACME DNS database, encrypt a new password in bcrypt and update it with sqlite

Certificates requests are limited by Let's Encrypt, updating the password avoids to reach that quota.

You can generate a uuid and pass it to bcrypt, or register a new password (and update DNS):

$ ssh docker-002
$ export SQLITE_HISTORY=/dev/null
$ sqlite3 /srv/acme/. lib/acme-dns.db
UPDATE records SET Password = "<new bcrypt hash>" WHERE Subdomain = "<DNS answer>"

You can generate any arbitrary password with any random tool.
It seems UUIDv4 is used by the client code, if you want one, you can use <code>uuidgen -r</code>.

To get password hash with bcrypt, you can use psysh on WindRiver:

$ psysh
password_hash("alpha", PASSWORD_BCRYPT)
$2y$10$N17lnt09YFIoVY4025AmCuGrlmA.ZiThm.RDufcakGq9.3IK4xVcW

Replace $2y$ by $2a$ as x and y are unique for PHP, the rest of the world uses $2a$.

== Notes & references ==
<references />

Operations grimoire/TLS certificates

2024-11-05T19:58:22Z

DorianWinty: /* Deploying of the certificates */

TLS certificates should be used for every service we provide, so we can encrypt the communications.

== Let's Encrypt ==
=== Certbot commands ===

'''acme-v02 migration.''' If you've a complaint acme-v01.api. isn't available, add <code>--server https://acme-v02.api.letsencrypt.org/directory</code>.

'''T1505 alignment.''' We don't use anymore Certbot as a Docker container on paas-docker role. As such, all paths are now unified. Symlinks have been added to old /srv paths to help transition.

==== Generate a certificate ====
<code>certbot certonly -a webroot --webroot-path=/var/letsencrypt-auto --deploy-hook "service nginx reload" -d foo.nasqueron.org</code>

<code>certbot certonly -a webroot --webroot-path=/var/letsencrypt-auto --deploy-hook "systemctl reload nginx" -d foo.nasqueron.org</code>

==== Generate a certificate for several sites ====
<code>-d foo.nasqueron.org -d bar.nasqueron.org</code>

If a certificate for foo already existed, it will offer to extend it to a new alternative name, which is probably a good idea.

==== Generate a certificate through DNS ====
DNS can be used to generate certificates for domains. For example, the Openfire XMPP certificate is generated like this:

certbot certonly --server https://acme-v02.api.letsencrypt.org/directory --manual --manual-auth-hook /etc/letsencrypt/acme-dns-auth --preferred-challenges dns --debug-challenges -d xmpp.nasqueron.org -d nasqueron.org -d conference.nasqueron.org

From December 2023, you can use this command on FreeBSD servers where {{T|1505}} Let's Encrypt standard installation have been deployed:

certbot certonly --manual --manual-auth-hook /usr/local/etc/letsencrypt/acme-dns-auth --preferred-challenges dns --debug-challenges -d subdomain.nasqueron.org

This uses a specialized DNS server deployed on our Docker PaaS to serve dynamic TLS records under .acme.nasqueron.org, {{Ops file|roles/paas-docker/containers/acme_dns.sls}}.

Support files were previously '''only deployed to paas-docker role''' through {{Ops file|roles/paas-docker/letsencrypt/files}}.

==== Renew all certificates ====
<code>certbot renew</code>

==== Installation on nginx ====
===== Allow Let's encrypt validation =====

include includes/letsencrypt;

This will use {{Ops file|roles/webserver-core/nginx/files/includes/letsencrypt}} nginx configuration.

===== Serve TLS certificate =====

include includes/tls;
ssl_certificate /srv/letsencrypt/etc/live/xmpp.nasqueron.org/fullchain.pem;
ssl_certificate_key /srv/letsencrypt/etc/live/xmpp.nasqueron.org/privkey.pem;

This will configure a compromise between security and compatibility, based on Intermediate Mozilla SSL config<ref>https://ssl-config.mozilla.org/</ref>. The current configuration is served by {{Ops file|roles/webserver-core/nginx/files/includes/tls}}.

If you prefer to restrict the resource for TLS 1.3, and accepts to block legacy clients, you can also use with {{D|3251}}:

include includes/tls-modern-only;
ssl_certificate /srv/letsencrypt/etc/live/xmpp.nasqueron.org/fullchain.pem;
ssl_certificate_key /srv/letsencrypt/etc/live/xmpp.nasqueron.org/privkey.pem;

==== Edit renewal hook ====
In /etc/letsencrypt/renewal or /usr/local/etc/letsencrypt/renewal you can edit this section to customize the command to run after renewal:

[renewalparams]
renew_hook = systemctl reload nginx

That can be applied in batch when nothing is set: <code><nowiki>gsed -i "s/\[\[webroot_map]]/renew_hook = systemctl reload nginx\n\n[[webroot_map]]/g" *.conf</nowiki></code>

=== Install certbot on CentOS 10 stream ===

Packages for Python 3.12 currently used as of 2024-08-04 on CentOS 10 stream can be fetched from Fedora 40 repository.

Installation on Dwellers was done like this:

mkdir /tmp/certbot && cd /tmp/certbot
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/c/certbot-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-acme-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-certbot-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-configargparse-1.7-3.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-josepy-1.13.0-8.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-parsedatetime-2.6-12.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pyOpenSSL-23.2.0-3.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pyrfc3339-1.1-18.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pytz-2024.1-1.fc40.noarch.rpm
dnf install ./*.rpm

=== acme.sh ===

acme.sh should be run with the called "acme" with a
sudo su - acme

==== Generate a certificate ====

==== Generate a certificate for several sites ====

==== Generate a certificate through DNS ====

For DNS certificate, we need to start the certificate generation, edit the DNS records to add the certificate verification. And launch the end of the certificate generation
export ACMEDNS_BASE_URL="https://acme.nasqueron.org"
acme.sh --issue --dns dns_acmedns -d example.com --server letsencrypt

==== Regenerate certificate how are from certbot ====

export ACMEDNS_BASE_URL="https://acme.nasqueron.org"
export ACMEDNS_USERNAME="<username>"
export ACMEDNS_PASSWORD="<password>"
export ACMEDNS_SUBDOMAIN="<subdomain>"

acme.sh --issue --dns dns_acmedns -d example.com --server letsencrypt

==== Renew all certificates ====

For the certificates renewal, anything is needed the cron tab will try to renew them every day "x" day before the expiracy

==== Deploying of the certificates ====

acme.sh --install-cert -d exemple.nasqueron.org \
--cert-file /var/certificates/exemple.nasqueron.org/cert.pem
--key-file /var/certificates/exemple.nasqueron.org/key.pem \
--fullchain-file /var/certificates/exemple.nasqueron.org/cert.pem \

==== on nginx ====

== Nasqueron PKI ==
=== Principles ===
Internal PKI material are stored in Vault, see [[Operations grimoire/Vault]] for information about how to generate, renew, etc.

In the operations repository, the {{Ops file|roles/core/certificates}} unit is responsible to deploy <code>nasqueron-vault-ca.crt</code>, used to authenticate for *.nasqueron.drake or IP services. For exemple, Vault clients require a certificate they can validate with a chain from that intermediate CA. Note as we don't currently use the root CA, a fullchain is probably not needed for those services.

=== Rollover a new certificate ===
If <code>nasqueron-vault-ca.crt</code> is updated or if we want instead to provision the root certificate, the following repository need to be updated and services to be redeployed:

{| class="wikitable sortable"
|+ Where to update certificate?
|-
! Repository !! Path !! Purpose !! Deployment instructions
|-
| operations || {{Ops file|roles/core/certificates/files/}} || Trust internal resources on every server || Salt: <code>salt '*' state.apply roles/core/certificates</code>
|-
| docker-airflow || [https://devcentral.nasqueron.org/source/docker-airflow/browse/main/files files/] || Connect to Vault from Airflow Python code || Docker: build and deploy new image nasqueron/airflow on Dwellers
|}

Deployment instructions are up-to-date when newly added on that table. Afterwards, they should be checked against current procedures.

== Special considerations ==
=== New server ===
Let's encrypt client is available on [[Ysul]] (natively) and [[Dwellers]] (as a wrapper script for a Docker container).

Fill a task in Servers component, subscribe Sandlayth and Dereckson to deploy it on a new server.

A salt state would be nice for such purpose. There is work-in-progress on that matter through {{D|3248}}.

=== Internationalized domain names ===
==== Punycode conversion ====
Both for web server configuration and certificate authority, name must be converted to Punycode ([https://www.ietf.org/rfc/rfc3492.txt RFC 3492]):
https://www.punycoder.com/

==== Let's encrypt support ====

Let's encrypt has supported IDN since 2016<ref>https://letsencrypt.org/2016/10/21/introducing-idn-support.html</ref>. We use it for dægrefn.nasqueron.org certificate.

Previously, they were afraid: attackers could register a domain with a Cyrillic character matching a real domains. As some people consider it's the responsibility of the CA to mitigate such risks, the feature has been several times postponed.

==== StartSSL ====

StartSSL is not in activity anymore. It was used at Nasqueron when Let's Encrypt didn't support IDN.

== Troubleshoot ==
=== Acme DNS ===
==== Can't reach ACME DNS API / HTTP Error 403: Forbidden ====
AcmeDNS API access is restricted to IPs addresses set in {{Ops file|roles/webserver-core/nginx/files/includes/geo_nasqueron}}.

If you got a 403 by running acme.sh (only visible in --debug mode):
* If the IP address is missing from geo_nasqueron, add it
* If the IP address is already there, check on the server if /etc/nginx/includes/geo_nasqueron matches
** Yes -> nginx must be reloaded (nginx -t reload)
** No -> Deploy with <code>salt docker-002 state.sls_id /etc/nginx/includes/geo_nasqueron roles/webserver-core/nginx</code>

Another solution is a specific routing to reach the API:
For Hervil, add a route to use router-001 gateway: route add 51.255.124.8/30 172.27.27.1

==== Troubleshoot records in ACME DNS database ====
To check the acme.nasqueron.org subdomain matching your _acme-challenge subdomain: <code>nslookup -type=TXT _acme-challenge.<subdomain>.nasqueron.org</code>

The DNS server to store acme.nasqueron.org records uses a sqlite3 database. To connect and check your parameters:

$ ssh docker-002
$ sqlite3 /srv/acme/lib/acme-dns.db
SELECT Username, Password FROM records WHERE Subdomain = "<DNS answer>"

==== Credential lost for a ACME DNS subdomain ====

Passwords are [https://github.com/joohoi/acme-dns/blob/9c6ca258e1d57e7441b60db4474a68c36356dae2/validation.go hashed with bcrypt], so it's not possible to recover them.

In that case, there are two options:

* if you prefer edit the DNS record, delete acme.sh information for the domain renewal then issue a new certificate, that will register a new account
* if you prefer edit the ACME DNS database, encrypt a new password in bcrypt and update it with sqlite

Certificates requests are limited by Let's Encrypt, updating the password avoids to reach that quota.

You can generate a uuid and pass it to bcrypt, or register a new password (and update DNS):

$ ssh docker-002
$ export SQLITE_HISTORY=/dev/null
$ sqlite3 /srv/acme/. lib/acme-dns.db
UPDATE records SET Password = "<new bcrypt hash>" WHERE Subdomain = "<DNS answer>"

You can generate any arbitrary password with any random tool.
It seems UUIDv4 is used by the client code, if you want one, you can use <code>uuidgen -r</code>.

To get password hash with bcrypt, you can use psysh on WindRiver:

$ psysh
password_hash("alpha", PASSWORD_BCRYPT)
$2y$10$N17lnt09YFIoVY4025AmCuGrlmA.ZiThm.RDufcakGq9.3IK4xVcW

Replace $2y$ by $2a$ as x and y are unique for PHP, the rest of the world uses $2a$.

== Notes & references ==
<references />

Operations grimoire/TLS certificates

2024-11-05T19:47:04Z

DorianWinty: /* Regenerate certificate how are from certbot */

TLS certificates should be used for every service we provide, so we can encrypt the communications.

== Let's Encrypt ==
=== Certbot commands ===

'''acme-v02 migration.''' If you've a complaint acme-v01.api. isn't available, add <code>--server https://acme-v02.api.letsencrypt.org/directory</code>.

'''T1505 alignment.''' We don't use anymore Certbot as a Docker container on paas-docker role. As such, all paths are now unified. Symlinks have been added to old /srv paths to help transition.

==== Generate a certificate ====
<code>certbot certonly -a webroot --webroot-path=/var/letsencrypt-auto --deploy-hook "service nginx reload" -d foo.nasqueron.org</code>

<code>certbot certonly -a webroot --webroot-path=/var/letsencrypt-auto --deploy-hook "systemctl reload nginx" -d foo.nasqueron.org</code>

==== Generate a certificate for several sites ====
<code>-d foo.nasqueron.org -d bar.nasqueron.org</code>

If a certificate for foo already existed, it will offer to extend it to a new alternative name, which is probably a good idea.

==== Generate a certificate through DNS ====
DNS can be used to generate certificates for domains. For example, the Openfire XMPP certificate is generated like this:

certbot certonly --server https://acme-v02.api.letsencrypt.org/directory --manual --manual-auth-hook /etc/letsencrypt/acme-dns-auth --preferred-challenges dns --debug-challenges -d xmpp.nasqueron.org -d nasqueron.org -d conference.nasqueron.org

From December 2023, you can use this command on FreeBSD servers where {{T|1505}} Let's Encrypt standard installation have been deployed:

certbot certonly --manual --manual-auth-hook /usr/local/etc/letsencrypt/acme-dns-auth --preferred-challenges dns --debug-challenges -d subdomain.nasqueron.org

This uses a specialized DNS server deployed on our Docker PaaS to serve dynamic TLS records under .acme.nasqueron.org, {{Ops file|roles/paas-docker/containers/acme_dns.sls}}.

Support files were previously '''only deployed to paas-docker role''' through {{Ops file|roles/paas-docker/letsencrypt/files}}.

==== Renew all certificates ====
<code>certbot renew</code>

==== Installation on nginx ====
===== Allow Let's encrypt validation =====

include includes/letsencrypt;

This will use {{Ops file|roles/webserver-core/nginx/files/includes/letsencrypt}} nginx configuration.

===== Serve TLS certificate =====

include includes/tls;
ssl_certificate /srv/letsencrypt/etc/live/xmpp.nasqueron.org/fullchain.pem;
ssl_certificate_key /srv/letsencrypt/etc/live/xmpp.nasqueron.org/privkey.pem;

This will configure a compromise between security and compatibility, based on Intermediate Mozilla SSL config<ref>https://ssl-config.mozilla.org/</ref>. The current configuration is served by {{Ops file|roles/webserver-core/nginx/files/includes/tls}}.

If you prefer to restrict the resource for TLS 1.3, and accepts to block legacy clients, you can also use with {{D|3251}}:

include includes/tls-modern-only;
ssl_certificate /srv/letsencrypt/etc/live/xmpp.nasqueron.org/fullchain.pem;
ssl_certificate_key /srv/letsencrypt/etc/live/xmpp.nasqueron.org/privkey.pem;

==== Edit renewal hook ====
In /etc/letsencrypt/renewal or /usr/local/etc/letsencrypt/renewal you can edit this section to customize the command to run after renewal:

[renewalparams]
renew_hook = systemctl reload nginx

That can be applied in batch when nothing is set: <code><nowiki>gsed -i "s/\[\[webroot_map]]/renew_hook = systemctl reload nginx\n\n[[webroot_map]]/g" *.conf</nowiki></code>

=== Install certbot on CentOS 10 stream ===

Packages for Python 3.12 currently used as of 2024-08-04 on CentOS 10 stream can be fetched from Fedora 40 repository.

Installation on Dwellers was done like this:

mkdir /tmp/certbot && cd /tmp/certbot
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/c/certbot-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-acme-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-certbot-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-configargparse-1.7-3.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-josepy-1.13.0-8.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-parsedatetime-2.6-12.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pyOpenSSL-23.2.0-3.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pyrfc3339-1.1-18.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pytz-2024.1-1.fc40.noarch.rpm
dnf install ./*.rpm

=== acme.sh ===

acme.sh should be run with the called "acme" with a
sudo su - acme

==== Generate a certificate ====

==== Generate a certificate for several sites ====

==== Generate a certificate through DNS ====

For DNS certificate, we need to start the certificate generation, edit the DNS records to add the certificate verification. And launch the end of the certificate generation
export ACMEDNS_BASE_URL="https://acme.nasqueron.org"
acme.sh --issue --dns dns_acmedns -d example.com --server letsencrypt

==== Regenerate certificate how are from certbot ====

export ACMEDNS_BASE_URL="https://acme.nasqueron.org"
export ACMEDNS_USERNAME="<username>"
export ACMEDNS_PASSWORD="<password>"
export ACMEDNS_SUBDOMAIN="<subdomain>"

acme.sh --issue --dns dns_acmedns -d example.com --server letsencrypt

==== Renew all certificates ====

For the certificates renewal, anything is needed the cron tab will try to renew them every day "x" day before the expiracy

==== Deploying of the certificates ====

==== on nginx ====

== Nasqueron PKI ==
=== Principles ===
Internal PKI material are stored in Vault, see [[Operations grimoire/Vault]] for information about how to generate, renew, etc.

In the operations repository, the {{Ops file|roles/core/certificates}} unit is responsible to deploy <code>nasqueron-vault-ca.crt</code>, used to authenticate for *.nasqueron.drake or IP services. For exemple, Vault clients require a certificate they can validate with a chain from that intermediate CA. Note as we don't currently use the root CA, a fullchain is probably not needed for those services.

=== Rollover a new certificate ===
If <code>nasqueron-vault-ca.crt</code> is updated or if we want instead to provision the root certificate, the following repository need to be updated and services to be redeployed:

{| class="wikitable sortable"
|+ Where to update certificate?
|-
! Repository !! Path !! Purpose !! Deployment instructions
|-
| operations || {{Ops file|roles/core/certificates/files/}} || Trust internal resources on every server || Salt: <code>salt '*' state.apply roles/core/certificates</code>
|-
| docker-airflow || [https://devcentral.nasqueron.org/source/docker-airflow/browse/main/files files/] || Connect to Vault from Airflow Python code || Docker: build and deploy new image nasqueron/airflow on Dwellers
|}

Deployment instructions are up-to-date when newly added on that table. Afterwards, they should be checked against current procedures.

== Special considerations ==
=== New server ===
Let's encrypt client is available on [[Ysul]] (natively) and [[Dwellers]] (as a wrapper script for a Docker container).

Fill a task in Servers component, subscribe Sandlayth and Dereckson to deploy it on a new server.

A salt state would be nice for such purpose. There is work-in-progress on that matter through {{D|3248}}.

=== Internationalized domain names ===
==== Punycode conversion ====
Both for web server configuration and certificate authority, name must be converted to Punycode ([https://www.ietf.org/rfc/rfc3492.txt RFC 3492]):
https://www.punycoder.com/

==== Let's encrypt support ====

Let's encrypt has supported IDN since 2016<ref>https://letsencrypt.org/2016/10/21/introducing-idn-support.html</ref>. We use it for dægrefn.nasqueron.org certificate.

Previously, they were afraid: attackers could register a domain with a Cyrillic character matching a real domains. As some people consider it's the responsibility of the CA to mitigate such risks, the feature has been several times postponed.

==== StartSSL ====

StartSSL is not in activity anymore. It was used at Nasqueron when Let's Encrypt didn't support IDN.

== Troubleshoot ==
=== Acme DNS ===
==== Can't reach ACME DNS API / HTTP Error 403: Forbidden ====
AcmeDNS API access is restricted to IPs addresses set in {{Ops file|roles/webserver-core/nginx/files/includes/geo_nasqueron}}.

If you got a 403 by running acme.sh (only visible in --debug mode):
* If the IP address is missing from geo_nasqueron, add it
* If the IP address is already there, check on the server if /etc/nginx/includes/geo_nasqueron matches
** Yes -> nginx must be reloaded (nginx -t reload)
** No -> Deploy with <code>salt docker-002 state.sls_id /etc/nginx/includes/geo_nasqueron roles/webserver-core/nginx</code>

Another solution is a specific routing to reach the API:
For Hervil, add a route to use router-001 gateway: route add 51.255.124.8/30 172.27.27.1

==== Troubleshoot records in ACME DNS database ====
To check the acme.nasqueron.org subdomain matching your _acme-challenge subdomain: <code>nslookup -type=TXT _acme-challenge.<subdomain>.nasqueron.org</code>

The DNS server to store acme.nasqueron.org records uses a sqlite3 database. To connect and check your parameters:

$ ssh docker-002
$ sqlite3 /srv/acme/lib/acme-dns.db
SELECT Username, Password FROM records WHERE Subdomain = "<DNS answer>"

==== Credential lost for a ACME DNS subdomain ====

Passwords are [https://github.com/joohoi/acme-dns/blob/9c6ca258e1d57e7441b60db4474a68c36356dae2/validation.go hashed with bcrypt], so it's not possible to recover them.

In that case, there are two options:

* if you prefer edit the DNS record, delete acme.sh information for the domain renewal then issue a new certificate, that will register a new account
* if you prefer edit the ACME DNS database, encrypt a new password in bcrypt and update it with sqlite

Certificates requests are limited by Let's Encrypt, updating the password avoids to reach that quota.

You can generate a uuid and pass it to bcrypt, or register a new password (and update DNS):

$ ssh docker-002
$ export SQLITE_HISTORY=/dev/null
$ sqlite3 /srv/acme/. lib/acme-dns.db
UPDATE records SET Password = "<new bcrypt hash>" WHERE Subdomain = "<DNS answer>"

You can generate any arbitrary password with any random tool.
It seems UUIDv4 is used by the client code, if you want one, you can use <code>uuidgen -r</code>.

To get password hash with bcrypt, you can use psysh on WindRiver:

$ psysh
password_hash("alpha", PASSWORD_BCRYPT)
$2y$10$N17lnt09YFIoVY4025AmCuGrlmA.ZiThm.RDufcakGq9.3IK4xVcW

Replace $2y$ by $2a$ as x and y are unique for PHP, the rest of the world uses $2a$.

== Notes & references ==
<references />

Operations grimoire/TLS certificates

2024-11-05T19:45:07Z

DorianWinty: /* Regenerate certificate how are from certbot */

TLS certificates should be used for every service we provide, so we can encrypt the communications.

== Let's Encrypt ==
=== Certbot commands ===

'''acme-v02 migration.''' If you've a complaint acme-v01.api. isn't available, add <code>--server https://acme-v02.api.letsencrypt.org/directory</code>.

'''T1505 alignment.''' We don't use anymore Certbot as a Docker container on paas-docker role. As such, all paths are now unified. Symlinks have been added to old /srv paths to help transition.

==== Generate a certificate ====
<code>certbot certonly -a webroot --webroot-path=/var/letsencrypt-auto --deploy-hook "service nginx reload" -d foo.nasqueron.org</code>

<code>certbot certonly -a webroot --webroot-path=/var/letsencrypt-auto --deploy-hook "systemctl reload nginx" -d foo.nasqueron.org</code>

==== Generate a certificate for several sites ====
<code>-d foo.nasqueron.org -d bar.nasqueron.org</code>

If a certificate for foo already existed, it will offer to extend it to a new alternative name, which is probably a good idea.

==== Generate a certificate through DNS ====
DNS can be used to generate certificates for domains. For example, the Openfire XMPP certificate is generated like this:

certbot certonly --server https://acme-v02.api.letsencrypt.org/directory --manual --manual-auth-hook /etc/letsencrypt/acme-dns-auth --preferred-challenges dns --debug-challenges -d xmpp.nasqueron.org -d nasqueron.org -d conference.nasqueron.org

From December 2023, you can use this command on FreeBSD servers where {{T|1505}} Let's Encrypt standard installation have been deployed:

certbot certonly --manual --manual-auth-hook /usr/local/etc/letsencrypt/acme-dns-auth --preferred-challenges dns --debug-challenges -d subdomain.nasqueron.org

This uses a specialized DNS server deployed on our Docker PaaS to serve dynamic TLS records under .acme.nasqueron.org, {{Ops file|roles/paas-docker/containers/acme_dns.sls}}.

Support files were previously '''only deployed to paas-docker role''' through {{Ops file|roles/paas-docker/letsencrypt/files}}.

==== Renew all certificates ====
<code>certbot renew</code>

==== Installation on nginx ====
===== Allow Let's encrypt validation =====

include includes/letsencrypt;

This will use {{Ops file|roles/webserver-core/nginx/files/includes/letsencrypt}} nginx configuration.

===== Serve TLS certificate =====

include includes/tls;
ssl_certificate /srv/letsencrypt/etc/live/xmpp.nasqueron.org/fullchain.pem;
ssl_certificate_key /srv/letsencrypt/etc/live/xmpp.nasqueron.org/privkey.pem;

This will configure a compromise between security and compatibility, based on Intermediate Mozilla SSL config<ref>https://ssl-config.mozilla.org/</ref>. The current configuration is served by {{Ops file|roles/webserver-core/nginx/files/includes/tls}}.

If you prefer to restrict the resource for TLS 1.3, and accepts to block legacy clients, you can also use with {{D|3251}}:

include includes/tls-modern-only;
ssl_certificate /srv/letsencrypt/etc/live/xmpp.nasqueron.org/fullchain.pem;
ssl_certificate_key /srv/letsencrypt/etc/live/xmpp.nasqueron.org/privkey.pem;

==== Edit renewal hook ====
In /etc/letsencrypt/renewal or /usr/local/etc/letsencrypt/renewal you can edit this section to customize the command to run after renewal:

[renewalparams]
renew_hook = systemctl reload nginx

That can be applied in batch when nothing is set: <code><nowiki>gsed -i "s/\[\[webroot_map]]/renew_hook = systemctl reload nginx\n\n[[webroot_map]]/g" *.conf</nowiki></code>

=== Install certbot on CentOS 10 stream ===

Packages for Python 3.12 currently used as of 2024-08-04 on CentOS 10 stream can be fetched from Fedora 40 repository.

Installation on Dwellers was done like this:

mkdir /tmp/certbot && cd /tmp/certbot
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/c/certbot-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-acme-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-certbot-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-configargparse-1.7-3.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-josepy-1.13.0-8.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-parsedatetime-2.6-12.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pyOpenSSL-23.2.0-3.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pyrfc3339-1.1-18.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pytz-2024.1-1.fc40.noarch.rpm
dnf install ./*.rpm

=== acme.sh ===

acme.sh should be run with the called "acme" with a
sudo su - acme

==== Generate a certificate ====

==== Generate a certificate for several sites ====

==== Generate a certificate through DNS ====

For DNS certificate, we need to start the certificate generation, edit the DNS records to add the certificate verification. And launch the end of the certificate generation
export ACMEDNS_BASE_URL="https://acme.nasqueron.org"
acme.sh --issue --dns dns_acmedns -d example.com --server letsencrypt

==== Regenerate certificate how are from certbot ====

export ACMEDNS_BASE_URL="https://auth.acme-dns.io"
export ACMEDNS_USERNAME="<username>"
export ACMEDNS_PASSWORD="<password>"
export ACMEDNS_SUBDOMAIN="<subdomain>"

acme.sh --issue --dns dns_acmedns -d example.com --server letsencrypt

==== Renew all certificates ====

For the certificates renewal, anything is needed the cron tab will try to renew them every day "x" day before the expiracy

==== Deploying of the certificates ====

==== on nginx ====

== Nasqueron PKI ==
=== Principles ===
Internal PKI material are stored in Vault, see [[Operations grimoire/Vault]] for information about how to generate, renew, etc.

In the operations repository, the {{Ops file|roles/core/certificates}} unit is responsible to deploy <code>nasqueron-vault-ca.crt</code>, used to authenticate for *.nasqueron.drake or IP services. For exemple, Vault clients require a certificate they can validate with a chain from that intermediate CA. Note as we don't currently use the root CA, a fullchain is probably not needed for those services.

=== Rollover a new certificate ===
If <code>nasqueron-vault-ca.crt</code> is updated or if we want instead to provision the root certificate, the following repository need to be updated and services to be redeployed:

{| class="wikitable sortable"
|+ Where to update certificate?
|-
! Repository !! Path !! Purpose !! Deployment instructions
|-
| operations || {{Ops file|roles/core/certificates/files/}} || Trust internal resources on every server || Salt: <code>salt '*' state.apply roles/core/certificates</code>
|-
| docker-airflow || [https://devcentral.nasqueron.org/source/docker-airflow/browse/main/files files/] || Connect to Vault from Airflow Python code || Docker: build and deploy new image nasqueron/airflow on Dwellers
|}

Deployment instructions are up-to-date when newly added on that table. Afterwards, they should be checked against current procedures.

== Special considerations ==
=== New server ===
Let's encrypt client is available on [[Ysul]] (natively) and [[Dwellers]] (as a wrapper script for a Docker container).

Fill a task in Servers component, subscribe Sandlayth and Dereckson to deploy it on a new server.

A salt state would be nice for such purpose. There is work-in-progress on that matter through {{D|3248}}.

=== Internationalized domain names ===
==== Punycode conversion ====
Both for web server configuration and certificate authority, name must be converted to Punycode ([https://www.ietf.org/rfc/rfc3492.txt RFC 3492]):
https://www.punycoder.com/

==== Let's encrypt support ====

Let's encrypt has supported IDN since 2016<ref>https://letsencrypt.org/2016/10/21/introducing-idn-support.html</ref>. We use it for dægrefn.nasqueron.org certificate.

Previously, they were afraid: attackers could register a domain with a Cyrillic character matching a real domains. As some people consider it's the responsibility of the CA to mitigate such risks, the feature has been several times postponed.

==== StartSSL ====

StartSSL is not in activity anymore. It was used at Nasqueron when Let's Encrypt didn't support IDN.

== Troubleshoot ==
=== Acme DNS ===
==== Can't reach ACME DNS API / HTTP Error 403: Forbidden ====
AcmeDNS API access is restricted to IPs addresses set in {{Ops file|roles/webserver-core/nginx/files/includes/geo_nasqueron}}.

If you got a 403 by running acme.sh (only visible in --debug mode):
* If the IP address is missing from geo_nasqueron, add it
* If the IP address is already there, check on the server if /etc/nginx/includes/geo_nasqueron matches
** Yes -> nginx must be reloaded (nginx -t reload)
** No -> Deploy with <code>salt docker-002 state.sls_id /etc/nginx/includes/geo_nasqueron roles/webserver-core/nginx</code>

Another solution is a specific routing to reach the API:
For Hervil, add a route to use router-001 gateway: route add 51.255.124.8/30 172.27.27.1

==== Troubleshoot records in ACME DNS database ====
To check the acme.nasqueron.org subdomain matching your _acme-challenge subdomain: <code>nslookup -type=TXT _acme-challenge.<subdomain>.nasqueron.org</code>

The DNS server to store acme.nasqueron.org records uses a sqlite3 database. To connect and check your parameters:

$ ssh docker-002
$ sqlite3 /srv/acme/lib/acme-dns.db
SELECT Username, Password FROM records WHERE Subdomain = "<DNS answer>"

==== Credential lost for a ACME DNS subdomain ====

Passwords are [https://github.com/joohoi/acme-dns/blob/9c6ca258e1d57e7441b60db4474a68c36356dae2/validation.go hashed with bcrypt], so it's not possible to recover them.

In that case, there are two options:

* if you prefer edit the DNS record, delete acme.sh information for the domain renewal then issue a new certificate, that will register a new account
* if you prefer edit the ACME DNS database, encrypt a new password in bcrypt and update it with sqlite

Certificates requests are limited by Let's Encrypt, updating the password avoids to reach that quota.

You can generate a uuid and pass it to bcrypt, or register a new password (and update DNS):

$ ssh docker-002
$ export SQLITE_HISTORY=/dev/null
$ sqlite3 /srv/acme/. lib/acme-dns.db
UPDATE records SET Password = "<new bcrypt hash>" WHERE Subdomain = "<DNS answer>"

You can generate any arbitrary password with any random tool.
It seems UUIDv4 is used by the client code, if you want one, you can use <code>uuidgen -r</code>.

To get password hash with bcrypt, you can use psysh on WindRiver:

$ psysh
password_hash("alpha", PASSWORD_BCRYPT)
$2y$10$N17lnt09YFIoVY4025AmCuGrlmA.ZiThm.RDufcakGq9.3IK4xVcW

Replace $2y$ by $2a$ as x and y are unique for PHP, the rest of the world uses $2a$.

== Notes & references ==
<references />

Operations grimoire/TLS certificates

2024-11-05T19:43:11Z

DorianWinty: /* Generate a certificate through DNS */

TLS certificates should be used for every service we provide, so we can encrypt the communications.

== Let's Encrypt ==
=== Certbot commands ===

'''acme-v02 migration.''' If you've a complaint acme-v01.api. isn't available, add <code>--server https://acme-v02.api.letsencrypt.org/directory</code>.

'''T1505 alignment.''' We don't use anymore Certbot as a Docker container on paas-docker role. As such, all paths are now unified. Symlinks have been added to old /srv paths to help transition.

==== Generate a certificate ====
<code>certbot certonly -a webroot --webroot-path=/var/letsencrypt-auto --deploy-hook "service nginx reload" -d foo.nasqueron.org</code>

<code>certbot certonly -a webroot --webroot-path=/var/letsencrypt-auto --deploy-hook "systemctl reload nginx" -d foo.nasqueron.org</code>

==== Generate a certificate for several sites ====
<code>-d foo.nasqueron.org -d bar.nasqueron.org</code>

If a certificate for foo already existed, it will offer to extend it to a new alternative name, which is probably a good idea.

==== Generate a certificate through DNS ====
DNS can be used to generate certificates for domains. For example, the Openfire XMPP certificate is generated like this:

certbot certonly --server https://acme-v02.api.letsencrypt.org/directory --manual --manual-auth-hook /etc/letsencrypt/acme-dns-auth --preferred-challenges dns --debug-challenges -d xmpp.nasqueron.org -d nasqueron.org -d conference.nasqueron.org

From December 2023, you can use this command on FreeBSD servers where {{T|1505}} Let's Encrypt standard installation have been deployed:

certbot certonly --manual --manual-auth-hook /usr/local/etc/letsencrypt/acme-dns-auth --preferred-challenges dns --debug-challenges -d subdomain.nasqueron.org

This uses a specialized DNS server deployed on our Docker PaaS to serve dynamic TLS records under .acme.nasqueron.org, {{Ops file|roles/paas-docker/containers/acme_dns.sls}}.

Support files were previously '''only deployed to paas-docker role''' through {{Ops file|roles/paas-docker/letsencrypt/files}}.

==== Renew all certificates ====
<code>certbot renew</code>

==== Installation on nginx ====
===== Allow Let's encrypt validation =====

include includes/letsencrypt;

This will use {{Ops file|roles/webserver-core/nginx/files/includes/letsencrypt}} nginx configuration.

===== Serve TLS certificate =====

include includes/tls;
ssl_certificate /srv/letsencrypt/etc/live/xmpp.nasqueron.org/fullchain.pem;
ssl_certificate_key /srv/letsencrypt/etc/live/xmpp.nasqueron.org/privkey.pem;

This will configure a compromise between security and compatibility, based on Intermediate Mozilla SSL config<ref>https://ssl-config.mozilla.org/</ref>. The current configuration is served by {{Ops file|roles/webserver-core/nginx/files/includes/tls}}.

If you prefer to restrict the resource for TLS 1.3, and accepts to block legacy clients, you can also use with {{D|3251}}:

include includes/tls-modern-only;
ssl_certificate /srv/letsencrypt/etc/live/xmpp.nasqueron.org/fullchain.pem;
ssl_certificate_key /srv/letsencrypt/etc/live/xmpp.nasqueron.org/privkey.pem;

==== Edit renewal hook ====
In /etc/letsencrypt/renewal or /usr/local/etc/letsencrypt/renewal you can edit this section to customize the command to run after renewal:

[renewalparams]
renew_hook = systemctl reload nginx

That can be applied in batch when nothing is set: <code><nowiki>gsed -i "s/\[\[webroot_map]]/renew_hook = systemctl reload nginx\n\n[[webroot_map]]/g" *.conf</nowiki></code>

=== Install certbot on CentOS 10 stream ===

Packages for Python 3.12 currently used as of 2024-08-04 on CentOS 10 stream can be fetched from Fedora 40 repository.

Installation on Dwellers was done like this:

mkdir /tmp/certbot && cd /tmp/certbot
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/c/certbot-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-acme-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-certbot-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-configargparse-1.7-3.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-josepy-1.13.0-8.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-parsedatetime-2.6-12.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pyOpenSSL-23.2.0-3.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pyrfc3339-1.1-18.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pytz-2024.1-1.fc40.noarch.rpm
dnf install ./*.rpm

=== acme.sh ===

acme.sh should be run with the called "acme" with a
sudo su - acme

==== Generate a certificate ====

==== Generate a certificate for several sites ====

==== Generate a certificate through DNS ====

For DNS certificate, we need to start the certificate generation, edit the DNS records to add the certificate verification. And launch the end of the certificate generation
export ACMEDNS_BASE_URL="https://acme.nasqueron.org"
acme.sh --issue --dns dns_acmedns -d example.com --server letsencrypt

==== Regenerate certificate how are from certbot ====

==== Renew all certificates ====

For the certificates renewal, anything is needed the cron tab will try to renew them every day "x" day before the expiracy

==== Deploying of the certificates ====

==== on nginx ====

== Nasqueron PKI ==
=== Principles ===
Internal PKI material are stored in Vault, see [[Operations grimoire/Vault]] for information about how to generate, renew, etc.

In the operations repository, the {{Ops file|roles/core/certificates}} unit is responsible to deploy <code>nasqueron-vault-ca.crt</code>, used to authenticate for *.nasqueron.drake or IP services. For exemple, Vault clients require a certificate they can validate with a chain from that intermediate CA. Note as we don't currently use the root CA, a fullchain is probably not needed for those services.

=== Rollover a new certificate ===
If <code>nasqueron-vault-ca.crt</code> is updated or if we want instead to provision the root certificate, the following repository need to be updated and services to be redeployed:

{| class="wikitable sortable"
|+ Where to update certificate?
|-
! Repository !! Path !! Purpose !! Deployment instructions
|-
| operations || {{Ops file|roles/core/certificates/files/}} || Trust internal resources on every server || Salt: <code>salt '*' state.apply roles/core/certificates</code>
|-
| docker-airflow || [https://devcentral.nasqueron.org/source/docker-airflow/browse/main/files files/] || Connect to Vault from Airflow Python code || Docker: build and deploy new image nasqueron/airflow on Dwellers
|}

Deployment instructions are up-to-date when newly added on that table. Afterwards, they should be checked against current procedures.

== Special considerations ==
=== New server ===
Let's encrypt client is available on [[Ysul]] (natively) and [[Dwellers]] (as a wrapper script for a Docker container).

Fill a task in Servers component, subscribe Sandlayth and Dereckson to deploy it on a new server.

A salt state would be nice for such purpose. There is work-in-progress on that matter through {{D|3248}}.

=== Internationalized domain names ===
==== Punycode conversion ====
Both for web server configuration and certificate authority, name must be converted to Punycode ([https://www.ietf.org/rfc/rfc3492.txt RFC 3492]):
https://www.punycoder.com/

==== Let's encrypt support ====

Let's encrypt has supported IDN since 2016<ref>https://letsencrypt.org/2016/10/21/introducing-idn-support.html</ref>. We use it for dægrefn.nasqueron.org certificate.

Previously, they were afraid: attackers could register a domain with a Cyrillic character matching a real domains. As some people consider it's the responsibility of the CA to mitigate such risks, the feature has been several times postponed.

==== StartSSL ====

StartSSL is not in activity anymore. It was used at Nasqueron when Let's Encrypt didn't support IDN.

== Troubleshoot ==
=== Acme DNS ===
==== Can't reach ACME DNS API / HTTP Error 403: Forbidden ====
AcmeDNS API access is restricted to IPs addresses set in {{Ops file|roles/webserver-core/nginx/files/includes/geo_nasqueron}}.

If you got a 403 by running acme.sh (only visible in --debug mode):
* If the IP address is missing from geo_nasqueron, add it
* If the IP address is already there, check on the server if /etc/nginx/includes/geo_nasqueron matches
** Yes -> nginx must be reloaded (nginx -t reload)
** No -> Deploy with <code>salt docker-002 state.sls_id /etc/nginx/includes/geo_nasqueron roles/webserver-core/nginx</code>

Another solution is a specific routing to reach the API:
For Hervil, add a route to use router-001 gateway: route add 51.255.124.8/30 172.27.27.1

==== Troubleshoot records in ACME DNS database ====
To check the acme.nasqueron.org subdomain matching your _acme-challenge subdomain: <code>nslookup -type=TXT _acme-challenge.<subdomain>.nasqueron.org</code>

The DNS server to store acme.nasqueron.org records uses a sqlite3 database. To connect and check your parameters:

$ ssh docker-002
$ sqlite3 /srv/acme/lib/acme-dns.db
SELECT Username, Password FROM records WHERE Subdomain = "<DNS answer>"

==== Credential lost for a ACME DNS subdomain ====

Passwords are [https://github.com/joohoi/acme-dns/blob/9c6ca258e1d57e7441b60db4474a68c36356dae2/validation.go hashed with bcrypt], so it's not possible to recover them.

In that case, there are two options:

* if you prefer edit the DNS record, delete acme.sh information for the domain renewal then issue a new certificate, that will register a new account
* if you prefer edit the ACME DNS database, encrypt a new password in bcrypt and update it with sqlite

Certificates requests are limited by Let's Encrypt, updating the password avoids to reach that quota.

You can generate a uuid and pass it to bcrypt, or register a new password (and update DNS):

$ ssh docker-002
$ export SQLITE_HISTORY=/dev/null
$ sqlite3 /srv/acme/. lib/acme-dns.db
UPDATE records SET Password = "<new bcrypt hash>" WHERE Subdomain = "<DNS answer>"

You can generate any arbitrary password with any random tool.
It seems UUIDv4 is used by the client code, if you want one, you can use <code>uuidgen -r</code>.

To get password hash with bcrypt, you can use psysh on WindRiver:

$ psysh
password_hash("alpha", PASSWORD_BCRYPT)
$2y$10$N17lnt09YFIoVY4025AmCuGrlmA.ZiThm.RDufcakGq9.3IK4xVcW

Replace $2y$ by $2a$ as x and y are unique for PHP, the rest of the world uses $2a$.

== Notes & references ==
<references />

Operations grimoire/TLS certificates

2024-11-05T18:51:47Z

DorianWinty: /* Generate a certificate through DNS */

TLS certificates should be used for every service we provide, so we can encrypt the communications.

== Let's Encrypt ==
=== Certbot commands ===

'''acme-v02 migration.''' If you've a complaint acme-v01.api. isn't available, add <code>--server https://acme-v02.api.letsencrypt.org/directory</code>.

'''T1505 alignment.''' We don't use anymore Certbot as a Docker container on paas-docker role. As such, all paths are now unified. Symlinks have been added to old /srv paths to help transition.

==== Generate a certificate ====
<code>certbot certonly -a webroot --webroot-path=/var/letsencrypt-auto --deploy-hook "service nginx reload" -d foo.nasqueron.org</code>

<code>certbot certonly -a webroot --webroot-path=/var/letsencrypt-auto --deploy-hook "systemctl reload nginx" -d foo.nasqueron.org</code>

==== Generate a certificate for several sites ====
<code>-d foo.nasqueron.org -d bar.nasqueron.org</code>

If a certificate for foo already existed, it will offer to extend it to a new alternative name, which is probably a good idea.

==== Generate a certificate through DNS ====
DNS can be used to generate certificates for domains. For example, the Openfire XMPP certificate is generated like this:

certbot certonly --server https://acme-v02.api.letsencrypt.org/directory --manual --manual-auth-hook /etc/letsencrypt/acme-dns-auth --preferred-challenges dns --debug-challenges -d xmpp.nasqueron.org -d nasqueron.org -d conference.nasqueron.org

From December 2023, you can use this command on FreeBSD servers where {{T|1505}} Let's Encrypt standard installation have been deployed:

certbot certonly --manual --manual-auth-hook /usr/local/etc/letsencrypt/acme-dns-auth --preferred-challenges dns --debug-challenges -d subdomain.nasqueron.org

This uses a specialized DNS server deployed on our Docker PaaS to serve dynamic TLS records under .acme.nasqueron.org, {{Ops file|roles/paas-docker/containers/acme_dns.sls}}.

Support files were previously '''only deployed to paas-docker role''' through {{Ops file|roles/paas-docker/letsencrypt/files}}.

==== Renew all certificates ====
<code>certbot renew</code>

==== Installation on nginx ====
===== Allow Let's encrypt validation =====

include includes/letsencrypt;

This will use {{Ops file|roles/webserver-core/nginx/files/includes/letsencrypt}} nginx configuration.

===== Serve TLS certificate =====

include includes/tls;
ssl_certificate /srv/letsencrypt/etc/live/xmpp.nasqueron.org/fullchain.pem;
ssl_certificate_key /srv/letsencrypt/etc/live/xmpp.nasqueron.org/privkey.pem;

This will configure a compromise between security and compatibility, based on Intermediate Mozilla SSL config<ref>https://ssl-config.mozilla.org/</ref>. The current configuration is served by {{Ops file|roles/webserver-core/nginx/files/includes/tls}}.

If you prefer to restrict the resource for TLS 1.3, and accepts to block legacy clients, you can also use with {{D|3251}}:

include includes/tls-modern-only;
ssl_certificate /srv/letsencrypt/etc/live/xmpp.nasqueron.org/fullchain.pem;
ssl_certificate_key /srv/letsencrypt/etc/live/xmpp.nasqueron.org/privkey.pem;

==== Edit renewal hook ====
In /etc/letsencrypt/renewal or /usr/local/etc/letsencrypt/renewal you can edit this section to customize the command to run after renewal:

[renewalparams]
renew_hook = systemctl reload nginx

That can be applied in batch when nothing is set: <code><nowiki>gsed -i "s/\[\[webroot_map]]/renew_hook = systemctl reload nginx\n\n[[webroot_map]]/g" *.conf</nowiki></code>

=== Install certbot on CentOS 10 stream ===

Packages for Python 3.12 currently used as of 2024-08-04 on CentOS 10 stream can be fetched from Fedora 40 repository.

Installation on Dwellers was done like this:

mkdir /tmp/certbot && cd /tmp/certbot
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/c/certbot-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-acme-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-certbot-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-configargparse-1.7-3.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-josepy-1.13.0-8.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-parsedatetime-2.6-12.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pyOpenSSL-23.2.0-3.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pyrfc3339-1.1-18.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pytz-2024.1-1.fc40.noarch.rpm
dnf install ./*.rpm

=== acme.sh ===

acme.sh should be run with the called "acme" with a
sudo su - acme

==== Generate a certificate ====

==== Generate a certificate for several sites ====

==== Generate a certificate through DNS ====

For DNS certificate, we need to start the certificate generation, edit the DNS records to add the certificate verification. And launch the end of the certificate generation

acme.sh --issue --dns dns_acmedns -d example.com --server letsencrypt

==== Regenerate certificate how are from certbot ====

==== Renew all certificates ====

For the certificates renewal, anything is needed the cron tab will try to renew them every day "x" day before the expiracy

==== Deploying of the certificates ====

==== on nginx ====

== Nasqueron PKI ==
=== Principles ===
Internal PKI material are stored in Vault, see [[Operations grimoire/Vault]] for information about how to generate, renew, etc.

In the operations repository, the {{Ops file|roles/core/certificates}} unit is responsible to deploy <code>nasqueron-vault-ca.crt</code>, used to authenticate for *.nasqueron.drake or IP services. For exemple, Vault clients require a certificate they can validate with a chain from that intermediate CA. Note as we don't currently use the root CA, a fullchain is probably not needed for those services.

=== Rollover a new certificate ===
If <code>nasqueron-vault-ca.crt</code> is updated or if we want instead to provision the root certificate, the following repository need to be updated and services to be redeployed:

{| class="wikitable sortable"
|+ Where to update certificate?
|-
! Repository !! Path !! Purpose !! Deployment instructions
|-
| operations || {{Ops file|roles/core/certificates/files/}} || Trust internal resources on every server || Salt: <code>salt '*' state.apply roles/core/certificates</code>
|-
| docker-airflow || [https://devcentral.nasqueron.org/source/docker-airflow/browse/main/files files/] || Connect to Vault from Airflow Python code || Docker: build and deploy new image nasqueron/airflow on Dwellers
|}

Deployment instructions are up-to-date when newly added on that table. Afterwards, they should be checked against current procedures.

== Special considerations ==
=== New server ===
Let's encrypt client is available on [[Ysul]] (natively) and [[Dwellers]] (as a wrapper script for a Docker container).

Fill a task in Servers component, subscribe Sandlayth and Dereckson to deploy it on a new server.

A salt state would be nice for such purpose. There is work-in-progress on that matter through {{D|3248}}.

=== Internationalized domain names ===
==== Punycode conversion ====
Both for web server configuration and certificate authority, name must be converted to Punycode ([https://www.ietf.org/rfc/rfc3492.txt RFC 3492]):
https://www.punycoder.com/

==== Let's encrypt support ====

Let's encrypt has supported IDN since 2016<ref>https://letsencrypt.org/2016/10/21/introducing-idn-support.html</ref>. We use it for dægrefn.nasqueron.org certificate.

Previously, they were afraid: attackers could register a domain with a Cyrillic character matching a real domains. As some people consider it's the responsibility of the CA to mitigate such risks, the feature has been several times postponed.

==== StartSSL ====

StartSSL is not in activity anymore. It was used at Nasqueron when Let's Encrypt didn't support IDN.

== Troubleshoot ==
=== Acme DNS ===
==== Can't reach ACME DNS API / HTTP Error 403: Forbidden ====
AcmeDNS API access is restricted to IPs addresses set in {{Ops file|roles/webserver-core/nginx/files/includes/geo_nasqueron}}.

If you got a 403 by running acme.sh (only visible in --debug mode):
* If the IP address is missing from geo_nasqueron, add it
* If the IP address is already there, check on the server if /etc/nginx/includes/geo_nasqueron matches
** Yes -> nginx must be reloaded (nginx -t reload)
** No -> Deploy with <code>salt docker-002 state.sls_id /etc/nginx/includes/geo_nasqueron roles/webserver-core/nginx</code>

Another solution is a specific routing to reach the API:
For Hervil, add a route to use router-001 gateway: route add 51.255.124.8/30 172.27.27.1

==== Troubleshoot records in ACME DNS database ====
To check the acme.nasqueron.org subdomain matching your _acme-challenge subdomain: <code>nslookup -type=TXT _acme-challenge.<subdomain>.nasqueron.org</code>

The DNS server to store acme.nasqueron.org records uses a sqlite3 database. To connect and check your parameters:

$ ssh docker-002
$ sqlite3 /srv/acme/lib/acme-dns.db
SELECT Username, Password FROM records WHERE Subdomain = "<DNS answer>"

==== Credential lost for a ACME DNS subdomain ====

Passwords are [https://github.com/joohoi/acme-dns/blob/9c6ca258e1d57e7441b60db4474a68c36356dae2/validation.go hashed with bcrypt], so it's not possible to recover them.

In that case, there are two options:

* if you prefer edit the DNS record, delete acme.sh information for the domain renewal then issue a new certificate, that will register a new account
* if you prefer edit the ACME DNS database, encrypt a new passw@ord in bcrypt and update it with sqlite

Certificates requests are limited by Let's Encrypt, updating the password avoids to reach that quota.

You can generate a uuid and pass it to bcrypt, or register a new password (and update DNS):

$ ssh docker-002
$ sqlite3 /srv/acme/. lib/acme-dns.db
UPDATE records SET Password = "<new bcrypt hash>" WHERE Subdomain = "<DNS answer>"

To get password hash with bcrypt, you can use psysh on WindRiver:

$ psysh
password_hash("alpha", PASSWORD_BCRYPT)
$2y$10$N17lnt09YFIoVY4025AmCuGrlmA.ZiThm.RDufcakGq9.3IK4xVcW

Replace $2y$ by $2a$ as x and y are unique for PHP, the rest of the world uses $2a$.

== Notes & references ==
<references />

Operations grimoire/TLS certificates

2024-11-05T17:57:04Z

DorianWinty: /* acme.sh */

TLS certificates should be used for every service we provide, so we can encrypt the communications.

== Let's Encrypt ==
=== Certbot commands ===

'''acme-v02 migration.''' If you've a complaint acme-v01.api. isn't available, add <code>--server https://acme-v02.api.letsencrypt.org/directory</code>.

'''T1505 alignment.''' We don't use anymore Certbot as a Docker container on paas-docker role. As such, all paths are now unified. Symlinks have been added to old /srv paths to help transition.

==== Generate a certificate ====
<code>certbot certonly -a webroot --webroot-path=/var/letsencrypt-auto --deploy-hook "service nginx reload" -d foo.nasqueron.org</code>

<code>certbot certonly -a webroot --webroot-path=/var/letsencrypt-auto --deploy-hook "systemctl reload nginx" -d foo.nasqueron.org</code>

==== Generate a certificate for several sites ====
<code>-d foo.nasqueron.org -d bar.nasqueron.org</code>

If a certificate for foo already existed, it will offer to extend it to a new alternative name, which is probably a good idea.

==== Generate a certificate through DNS ====
DNS can be used to generate certificates for domains. For example, the Openfire XMPP certificate is generated like this:

certbot certonly --server https://acme-v02.api.letsencrypt.org/directory --manual --manual-auth-hook /etc/letsencrypt/acme-dns-auth --preferred-challenges dns --debug-challenges -d xmpp.nasqueron.org -d nasqueron.org -d conference.nasqueron.org

From December 2023, you can use this command on FreeBSD servers where {{T|1505}} Let's Encrypt standard installation have been deployed:

certbot certonly --manual --manual-auth-hook /usr/local/etc/letsencrypt/acme-dns-auth --preferred-challenges dns --debug-challenges -d subdomain.nasqueron.org

This uses a specialized DNS server deployed on our Docker PaaS to serve dynamic TLS records under .acme.nasqueron.org, {{Ops file|roles/paas-docker/containers/acme_dns.sls}}.

Support files were previously '''only deployed to paas-docker role''' through {{Ops file|roles/paas-docker/letsencrypt/files}}.

==== Renew all certificates ====
<code>certbot renew</code>

==== Installation on nginx ====
===== Allow Let's encrypt validation =====

include includes/letsencrypt;

This will use {{Ops file|roles/webserver-core/nginx/files/includes/letsencrypt}} nginx configuration.

===== Serve TLS certificate =====

include includes/tls;
ssl_certificate /srv/letsencrypt/etc/live/xmpp.nasqueron.org/fullchain.pem;
ssl_certificate_key /srv/letsencrypt/etc/live/xmpp.nasqueron.org/privkey.pem;

This will configure a compromise between security and compatibility, based on Intermediate Mozilla SSL config<ref>https://ssl-config.mozilla.org/</ref>. The current configuration is served by {{Ops file|roles/webserver-core/nginx/files/includes/tls}}.

If you prefer to restrict the resource for TLS 1.3, and accepts to block legacy clients, you can also use with {{D|3251}}:

include includes/tls-modern-only;
ssl_certificate /srv/letsencrypt/etc/live/xmpp.nasqueron.org/fullchain.pem;
ssl_certificate_key /srv/letsencrypt/etc/live/xmpp.nasqueron.org/privkey.pem;

==== Edit renewal hook ====
In /etc/letsencrypt/renewal or /usr/local/etc/letsencrypt/renewal you can edit this section to customize the command to run after renewal:

[renewalparams]
renew_hook = systemctl reload nginx

That can be applied in batch when nothing is set: <code><nowiki>gsed -i "s/\[\[webroot_map]]/renew_hook = systemctl reload nginx\n\n[[webroot_map]]/g" *.conf</nowiki></code>

=== Install certbot on CentOS 10 stream ===

Packages for Python 3.12 currently used as of 2024-08-04 on CentOS 10 stream can be fetched from Fedora 40 repository.

Installation on Dwellers was done like this:

mkdir /tmp/certbot && cd /tmp/certbot
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/c/certbot-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-acme-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-certbot-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-configargparse-1.7-3.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-josepy-1.13.0-8.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-parsedatetime-2.6-12.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pyOpenSSL-23.2.0-3.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pyrfc3339-1.1-18.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pytz-2024.1-1.fc40.noarch.rpm
dnf install ./*.rpm

=== acme.sh ===

acme.sh should be run with the called "acme" with a
sudo su - acme

==== Generate a certificate ====

==== Generate a certificate for several sites ====

==== Generate a certificate through DNS ====

For DNS certificate, we need to start the certificate generation, edit the DNS records to add the certificate verification. And launch the end of the certificate generation

acme.sh --issue --dns dns_acmedns -d example.com
--server letsencrypt

==== Regenerate certificate how are from certbot ====

==== Renew all certificates ====

For the certificates renewal, anything is needed the cron tab will try to renew them every day "x" day before the expiracy

==== Deploying of the certificates ====

==== on nginx ====

== Nasqueron PKI ==
=== Principles ===
Internal PKI material are stored in Vault, see [[Operations grimoire/Vault]] for information about how to generate, renew, etc.

In the operations repository, the {{Ops file|roles/core/certificates}} unit is responsible to deploy <code>nasqueron-vault-ca.crt</code>, used to authenticate for *.nasqueron.drake or IP services. For exemple, Vault clients require a certificate they can validate with a chain from that intermediate CA. Note as we don't currently use the root CA, a fullchain is probably not needed for those services.

=== Rollover a new certificate ===
If <code>nasqueron-vault-ca.crt</code> is updated or if we want instead to provision the root certificate, the following repository need to be updated and services to be redeployed:

{| class="wikitable sortable"
|+ Where to update certificate?
|-
! Repository !! Path !! Purpose !! Deployment instructions
|-
| operations || {{Ops file|roles/core/certificates/files/}} || Trust internal resources on every server || Salt: <code>salt '*' state.apply roles/core/certificates</code>
|-
| docker-airflow || [https://devcentral.nasqueron.org/source/docker-airflow/browse/main/files files/] || Connect to Vault from Airflow Python code || Docker: build and deploy new image nasqueron/airflow on Dwellers
|}

Deployment instructions are up-to-date when newly added on that table. Afterwards, they should be checked against current procedures.

== Special considerations ==
=== New server ===
Let's encrypt client is available on [[Ysul]] (natively) and [[Dwellers]] (as a wrapper script for a Docker container).

Fill a task in Servers component, subscribe Sandlayth and Dereckson to deploy it on a new server.

A salt state would be nice for such purpose. There is work-in-progress on that matter through {{D|3248}}.

=== Internationalized domain names ===
==== Punycode conversion ====
Both for web server configuration and certificate authority, name must be converted to Punycode ([https://www.ietf.org/rfc/rfc3492.txt RFC 3492]):
https://www.punycoder.com/

==== Let's encrypt support ====

Let's encrypt has supported IDN since 2016<ref>https://letsencrypt.org/2016/10/21/introducing-idn-support.html</ref>. We use it for dægrefn.nasqueron.org certificate.

Previously, they were afraid: attackers could register a domain with a Cyrillic character matching a real domains. As some people consider it's the responsibility of the CA to mitigate such risks, the feature has been several times postponed.

==== StartSSL ====

StartSSL is not in activity anymore. It was used at Nasqueron when Let's Encrypt didn't support IDN.

== Troubleshoot ==
=== Acme DNS ===
==== Can't reach ACME DNS API / HTTP Error 403: Forbidden ====
AcmeDNS API access is restricted to IPs addresses set in {{Ops file|roles/webserver-core/nginx/files/includes/geo_nasqueron}}.

If you got a 403 by running acme.sh (only visible in --debug mode):
* If the IP address is missing from geo_nasqueron, add it
* If the IP address is already there, check on the server if /etc/nginx/includes/geo_nasqueron matches
** Yes -> nginx must be reloaded (nginx -t reload)
** No -> Deploy with <code>salt docker-002 state.sls_id /etc/nginx/includes/geo_nasqueron roles/webserver-core/nginx</code>

Another solution is a specific routing to reach the API:
For Hervil, add a route to use router-001 gateway: route add 51.255.124.8/30 172.27.27.1

==== Troubleshoot records in ACME DNS database ====
To check the acme.nasqueron.org subdomain matching your _acme-challenge subdomain: <code>nslookup -type=TXT _acme-challenge.<subdomain>.nasqueron.org</code>

The DNS server to store acme.nasqueron.org records uses a sqlite3 database. To connect and check your parameters:

$ ssh docker-002
$ sqlite3 /srv/acme/lib/acme-dns.db
SELECT Username, Password FROM records WHERE Subdomain = "<DNS answer>"

==== Credential lost for a ACME DNS subdomain ====

Passwords are [https://github.com/joohoi/acme-dns/blob/9c6ca258e1d57e7441b60db4474a68c36356dae2/validation.go hashed with bcrypt], so it's not possible to recover them.

In that case, there are two options:

* if you prefer edit the DNS record, delete acme.sh information for the domain renewal then issue a new certificate, that will register a new account
* if you prefer edit the ACME DNS database, encrypt a new passw@ord in bcrypt and update it with sqlite

Certificates requests are limited by Let's Encrypt, updating the password avoids to reach that quota.

You can generate a uuid and pass it to bcrypt, or register a new password (and update DNS):

$ ssh docker-002
$ sqlite3 /srv/acme/. lib/acme-dns.db
UPDATE records SET Password = "<new bcrypt hash>" WHERE Subdomain = "<DNS answer>"

To get password hash with bcrypt, you can use psysh on WindRiver:

$ psysh
password_hash("alpha", PASSWORD_BCRYPT)
$2y$10$N17lnt09YFIoVY4025AmCuGrlmA.ZiThm.RDufcakGq9.3IK4xVcW

Replace $2y$ by $2a$ as x and y are unique for PHP, the rest of the world uses $2a$.

== Notes & references ==
<references />

Operations grimoire/TLS certificates

2024-11-05T17:43:29Z

DorianWinty:

TLS certificates should be used for every service we provide, so we can encrypt the communications.

== Let's Encrypt ==
=== Certbot commands ===

'''acme-v02 migration.''' If you've a complaint acme-v01.api. isn't available, add <code>--server https://acme-v02.api.letsencrypt.org/directory</code>.

'''T1505 alignment.''' We don't use anymore Certbot as a Docker container on paas-docker role. As such, all paths are now unified. Symlinks have been added to old /srv paths to help transition.

==== Generate a certificate ====
<code>certbot certonly -a webroot --webroot-path=/var/letsencrypt-auto --deploy-hook "service nginx reload" -d foo.nasqueron.org</code>

<code>certbot certonly -a webroot --webroot-path=/var/letsencrypt-auto --deploy-hook "systemctl reload nginx" -d foo.nasqueron.org</code>

==== Generate a certificate for several sites ====
<code>-d foo.nasqueron.org -d bar.nasqueron.org</code>

If a certificate for foo already existed, it will offer to extend it to a new alternative name, which is probably a good idea.

==== Generate a certificate through DNS ====
DNS can be used to generate certificates for domains. For example, the Openfire XMPP certificate is generated like this:

certbot certonly --server https://acme-v02.api.letsencrypt.org/directory --manual --manual-auth-hook /etc/letsencrypt/acme-dns-auth --preferred-challenges dns --debug-challenges -d xmpp.nasqueron.org -d nasqueron.org -d conference.nasqueron.org

From December 2023, you can use this command on FreeBSD servers where {{T|1505}} Let's Encrypt standard installation have been deployed:

certbot certonly --manual --manual-auth-hook /usr/local/etc/letsencrypt/acme-dns-auth --preferred-challenges dns --debug-challenges -d subdomain.nasqueron.org

This uses a specialized DNS server deployed on our Docker PaaS to serve dynamic TLS records under .acme.nasqueron.org, {{Ops file|roles/paas-docker/containers/acme_dns.sls}}.

Support files were previously '''only deployed to paas-docker role''' through {{Ops file|roles/paas-docker/letsencrypt/files}}.

==== Renew all certificates ====
<code>certbot renew</code>

==== Installation on nginx ====
===== Allow Let's encrypt validation =====

include includes/letsencrypt;

This will use {{Ops file|roles/webserver-core/nginx/files/includes/letsencrypt}} nginx configuration.

===== Serve TLS certificate =====

include includes/tls;
ssl_certificate /srv/letsencrypt/etc/live/xmpp.nasqueron.org/fullchain.pem;
ssl_certificate_key /srv/letsencrypt/etc/live/xmpp.nasqueron.org/privkey.pem;

This will configure a compromise between security and compatibility, based on Intermediate Mozilla SSL config<ref>https://ssl-config.mozilla.org/</ref>. The current configuration is served by {{Ops file|roles/webserver-core/nginx/files/includes/tls}}.

If you prefer to restrict the resource for TLS 1.3, and accepts to block legacy clients, you can also use with {{D|3251}}:

include includes/tls-modern-only;
ssl_certificate /srv/letsencrypt/etc/live/xmpp.nasqueron.org/fullchain.pem;
ssl_certificate_key /srv/letsencrypt/etc/live/xmpp.nasqueron.org/privkey.pem;

==== Edit renewal hook ====
In /etc/letsencrypt/renewal or /usr/local/etc/letsencrypt/renewal you can edit this section to customize the command to run after renewal:

[renewalparams]
renew_hook = systemctl reload nginx

That can be applied in batch when nothing is set: <code><nowiki>gsed -i "s/\[\[webroot_map]]/renew_hook = systemctl reload nginx\n\n[[webroot_map]]/g" *.conf</nowiki></code>

=== Install certbot on CentOS 10 stream ===

Packages for Python 3.12 currently used as of 2024-08-04 on CentOS 10 stream can be fetched from Fedora 40 repository.

Installation on Dwellers was done like this:

mkdir /tmp/certbot && cd /tmp/certbot
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/c/certbot-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-acme-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-certbot-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-configargparse-1.7-3.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-josepy-1.13.0-8.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-parsedatetime-2.6-12.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pyOpenSSL-23.2.0-3.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pyrfc3339-1.1-18.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pytz-2024.1-1.fc40.noarch.rpm
dnf install ./*.rpm

=== acme.sh ===

== Nasqueron PKI ==
=== Principles ===
Internal PKI material are stored in Vault, see [[Operations grimoire/Vault]] for information about how to generate, renew, etc.

In the operations repository, the {{Ops file|roles/core/certificates}} unit is responsible to deploy <code>nasqueron-vault-ca.crt</code>, used to authenticate for *.nasqueron.drake or IP services. For exemple, Vault clients require a certificate they can validate with a chain from that intermediate CA. Note as we don't currently use the root CA, a fullchain is probably not needed for those services.

=== Rollover a new certificate ===
If <code>nasqueron-vault-ca.crt</code> is updated or if we want instead to provision the root certificate, the following repository need to be updated and services to be redeployed:

{| class="wikitable sortable"
|+ Where to update certificate?
|-
! Repository !! Path !! Purpose !! Deployment instructions
|-
| operations || {{Ops file|roles/core/certificates/files/}} || Trust internal resources on every server || Salt: <code>salt '*' state.apply roles/core/certificates</code>
|-
| docker-airflow || [https://devcentral.nasqueron.org/source/docker-airflow/browse/main/files files/] || Connect to Vault from Airflow Python code || Docker: build and deploy new image nasqueron/airflow on Dwellers
|}

Deployment instructions are up-to-date when newly added on that table. Afterwards, they should be checked against current procedures.

== Special considerations ==
=== New server ===
Let's encrypt client is available on [[Ysul]] (natively) and [[Dwellers]] (as a wrapper script for a Docker container).

Fill a task in Servers component, subscribe Sandlayth and Dereckson to deploy it on a new server.

A salt state would be nice for such purpose. There is work-in-progress on that matter through {{D|3248}}.

=== Internationalized domain names ===
==== Punycode conversion ====
Both for web server configuration and certificate authority, name must be converted to Punycode ([https://www.ietf.org/rfc/rfc3492.txt RFC 3492]):
https://www.punycoder.com/

==== Let's encrypt support ====

Let's encrypt has supported IDN since 2016<ref>https://letsencrypt.org/2016/10/21/introducing-idn-support.html</ref>. We use it for dægrefn.nasqueron.org certificate.

Previously, they were afraid: attackers could register a domain with a Cyrillic character matching a real domains. As some people consider it's the responsibility of the CA to mitigate such risks, the feature has been several times postponed.

==== StartSSL ====

StartSSL is not in activity anymore. It was used at Nasqueron when Let's Encrypt didn't support IDN.

== Troubleshoot ==
=== Acme DNS ===
==== Can't reach ACME DNS API / HTTP Error 403: Forbidden ====
AcmeDNS API access is restricted to IPs addresses set in {{Ops file|roles/webserver-core/nginx/files/includes/geo_nasqueron}}.

If you got a 403 by running acme.sh (only visible in --debug mode):
* If the IP address is missing from geo_nasqueron, add it
* If the IP address is already there, check on the server if /etc/nginx/includes/geo_nasqueron matches
** Yes -> nginx must be reloaded (nginx -t reload)
** No -> Deploy with <code>salt docker-002 state.sls_id /etc/nginx/includes/geo_nasqueron roles/webserver-core/nginx</code>

Another solution is a specific routing to reach the API:
For Hervil, add a route to use router-001 gateway: route add 51.255.124.8/30 172.27.27.1

==== Troubleshoot records in ACME DNS database ====
To check the acme.nasqueron.org subdomain matching your _acme-challenge subdomain: <code>nslookup -type=TXT _acme-challenge.<subdomain>.nasqueron.org</code>

The DNS server to store acme.nasqueron.org records uses a sqlite3 database. To connect and check your parameters:

$ ssh docker-002
$ sqlite3 /srv/acme/lib/acme-dns.db
SELECT Username, Password FROM records WHERE Subdomain = "<DNS answer>"

==== Credential lost for a ACME DNS subdomain ====

Passwords are [https://github.com/joohoi/acme-dns/blob/9c6ca258e1d57e7441b60db4474a68c36356dae2/validation.go hashed with bcrypt], so it's not possible to recover them.

In that case, there are two options:

* if you prefer edit the DNS record, delete acme.sh information for the domain renewal then issue a new certificate, that will register a new account
* if you prefer edit the ACME DNS database, encrypt a new passw@ord in bcrypt and update it with sqlite

Certificates requests are limited by Let's Encrypt, updating the password avoids to reach that quota.

You can generate a uuid and pass it to bcrypt, or register a new password (and update DNS):

$ ssh docker-002
$ sqlite3 /srv/acme/. lib/acme-dns.db
UPDATE records SET Password = "<new bcrypt hash>" WHERE Subdomain = "<DNS answer>"

To get password hash with bcrypt, you can use psysh on WindRiver:

$ psysh
password_hash("alpha", PASSWORD_BCRYPT)
$2y$10$N17lnt09YFIoVY4025AmCuGrlmA.ZiThm.RDufcakGq9.3IK4xVcW

Replace $2y$ by $2a$ as x and y are unique for PHP, the rest of the world uses $2a$.

== Notes & references ==
<references />

Operations grimoire/TLS certificates

2024-11-05T17:42:57Z

DorianWinty:

TLS certificates should be used for every service we provide, so we can encrypt the communications.

== Let's Encrypt ==
=== Certbot commands ===

'''acme-v02 migration.''' If you've a complaint acme-v01.api. isn't available, add <code>--server https://acme-v02.api.letsencrypt.org/directory</code>.

'''T1505 alignment.''' We don't use anymore Certbot as a Docker container on paas-docker role. As such, all paths are now unified. Symlinks have been added to old /srv paths to help transition.

==== Generate a certificate ====
<code>certbot certonly -a webroot --webroot-path=/var/letsencrypt-auto --deploy-hook "service nginx reload" -d foo.nasqueron.org</code>

<code>certbot certonly -a webroot --webroot-path=/var/letsencrypt-auto --deploy-hook "systemctl reload nginx" -d foo.nasqueron.org</code>

==== Generate a certificate for several sites ====
<code>-d foo.nasqueron.org -d bar.nasqueron.org</code>

If a certificate for foo already existed, it will offer to extend it to a new alternative name, which is probably a good idea.

==== Generate a certificate through DNS ====
DNS can be used to generate certificates for domains. For example, the Openfire XMPP certificate is generated like this:

certbot certonly --server https://acme-v02.api.letsencrypt.org/directory --manual --manual-auth-hook /etc/letsencrypt/acme-dns-auth --preferred-challenges dns --debug-challenges -d xmpp.nasqueron.org -d nasqueron.org -d conference.nasqueron.org

From December 2023, you can use this command on FreeBSD servers where {{T|1505}} Let's Encrypt standard installation have been deployed:

certbot certonly --manual --manual-auth-hook /usr/local/etc/letsencrypt/acme-dns-auth --preferred-challenges dns --debug-challenges -d subdomain.nasqueron.org

This uses a specialized DNS server deployed on our Docker PaaS to serve dynamic TLS records under .acme.nasqueron.org, {{Ops file|roles/paas-docker/containers/acme_dns.sls}}.

Support files were previously '''only deployed to paas-docker role''' through {{Ops file|roles/paas-docker/letsencrypt/files}}.

==== Renew all certificates ====
<code>certbot renew</code>

==== Installation on nginx ====
===== Allow Let's encrypt validation =====

include includes/letsencrypt;

This will use {{Ops file|roles/webserver-core/nginx/files/includes/letsencrypt}} nginx configuration.

===== Serve TLS certificate =====

include includes/tls;
ssl_certificate /srv/letsencrypt/etc/live/xmpp.nasqueron.org/fullchain.pem;
ssl_certificate_key /srv/letsencrypt/etc/live/xmpp.nasqueron.org/privkey.pem;

This will configure a compromise between security and compatibility, based on Intermediate Mozilla SSL config<ref>https://ssl-config.mozilla.org/</ref>. The current configuration is served by {{Ops file|roles/webserver-core/nginx/files/includes/tls}}.

If you prefer to restrict the resource for TLS 1.3, and accepts to block legacy clients, you can also use with {{D|3251}}:

include includes/tls-modern-only;
ssl_certificate /srv/letsencrypt/etc/live/xmpp.nasqueron.org/fullchain.pem;
ssl_certificate_key /srv/letsencrypt/etc/live/xmpp.nasqueron.org/privkey.pem;

==== Edit renewal hook ====
In /etc/letsencrypt/renewal or /usr/local/etc/letsencrypt/renewal you can edit this section to customize the command to run after renewal:

[renewalparams]
renew_hook = systemctl reload nginx

That can be applied in batch when nothing is set: <code><nowiki>gsed -i "s/\[\[webroot_map]]/renew_hook = systemctl reload nginx\n\n[[webroot_map]]/g" *.conf</nowiki></code>

=== Install certbot on CentOS 10 stream ===

Packages for Python 3.12 currently used as of 2024-08-04 on CentOS 10 stream can be fetched from Fedora 40 repository.

Installation on Dwellers was done like this:

mkdir /tmp/certbot && cd /tmp/certbot
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/c/certbot-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-acme-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-certbot-2.9.0-1.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-configargparse-1.7-3.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-josepy-1.13.0-8.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-parsedatetime-2.6-12.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pyOpenSSL-23.2.0-3.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pyrfc3339-1.1-18.fc40.noarch.rpm
wget https://dl.fedoraproject.org/pub/fedora/linux/releases/40/Everything/x86_64/os/Packages/p/python3-pytz-2024.1-1.fc40.noarch.rpm
dnf install ./*.rpm

== acme.sh ==

== Nasqueron PKI ==
=== Principles ===
Internal PKI material are stored in Vault, see [[Operations grimoire/Vault]] for information about how to generate, renew, etc.

In the operations repository, the {{Ops file|roles/core/certificates}} unit is responsible to deploy <code>nasqueron-vault-ca.crt</code>, used to authenticate for *.nasqueron.drake or IP services. For exemple, Vault clients require a certificate they can validate with a chain from that intermediate CA. Note as we don't currently use the root CA, a fullchain is probably not needed for those services.

=== Rollover a new certificate ===
If <code>nasqueron-vault-ca.crt</code> is updated or if we want instead to provision the root certificate, the following repository need to be updated and services to be redeployed:

{| class="wikitable sortable"
|+ Where to update certificate?
|-
! Repository !! Path !! Purpose !! Deployment instructions
|-
| operations || {{Ops file|roles/core/certificates/files/}} || Trust internal resources on every server || Salt: <code>salt '*' state.apply roles/core/certificates</code>
|-
| docker-airflow || [https://devcentral.nasqueron.org/source/docker-airflow/browse/main/files files/] || Connect to Vault from Airflow Python code || Docker: build and deploy new image nasqueron/airflow on Dwellers
|}

Deployment instructions are up-to-date when newly added on that table. Afterwards, they should be checked against current procedures.

== Special considerations ==
=== New server ===
Let's encrypt client is available on [[Ysul]] (natively) and [[Dwellers]] (as a wrapper script for a Docker container).

Fill a task in Servers component, subscribe Sandlayth and Dereckson to deploy it on a new server.

A salt state would be nice for such purpose. There is work-in-progress on that matter through {{D|3248}}.

=== Internationalized domain names ===
==== Punycode conversion ====
Both for web server configuration and certificate authority, name must be converted to Punycode ([https://www.ietf.org/rfc/rfc3492.txt RFC 3492]):
https://www.punycoder.com/

==== Let's encrypt support ====

Let's encrypt has supported IDN since 2016<ref>https://letsencrypt.org/2016/10/21/introducing-idn-support.html</ref>. We use it for dægrefn.nasqueron.org certificate.

Previously, they were afraid: attackers could register a domain with a Cyrillic character matching a real domains. As some people consider it's the responsibility of the CA to mitigate such risks, the feature has been several times postponed.

==== StartSSL ====

StartSSL is not in activity anymore. It was used at Nasqueron when Let's Encrypt didn't support IDN.

== Troubleshoot ==
=== Acme DNS ===
==== Can't reach ACME DNS API / HTTP Error 403: Forbidden ====
AcmeDNS API access is restricted to IPs addresses set in {{Ops file|roles/webserver-core/nginx/files/includes/geo_nasqueron}}.

If you got a 403 by running acme.sh (only visible in --debug mode):
* If the IP address is missing from geo_nasqueron, add it
* If the IP address is already there, check on the server if /etc/nginx/includes/geo_nasqueron matches
** Yes -> nginx must be reloaded (nginx -t reload)
** No -> Deploy with <code>salt docker-002 state.sls_id /etc/nginx/includes/geo_nasqueron roles/webserver-core/nginx</code>

Another solution is a specific routing to reach the API:
For Hervil, add a route to use router-001 gateway: route add 51.255.124.8/30 172.27.27.1

==== Troubleshoot records in ACME DNS database ====
To check the acme.nasqueron.org subdomain matching your _acme-challenge subdomain: <code>nslookup -type=TXT _acme-challenge.<subdomain>.nasqueron.org</code>

The DNS server to store acme.nasqueron.org records uses a sqlite3 database. To connect and check your parameters:

$ ssh docker-002
$ sqlite3 /srv/acme/lib/acme-dns.db
SELECT Username, Password FROM records WHERE Subdomain = "<DNS answer>"

==== Credential lost for a ACME DNS subdomain ====

Passwords are [https://github.com/joohoi/acme-dns/blob/9c6ca258e1d57e7441b60db4474a68c36356dae2/validation.go hashed with bcrypt], so it's not possible to recover them.

In that case, there are two options:

* if you prefer edit the DNS record, delete acme.sh information for the domain renewal then issue a new certificate, that will register a new account
* if you prefer edit the ACME DNS database, encrypt a new passw@ord in bcrypt and update it with sqlite

Certificates requests are limited by Let's Encrypt, updating the password avoids to reach that quota.

You can generate a uuid and pass it to bcrypt, or register a new password (and update DNS):

$ ssh docker-002
$ sqlite3 /srv/acme/. lib/acme-dns.db
UPDATE records SET Password = "<new bcrypt hash>" WHERE Subdomain = "<DNS answer>"

To get password hash with bcrypt, you can use psysh on WindRiver:

$ psysh
password_hash("alpha", PASSWORD_BCRYPT)
$2y$10$N17lnt09YFIoVY4025AmCuGrlmA.ZiThm.RDufcakGq9.3IK4xVcW

Replace $2y$ by $2a$ as x and y are unique for PHP, the rest of the world uses $2a$.

== Notes & references ==
<references />

Operations grimoire/Mail

2024-10-03T20:06:25Z

DorianWinty: /* Network */

The mail infrastructure is shared between third party services (Mailgun, Sendgrid) for web applications willing to use API and our own mail server for regular mailboxes.

<div style="float: right; width: 22em; border: solid 2px black; padding: 1em;>
'''{{FULLPAGENAME}}'''
{{Special:Prefixindex/{{FULLPAGENAME}}/}}
</div>

== Third party services ==
Mainly, they provide configuration wizards, logs, and API keys on a web interface.

[[Operations grimoire/External services]] says who to contact to debug any issue, configure them, etc.

== Nasqueron mail services ==
=== Architecture ===
We use the following servers:
* Postfix
** 25 is for mail servers
** 587 is for STARTTLS + user auth
* dovecot for IMAP / POP
* SpamAssassin, OpenDKIM (see [[/DKIM]])
* Sympa for the mailing lists
* MySQL to store user accounts mailboxes and sympa data
* nginx to serve web applications

/etc/postfix and /etc/dovecot are Git repositories, so commit your configuration changes.

User accounts are stored in a MySQL database. They are managed by ViMbAdmin (on https://vma.nasqueron.org).

Sympa manages the mailing lists.

A nginx server serves vma as vma.nasqinternal, Roundcube as mail.nasqinternal and Sympa.
On Dwellers, nginx assumes SSL termination and the relevant vhosts like mail.nasqueron.org, mail.wolfplex.be, etc.

All that should be migrated to configuration as code to be managed through Salt.

A lxc container has been chosen for more stability: Docker assumes we can respin containers, host OS can change. The lxc container is isolated, stable and lxc doesn't ask restarts.

=== Log in to the server ===
Mail server lives on the lxc container <code>mailserver</code> on [[Dwellers]].

To access it, you must so:
* ssh dwellers
* attach to the container (<code>lxc-attach -n mailserver [tcsh]</code>)

If you need to access a lxc container, you can script something do to:
<code>$SSH $LXC_SERVER $LXC_EXEC $CONTAINER_NAME $LXC_COMMAND</code>

Here, it would be <code>ssh -t dwellers.nasqueron.org sudo lxc-attach -n mailserver tcsh</code>.

To be able to use sudo for lxc-attach, you must belong to the `ops` group.

=== Add a domain ===

# Add it to https://vma.nasqueron.org
# Follow [[/DKIM]] procedure

It's ready.

You can also be willing to declare the domain to autoconfig/autodiscover, but that's blocked by https://devcentral.nasqueron.org/T1116.

== New deploy thinking ==

{| class="wikitable"
|+ pour ou contre
|-
! ViMbAdmin aventage
! ViMbAdmin inconvenient
! Salt pillar aventage
! Salt pillar inconvenient
|-
| Ediatable interface
| Configuration splited
| Unified configuration
| Private salt repository needed for privacy
|-
| Easy to use interface
| Should be installed on the same server as the mailserver
|
|
|-
| Easy solution for users to changes their passwords
|
|
|
|-
| An administrator of an external domain for hosting of other open source project
|
|
|
|-
|
|
|
|
|}

== Hervil ==

=== Network ===
Routing tables

Internet:
Destination Gateway Flags Netif Expire
default 51.210.99.254 UGS vmx1
51.210.99.254 link#2 UHS vmx1
51.255.124.8/30 172.27.27.1 UGS vmx0

For SPF record we need to make the mail go out from the ip : 51.210.99.254
To permit acme.sh we make a route thrugh router.OO1

== Troubleshoot ==
=== Dashboards ===
On Grafana, the following dashboards are available:
* [https://grafana.nasqueron.org/d/h36Havfik/postfix-legacy-dashboard?orgId=1 Postfix]

The following dashboards are missing, feel free to create one:
* Dovecot

=== Logs ===
On FreeBSD servers, mail logs are consolidated into <code>/var/log/maillog</code> by syslog daemon.

=== Test to send e-mail with telnet or openssl clients ===

You can test STARTTLS with openssl s_client:

openssl s_client -connect mail.nasqueron.org:587 -starttls smtp -ign_eof -crlf

Flags:
* The -starttls smtp option is the one to send the STARTTLS command
* The -ign_eof flag disables interactive commands, to avoid to renegotiate the TLS session when you press R (like in RCPT TO).
* The -crlf flag doesn't seem needed on FreeBSD, but seem needed on Fedora. It allows to always use \CR\LF (\r\n) as EOL.

=== @nasqueron.org must be sent through mail.nasqueron.org ===
A SPF record ''v=spf1 a:mail.nasqueron.org -all'' should exist.

If set, any application app.nasqueron.org should:
* send mails using @app.nasqueron.org domain, not the generic @nasqueron.org one;
* define SPF and if possible DKIM records for this @app.nasqueron.org mail domain.

Operations grimoire/Mail

2024-10-03T20:05:07Z

DorianWinty:

Operations grimoire/Changelog

2024-07-04T17:12:11Z

DorianWinty:

== 2024 ==
=== July 2024 ===
* [2024-07-07 20:30] [WindRiver] <Dereckson> Upgrade done.
* [2024-07-07 19:50] [WindRiver] <Dereckson> Upgrade to FreeBSD 14.1. Rebooting.
* [2024-07-04 19:10] [Complector] <DorianWinty> Reinstall Git

Operations grimoire/PostgreSQL

2024-06-20T19:50:22Z

DorianWinty:

PostgreSQL is available as a standalone role and can also be enabled on devserver.

Applications on the Docker PaaS can use our PostgreSQL infrastructure (Airflow, datasources) or have their own container (Sentry, with custom wal2json extension). This resource documents our own db- servers.

== Howto ==

=== Open a console ===
Use peer authentication from the postgres user: <code>sudo -u postgres psql</code>

=== Run queries against database ===
The important things are:

# prepare the operation with a .sql file
# put that SQL file to a task
# avoid data destruction and allow rollback
## if a table needs to be dropped, we can rename it to _unused
## if an application doesn't use a column anymore, we can rename it from foo to foo_unused so we can rollback by renaming back
## actual deletion can be done some weeks later

Recommend procedure:

# Create a task on DevCentral or reuse an existing deployment task
# Upload the .sql file on DevCentral or to a repository:
## One-shot maintenance can go to https://devcentral.nasqueron.org/paste/ or sent with <code>cat foo.sql | arc paste --title foo.sql --</code>
## Regular queries can go to the [https://devcentral.nasqueron.org/source/reports/browse/main/sql/ reports repository]
# Link it in the task
# Run it

To run foo.sql against acme database:

# upload the foo.sql file to the relevant server (for example db-A-001 for db-A cluster)
# <code>sudo -u postgres psql acme < foo.sql</code>

=== Create a new database or user ===

It's a simple two steps process:

# Edit the relevant pillar file, for example pillar/dbserver/cluster-A.sls in rOPS
# Deploy the change: <code>salt db-A-001 state.apply roles/dbserver-pgsql</code>

You generally need:

* an user block in dbserver_postgresql.users
** block title is the username
** a password needs to be set in Flow, key is under ops/secrets/
* a database block
* a connection triplet db, user, ips (use /32 for an unique IP)

By default, external connections are NOT enabled for any user and database, an entry for pg_hba.conf MUST be added in connections part.

Servers don't have a public ICANN IP, so you can only connect from other Nasqueron servers.

=== Grant privileges to an user ===

Some applications don't set privileges, even if they own the database. That's for example the case of Orbeon Forms DDL.

To determine if acme has already the rights on the database:

SELECT grantor, grantee, table_schema, table_name, privilege_type
FROM information_schema.table_privileges
WHERE grantee = 'acme';

If not:

GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO acme;

This set of privileges can be handled directly by Salt in cluster privileges definition:

privileges:
- database: <database name>
scope: table
schema: public
tables:
- ALL
privileges:
- ALL

See {{D|3252}} or {{D|3253}} for an example.

As a known issue, this query will ALWAYS be executed by Salt.

; Sequences privileges

Sequences aren't included in that GRANT ALL PRIVILEGES. We can generate GRANT queries for them:

SELECT 'GRANT USAGE ON SEQUENCE ' || sequence_schema || '.' || sequence_name || ' TO "acme";'
FROM information_schema.sequences
WHERE sequence_schema NOT LIKE 'pg_%'
AND sequence_schema != 'information_schema';

=== What to do after an update of PostgreSQL servers? ===

The following services need their connection to explicitly be restarted:

{| class="wikitable"
|+ Services to tweak
|-
! Cluster !! Service !! Procedure
|-
| A || Airflow > airflow_triggerer || On Dwellers, docker restart airflow_triggerer
|-
| A || Airflow > airflow_scheduler || On Dwellers, docker restart airflow_scheduler
|}

== Clusters ==

Letters can be discontinuous: for example, B will be a MySQL cluster.

=== Cluster A ===
A is the general cluster, for Nasqueron services. It currently has one server, db-A-001.

{| class="wikitable"
|+ Databases
|-
! Database !! Managed by !! Description
|-
| airflow || Nasqueron Ops || Workflows runner, used by datasources
|-
| fantoir || Datasources || PostgreSQL version of FANTOIR for geocoding
|}

== Troubleshoot ==
=== An application stops to work correctly with PostgreSQL ===
Issues with queries are logged to /var/log/messages:

$ tail -n100 -f /var/log/messages | grep postgres

For example, {{T|1943}} issue was found with this line:

Jan 14 23:13:58 db-A-001 postgres[35351]: [8-1] 2024-01-14 23:13:58.081 UTC [35351] ERROR: unsupported XML feature

=== Unsupported XML feature ===
For Orbeon, XML support is required. That's not by default the case on FreeBSD, the port needs to be built manually. That's the case when provisioning a new server through {{Ops file|roles/dbserver-pgsql/server/build.sls}} but upgrade can break this port.

If so, rebuild the package. For that, you can follow the instructions of [[Operations grimoire/FreeBSD]], section PostgreSQL.

=== Some postgres_privileges.present are always changed by Salt ===

Some ON ALL TABLE privilege queries are always run by the Salt PostgreSQL execution module. For example:

ID: dbserver_pgsql_user_orbeon_privilege_2_ALL
Function: postgres_privileges.present
Name: orbeon
Result: True
Comment: The privilege(s): ALL have been granted to orbeon
Started: 21:54:53.656120
Duration: 284.394 ms
Changes:
----------
orbeon:
Present

Those statements aren't idempotent. This is a known issue, and doesn't affect deployment.

Operations grimoire/Mail

2024-06-18T20:46:27Z

DorianWinty:

Operations grimoire/Create and revoke user accounts on Salt servers

2023-04-02T18:26:28Z

DorianWinty:

== Create an user ==
# Add entry to shellusers in the <code>pillar/core/users.sls</code>
## uid: you can run <code>utils/next-uid.py</code> to generate one (if not, take the greatest 2xxx and do +1)
## shell: default shell is bash, other availables are fish / nologin / tcsh / zsh
## fullname: this is a public information, so publish only if the user is comfortable with that (ie if they publish the full name elsewhere like DevCentral, GitHub, etc.)

== Revoke an user ==
# Keep the entry in shellusers at pillar/core/users.sls
# Append the element to the 'revokedusers' list
# Remove it from relevant groups in pillar/core/groups.sls

== Assign an user to a group ==
If you only put an user in shellusers, that's a no op operation.

Each server take users through the shellgroups dictionary in pillar/core/groups.sls.

{| class="wikitable"
|-
! Group !! Description !! Example of servers
|-
| shell || Shell access on Eglide (e.g. for IRC purpose) || eglide.org
|-
| ops || Nasqueron Operations || Access everywhere
|-
| nasquenautes || Shell access on Nasqueron dev servers || ysul, WindRiver
|}

Generally, the target groups are `shell` or `nasquenautes`.

Some specialized groups exist for a particular piece of software or service.
It's generally ignored when adding a new user.

== Run the Salt ==
If you've access to the Salt primary server in production:

<code>salt eglide state.apply roles/core/users</code>

Or locally while Eglide doesn't have have a direct access to the Salt primary server:

<code>sudo salt-call --local state.apply roles/shellserver/users</code>

Take care of the SSH key changes during the output. Please notify concerned users about any SSH keys change when running this.

See [[Operations grimoire/Deploy with Salt]].

Operations grimoire/Create and revoke user accounts on Salt servers

2023-04-02T18:26:07Z

DorianWinty:

== Create an user ==
# Add entry to shellusers in the <code>pillar/core/users.sls</code>
## uid: you can run <code>utils/next-uid.py</code> to generate one (if not, take the greatest 2xxx and do +1)
## shell: default shell is bash, other availables are fish / nologin / tcsh / zsh
## fullname: this is a public information, so publish only if the user is comfortable with that (ie if they publish the full name elsewhere like DevCentral, GitHub, etc.)

== Revoke an user ==
# Keep the entry in shellusers at pillar/core/users.sls
# Append the element to the 'revokedusers' list
# Remove it from relevant groups in pillar/core/groups.sls

== Assign an user to a group ==
If you only put an user in shellusers, that's a no op operation.

Each server take users through the shellgroups dictionary in pillar/core/groups.sls.

{| class="wikitable"
|-
! Group !! Description !! Example of servers
|-
| shell || Shell access on Eglide (e.g. for IRC purpose) || eglide.org
|-
| ops || Nasqueron Operations || Access everywhere
|-
| nasquenautes || Shell access on Nasqueron dev servers || ysul, WindRiver
|}

Generally, the target groups are `shell` or `nasquenautes`.

Some specialized groups exist for a particular piece of software or service.
It's generally ignored when adding a new user.

== Run the Salt ==
If you've access to the Salt primary server in production:

<code>salt eglide state.apply oles/core/users</code>

Or locally while Eglide doesn't have have a direct access to the Salt primary server:

<code>sudo salt-call --local state.apply roles/shellserver/users</code>

Take care of the SSH key changes during the output. Please notify concerned users about any SSH keys change when running this.

See [[Operations grimoire/Deploy with Salt]].