Operations grimoire/Notifications center: Difference between revisions

From Nasqueron Agora
(+links)
m (syntaxhighlight instead of source tag)
 
(23 intermediate revisions by 2 users not shown)
Line 1: Line 1:
The notifications centers is an HTTP to HTTP and HTTP to AQMP gateway for our CI infrastructure.
The notifications centers is an HTTP to HTTP and HTTP to AMQP gateway for our CI infrastructure.


It allows to receive events from GitHub, Docker Hub and Phabricator, and send them to a RabbitMQ broker (white-rabbit) or Phabricator.
It allows to receive events from GitHub, Docker Hub and Phabricator, and send them to a RabbitMQ broker (white-rabbit) or Phabricator.
Line 7: Line 7:
* [[Operations grimoire/RabbitMQ|RabbitMQ broker]]
* [[Operations grimoire/RabbitMQ|RabbitMQ broker]]


== Start a new container ==
== Run it ==
=== Start a new container ===
<code>run-notifications</code>
<code>run-notifications</code>


== Upgrade a live container ==
=== Upgrade a live container ===
# Enter notifications container as app user
# Enter notifications container as app user : <code>docker exec -it --user=app notifications bash</code>
# Check we're on the master branch and the history is clean
# Check we're on the master branch and the history is clean
# Update code with <code>git pull</code>
# Update code rebasing the production branch against origin/master
# If composer has been touched, <code>composer update</code> (could be faster to docker pull, stop this container and start a new container)
# If composer has been touched, <code>composer update --ignore-platform-reqs --no-dev</code> (be careful if so, it could be faster to instead do a docker pull, stop this container and start a new container)
# If entered as root, fix ownership with <code>chown -R app:app /var/wwwroot/default</code>
# If entered as root, fix ownership with <code>chown -R app:app /var/wwwroot/default</code>
# Restart PHP FPM with <code>sv restart php-fpm</code>
# Restart PHP FPM with <code>sv restart php-fpm</code>
# Run <code>php artisan optimize</code> to refresh compiled class
# Run <code>php artisan config:cache</code> to cache updated config, to use if preferences are updated in the code or by us
# Restart again PHP FPM with <code>sv restart php-fpm</code>
 
For longer upgrade, you can put the application in maintenance mode with <code>php artisan down</code>.
 
=== How to rebase against master? ===
 
# update master to match origin/master
# rebase production against master
 
<syntaxhighlight lang="console">
$ git fetch
$ git checkout master
$ git rebase origin/master
$ git checkout production
$ git rebase master
</syntaxhighlight>
 
If there is a merge conflict:
 
# Edit the conflict files
# <code>git add <edited files></code>
# <code>git rebase --continue</code>
 
== Configure it ==
=== Edit configuration ===
Configuration is stored in {{Ops file|pillar/notifications/config.sls}}.
 
It can be applied through Salt:
 
<code>salt-call --local state.apply roles/paas-docker/containers/notifications</code>
 
That will spin a new container too if the image or the rOPS config has been updated.
 
=== Add a new GitHub organization ===
# Generate a random string to be used as a secret token shared between GitHub and the notifications center
# On Dwellers, add credentials to /data/notifications/storage/app/credentials.json
# On GitHub, go to organization settings, then Webhooks, e.g. https://github.com/organizations/acme/settings/hooks
# Add webhook:
## URL should be https://notifications.nasqueron.org/gate/GitHub/Acme where Acme is the ucfirst name of the organization account
## Your secret token go to secret field
## Switch from push events to "Send me everything".
## Let default settings: active, SSL verification, application/json
# Run the CLI client to receive notifications (`notifications` on Ysul, require to be in the `notifications` group, editable through /etc/group)
# GitHub will then fire immediately a ping, check in recent deliveries it's 200
# Fix and redeliver it if not
# Check in the CLI you've got a correct reply, e.g. [16:36:13] <Eglide/orgz> « Anything added dilutes everything else. » — GitHub Webhooks ping zen aphorism.
 
For payload delivery code returned by notifications.nasqueron.org, a 500 means probably a JSON syntax error in credentials.json, a 403 tokens don't match, note the center doesn't verify token if an empty string is left in the config (or you made a typo to the property).
 
=== Regenerate Docker Hub configuration ===
Docker Hub doesn't seem to currently offer a comprehensive machine-readable API, but the website uses JSON documents to get data, so we can login as user and use browser console to fetch and consolidate data we need.
 
Procedure: https://gist.github.com/dereckson/23181a751e014796c52a4011e2a7de22
 
Target file: /srv/notifications/storage/app/DockerHubBuildTriggers.js @ Docker server hosting the notifications center container (currently docker-001)
 
''The notifications center won't currently parse this file, it's indented for a future improvement, as Docker Hub broke old token system.''
 
=== Commit configuration changes to rTESTSPRODENV ===
The test repository [https://devcentral.nasqueron.org/source/tests-prod-environment-behaves-correctly/ rTESTSPRODENV] contains a reference JSON document.
 
This document will be compared with https://notifications.nasqueron.org/config output and must be equals.
 
You can also regenerate it from current configuration, but please double check it against current config through <code>git diff</code> use.
 
<syntaxhighlight lang="console">
$ curl https://notifications.nasqueron.org/config > data/notifications.config.json
  % Total    % Received % Xferd  Average Speed  Time    Time    Time  Current
                                Dload  Upload  Total  Spent    Left  Speed
100  354    0  354    0    0    263      0 --:--:--  0:00:01 --:--:--  263
$ git diff
</syntaxhighlight>
 
'''Don't use jsonlint to pretty print the file.'''
 
It's probably a good idea to format to pretty print the JSON, but `jsonlint` badly encode URL:
 
<code>curl https://notifications.nasqueron.org/config | jsonlint</code> will be
https:\\/\\/devcentral.nasqueron.org"
 
This double encoding is write, as PHP will decode the string this string as this:
 
<syntaxhighlight lang="console">
$ psysh
Psy Shell v0.6.1 (PHP 5.6.29 — cli) by Justin Hileman
>>> $url = '"https:\\/\\/devcentral.nasqueron.org"'
=> ""https:\/\/devcentral.nasqueron.org""
>>> $url = '"https:\\\\/\\\\/devcentral.nasqueron.org"'
=> ""https:\\/\\/devcentral.nasqueron.org""
>>> json_decode($url)
=> "https:\/\/devcentral.nasqueron.org"
</syntaxhighlight>


== Links ==
== Links ==
Line 22: Line 117:
# [https://devcentral.nasqueron.org/tag/notifications_center/ DevCentral board]
# [https://devcentral.nasqueron.org/tag/notifications_center/ DevCentral board]


{{Configuration as Code}}
{{Vault secrets migration}}

Latest revision as of 12:18, 20 February 2022

The notifications centers is an HTTP to HTTP and HTTP to AMQP gateway for our CI infrastructure.

It allows to receive events from GitHub, Docker Hub and Phabricator, and send them to a RabbitMQ broker (white-rabbit) or Phabricator.

Requirements

Run it

Start a new container

run-notifications

Upgrade a live container

  1. Enter notifications container as app user : docker exec -it --user=app notifications bash
  2. Check we're on the master branch and the history is clean
  3. Update code rebasing the production branch against origin/master
  4. If composer has been touched, composer update --ignore-platform-reqs --no-dev (be careful if so, it could be faster to instead do a docker pull, stop this container and start a new container)
  5. If entered as root, fix ownership with chown -R app:app /var/wwwroot/default
  6. Restart PHP FPM with sv restart php-fpm
  7. Run php artisan optimize to refresh compiled class
  8. Run php artisan config:cache to cache updated config, to use if preferences are updated in the code or by us
  9. Restart again PHP FPM with sv restart php-fpm

For longer upgrade, you can put the application in maintenance mode with php artisan down.

How to rebase against master?

  1. update master to match origin/master
  2. rebase production against master
$ git fetch
$ git checkout master
$ git rebase origin/master
$ git checkout production
$ git rebase master

If there is a merge conflict:

  1. Edit the conflict files
  2. git add <edited files>
  3. git rebase --continue

Configure it

Edit configuration

Configuration is stored in rOPS: pillar/notifications/config.sls.

It can be applied through Salt:

salt-call --local state.apply roles/paas-docker/containers/notifications

That will spin a new container too if the image or the rOPS config has been updated.

Add a new GitHub organization

  1. Generate a random string to be used as a secret token shared between GitHub and the notifications center
  2. On Dwellers, add credentials to /data/notifications/storage/app/credentials.json
  3. On GitHub, go to organization settings, then Webhooks, e.g. https://github.com/organizations/acme/settings/hooks
  4.  Add webhook:
    1.  URL should be https://notifications.nasqueron.org/gate/GitHub/Acme where Acme is the ucfirst name of the organization account
    2.  Your secret token go to secret field
    3. Switch from push events to "Send me everything".
    4. Let default settings: active, SSL verification, application/json
  5. Run the CLI client to receive notifications (`notifications` on Ysul, require to be in the `notifications` group, editable through /etc/group)
  6. GitHub will then fire immediately a ping, check in recent deliveries it's 200
  7. Fix and redeliver it if not
  8. Check in the CLI you've got a correct reply, e.g. [16:36:13] <Eglide/orgz> « Anything added dilutes everything else. » — GitHub Webhooks ping zen aphorism.

For payload delivery code returned by notifications.nasqueron.org, a 500 means probably a JSON syntax error in credentials.json, a 403 tokens don't match, note the center doesn't verify token if an empty string is left in the config (or you made a typo to the property).

Regenerate Docker Hub configuration

Docker Hub doesn't seem to currently offer a comprehensive machine-readable API, but the website uses JSON documents to get data, so we can login as user and use browser console to fetch and consolidate data we need.

Procedure: https://gist.github.com/dereckson/23181a751e014796c52a4011e2a7de22

Target file: /srv/notifications/storage/app/DockerHubBuildTriggers.js @ Docker server hosting the notifications center container (currently docker-001)

The notifications center won't currently parse this file, it's indented for a future improvement, as Docker Hub broke old token system.

Commit configuration changes to rTESTSPRODENV

The test repository rTESTSPRODENV contains a reference JSON document.

This document will be compared with https://notifications.nasqueron.org/config output and must be equals.

You can also regenerate it from current configuration, but please double check it against current config through git diff use.

$ curl https://notifications.nasqueron.org/config > data/notifications.config.json
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   354    0   354    0     0    263      0 --:--:--  0:00:01 --:--:--   263
$ git diff

Don't use jsonlint to pretty print the file.

It's probably a good idea to format to pretty print the JSON, but `jsonlint` badly encode URL:

curl https://notifications.nasqueron.org/config | jsonlint will be https:\\/\\/devcentral.nasqueron.org"

This double encoding is write, as PHP will decode the string this string as this:

$ psysh
Psy Shell v0.6.1 (PHP 5.6.29 — cli) by Justin Hileman
>>> $url = '"https:\\/\\/devcentral.nasqueron.org"'
=> ""https:\/\/devcentral.nasqueron.org""
>>> $url = '"https:\\\\/\\\\/devcentral.nasqueron.org"'
=> ""https:\\/\\/devcentral.nasqueron.org""
>>> json_decode($url)
=> "https:\/\/devcentral.nasqueron.org"

Links

  1. Source code
  2.  DevCentral board

Vault secrets migration.

This service have secrets. They should be properly migrated in Vault.