Operations grimoire/Notifications center

The notifications centers is an HTTP to HTTP and HTTP to AMQP gateway for our CI infrastructure.

It allows to receive events from GitHub, Docker Hub and Phabricator, and send them to a RabbitMQ broker (white-rabbit) or Phabricator.

Requirements

• Docker engine
• RabbitMQ broker

Run it

Start a new container

run-notifications

Upgrade a live container

1. Enter notifications container as app user : docker exec -it --user=app notifications bash
2. Check we're on the master branch and the history is clean
3. Update code rebasing the production branch against origin/master
4. If composer has been touched, composer update --ignore-platform-reqs --no-dev (be careful if so, it could be faster to instead do a docker pull, stop this container and start a new container)
5. If entered as root, fix ownership with chown -R app:app /var/wwwroot/default
6. Restart PHP FPM with sv restart php-fpm
7. Run php artisan optimize to refresh compiled class
8. Run php artisan config:cache to cache updated config, to use if preferences are updated in the code or by us
9. Restart again PHP FPM with sv restart php-fpm

For longer upgrade, you can put the application in maintenance mode with php artisan down.

How to rebase against master?

1. update master to match origin/master
2. rebase production against master

$ git fetch
$ git checkout master
$ git rebase origin/master
$ git checkout production
$ git rebase master

If there is a merge conflict:

1. Edit the conflict files
2. git add <edited files>
3. git rebase --continue

Configure it

Edit configuration

Configuration is stored in rOPS: pillar/notifications/config.sls.

It can be applied through Salt:

salt-call --local state.apply roles/paas-docker/containers/notifications

That will spin a new container too if the image or the rOPS config has been updated.

Add a new GitHub organization

1. Generate a random string to be used as a secret token shared between GitHub and the notifications center
2. On Dwellers, add credentials to /data/notifications/storage/app/credentials.json
3. On GitHub, go to organization settings, then Webhooks, e.g. https://github.com/organizations/acme/settings/hooks
4.  Add webhook:
   1.  URL should be https://notifications.nasqueron.org/gate/GitHub/Acme where Acme is the ucfirst name of the organization account
   2.  Your secret token go to secret field
   3. Switch from push events to "Send me everything".
   4. Let default settings: active, SSL verification, application/json
5. Run the CLI client to receive notifications (`notifications` on Ysul, require to be in the `notifications` group, editable through /etc/group)
6. GitHub will then fire immediately a ping, check in recent deliveries it's 200
7. Fix and redeliver it if not
8. Check in the CLI you've got a correct reply, e.g. [16:36:13] <Eglide/orgz> « Anything added dilutes everything else. » — GitHub Webhooks ping zen aphorism.

For payload delivery code returned by notifications.nasqueron.org, a 500 means probably a JSON syntax error in credentials.json, a 403 tokens don't match, note the center doesn't verify token if an empty string is left in the config (or you made a typo to the property).

Regenerate Docker Hub configuration

Docker Hub doesn't seem to currently offer a comprehensive machine-readable API, but the website uses JSON documents to get data, so we can login as user and use browser console to fetch and consolidate data we need.

Procedure: https://gist.github.com/dereckson/23181a751e014796c52a4011e2a7de22

Target file: /srv/notifications/storage/app/DockerHubBuildTriggers.js @ Docker server hosting the notifications center container (currently docker-001)

The notifications center won't currently parse this file, it's indented for a future improvement, as Docker Hub broke old token system.

Commit configuration changes to rTESTSPRODENV

The test repository rTESTSPRODENV contains a reference JSON document.

This document will be compared with https://notifications.nasqueron.org/config output and must be equals.

You can also regenerate it from current configuration, but please double check it against current config through git diff use.

$ curl https://notifications.nasqueron.org/config > data/notifications.config.json
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   354    0   354    0     0    263      0 --:--:--  0:00:01 --:--:--   263
$ git diff

Don't use jsonlint to pretty print the file.

It's probably a good idea to format to pretty print the JSON, but `jsonlint` badly encode URL:

curl https://notifications.nasqueron.org/config | jsonlint will be https:\\/\\/devcentral.nasqueron.org"

This double encoding is write, as PHP will decode the string this string as this:

$ psysh
Psy Shell v0.6.1 (PHP 5.6.29 — cli) by Justin Hileman
>>> $url = '"https:\\/\\/devcentral.nasqueron.org"'
=> ""https:\/\/devcentral.nasqueron.org""
>>> $url = '"https:\\\\/\\\\/devcentral.nasqueron.org"'
=> ""https:\\/\\/devcentral.nasqueron.org""
>>> json_decode($url)
=> "https:\/\/devcentral.nasqueron.org"

Links

1. Source code
2.  DevCentral board

Vault secrets migration.

This service have secrets. They should be properly migrated in Vault.