Operations grimoire/Notifications center

The notifications centers is an HTTP to HTTP and HTTP to AMQP gateway for our CI infrastructure.

It allows to receive events from GitHub, Docker Hub and Phabricator, and send them to a RabbitMQ broker (white-rabbit) or Phabricator.

Requirements

• Docker engine
• RabbitMQ broker

Run it

Start a new container

run-notifications

Upgrade a live container

1. Enter notifications container as app user : docker exec -it --user=app notifications bash
2. Check we're on the master branch and the history is clean
3.  Update code rebasing the production branch against origin/master
4. If composer has been touched, composer update (could be faster to docker pull, stop this container and start a new container)
5.  If entered as root, fix ownership with chown -R app:app /var/wwwroot/default
6.  Restart PHP FPM with sv restart php-fpm
7. Run php optimize to refresh compiled class
8.  Restart again PHP FPM with sv restart php-fpm

For longer upgrade, you can put the application in maintenance mode with php artisan down.

If preferences are updated (in the code or by us), php artisan config:clear is needed.

How to rebase against master?

1. update master to match origin/master
2. rebase production against master

$ git fetch
$ git checkout master
$ git rebase origin/master
$ git checkout production
$ git rebase master

If there is a merge conflict:

1. Edit the conflict files
2. git add <edited files>
3. git rebase --continue

Configure it

Add a new GitHub organization

1. Generate a random string to be used as a secret token shared between GitHub and the notifications center
2. On Dwellers, add credentials to /data/notifications/storage/app/credentials.json
3. On GitHub, go to organization settings, then Webhooks, e.g. https://github.com/organizations/acme/settings/hooks
4.  Add webhook:
   1.  URL should be https://notifications.nasqueron.org/gate/GitHub/Acme where Acme is the ucfirst name of the organization account
   2.  Your secret token go to secret field
   3. Switch from push events to "Send me everything".
   4. Let default settings: active, SSL verification, application/json
5. Run the CLI client to receive notifications (`notifications` on Ysul, require to be in the `notifications` group, editable through /etc/group)
6. GitHub will then fire immediately a ping, check in recent deliveries it's 200
7. Fix and redeliver it if not
8. Check in the CLI you've got a correct reply, e.g. [16:36:13] <Eglide/orgz> « Anything added dilutes everything else. » — GitHub Webhooks ping zen aphorism.

For payload delivery code returned by notifications.nasqueron.org, a 500 means probably a JSON syntax error in credentials.json, a 403 tokens don't match, note the center doesn't verify token if an empty string is left in the config (or you made a typo to the property).

Links

1. Source code
2.  DevCentral board

Configuration as code.

This service should be properly defined in rOPS repository as a Salt state.

Vault secrets migration.

This service have secrets. They should be properly migrated in Vault.