Operations grimoire/Notifications center
The notifications centers is an HTTP to HTTP and HTTP to AMQP gateway for our CI infrastructure.
It allows to receive events from GitHub, Docker Hub and Phabricator, and send them to a RabbitMQ broker (white-rabbit) or Phabricator.
Start a new container
Upgrade a live container
- Enter notifications container as app user :
docker exec -it --user=app notifications bash
- Check we're on the master branch and the history is clean
- Update code rebasing the production branch against origin/master
- If composer has been touched,
composer update --ignore-platform-reqs --no-dev(be careful if so, it could be faster to instead do a docker pull, stop this container and start a new container)
- If entered as root, fix ownership with
chown -R app:app /var/wwwroot/default
- Restart PHP FPM with
sv restart php-fpm
php artisan optimizeto refresh compiled class
php artisan config:cacheto cache updated config, to use if preferences are updated in the code or by us
- Restart again PHP FPM with
sv restart php-fpm
For longer upgrade, you can put the application in maintenance mode with
php artisan down.
How to rebase against master?
- update master to match origin/master
- rebase production against master
$ git fetch $ git checkout master $ git rebase origin/master $ git checkout production $ git rebase master
If there is a merge conflict:
- Edit the conflict files
git add <edited files>
git rebase --continue
It can be applied through Salt:
salt-call --local state.apply roles/paas-docker/containers/notifications
That will spin a new container too if the image or the rOPS config has been updated.
Add a new GitHub organization
- Generate a random string to be used as a secret token shared between GitHub and the notifications center
- On Dwellers, add credentials to /data/notifications/storage/app/credentials.json
- On GitHub, go to organization settings, then Webhooks, e.g. https://github.com/organizations/acme/settings/hooks
- Add webhook:
- URL should be https://notifications.nasqueron.org/gate/GitHub/Acme where Acme is the ucfirst name of the organization account
- Your secret token go to secret field
- Switch from push events to "Send me everything".
- Let default settings: active, SSL verification, application/json
- Run the CLI client to receive notifications (`notifications` on Ysul, require to be in the `notifications` group, editable through /etc/group)
- GitHub will then fire immediately a ping, check in recent deliveries it's 200
- Fix and redeliver it if not
- Check in the CLI you've got a correct reply, e.g. [16:36:13] <Eglide/orgz> « Anything added dilutes everything else. » — GitHub Webhooks ping zen aphorism.
For payload delivery code returned by notifications.nasqueron.org, a 500 means probably a JSON syntax error in credentials.json, a 403 tokens don't match, note the center doesn't verify token if an empty string is left in the config (or you made a typo to the property).
Regenerate Docker Hub configuration
Docker Hub doesn't seem to currently offer a comprehensive machine-readable API, but the website uses JSON documents to get data, so we can login as user and use browser console to fetch and consolidate data we need.
Target file: /srv/notifications/storage/app/DockerHubBuildTriggers.js @ Docker server hosting the notifications center container (currently docker-001)
The notifications center won't currently parse this file, it's indented for a future improvement, as Docker Hub broke old token system.
Commit configuration changes to rTESTSPRODENV
The test repository rTESTSPRODENV contains a reference JSON document.
This document will be compared with https://notifications.nasqueron.org/config output and must be equals.
You can also regenerate it from current configuration, but please double check it against current config through
git diff use.
$ curl https://notifications.nasqueron.org/config > data/notifications.config.json % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 354 0 354 0 0 263 0 --:--:-- 0:00:01 --:--:-- 263 $ git diff
Don't use jsonlint to pretty print the file.
It's probably a good idea to format to pretty print the JSON, but `jsonlint` badly encode URL:
curl https://notifications.nasqueron.org/config | jsonlint will be
This double encoding is write, as PHP will decode the string this string as this:
$ psysh Psy Shell v0.6.1 (PHP 5.6.29 — cli) by Justin Hileman >>> $url = '"https:\\/\\/devcentral.nasqueron.org"' => ""https:\/\/devcentral.nasqueron.org"" >>> $url = '"https:\\\\/\\\\/devcentral.nasqueron.org"' => ""https:\\/\\/devcentral.nasqueron.org"" >>> json_decode($url) => "https:\/\/devcentral.nasqueron.org"