Operations grimoire/Restart a Docker engine
A lot of vital components belong to Docker.
Ideally, we should have redundancy to avoid container lost.
- Restart Docker engine
- Restart containers, either manually, either with docker-containers systemd unit
- Run production tests
- Run
Services needing manual tweaking after container restart
Phabricator instances
DevCentral needs:
- bin/phd status
- bin/phd stop <the PID of PhabricatorBot>
- sv restart phd
- chpst -u app bin/phd launch PhabricatorBot /opt/phabricator/conf/xessife.json
Other instances needs a sv restart phd.
Broker
Wearg needs to be manually reconnected to the broker:
- .tcl mq disconnect
- .tcl utimers and kill remaining timers if needed
- .tcl mq broker::connect
- .tcl broker::on_tick
That requires a owner access to Wearg (ping Dereckson).
Troubleshooting
When MySQL isn't reacheable
If the MySQL container (acquisitariat) IP changed, you need to tweak /etc/hosts in every depending container (Phabricator instances, cachet, pad, login) and ensure there is a <correct IP> mysql line.
When a container doesn't want to restart
Try first to see what happens with docker logs <container name>.
If that doesn't work, go to the services section of the grimoire and reprovision it.
Commit a backup Docker image based on the current content with with docker commit <name> <name>-bak. That will allow investigation.
