Operations grimoire/Restart a Docker engine

From Nasqueron Agora
Revision as of 20:07, 21 July 2016 by Dereckson (talk | contribs) (Created page with "A lot of vital components belong to Docker. Ideally, we should have redundancy to avoid container lost. # Restart Docker engine # Restart containers, either manually, either...")
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

A lot of vital components belong to Docker.

Ideally, we should have redundancy to avoid container lost.

  1. Restart Docker engine
  2. Restart containers, either manually, either with docker-containers systemd unit
  3. Run production tests
  4. Run

Services needing manual tweaking after container restart

Phabricator instances

DevCentral needs:

  1. bin/phd status
  2. bin/phd stop <the PID of PhabricatorBot>
  3. sv restart phd
  4. chpst -u app bin/phd launch PhabricatorBot /opt/phabricator/conf/xessife.json

Other instances needs a sv restart phd.

Broker

Wearg needs to be manually reconnected to the broker:

  1. .tcl mq disconnect
  2. .tcl utimers and kill remaining timers if needed
  3. .tcl mq broker::connect
  4. .tcl broker::on_tick

That requires a owner access to Wearg (ping Dereckson).

Troubleshooting

When MySQL isn't reacheable

If the MySQL container (acquisitariat) IP changed, you need to tweak /etc/hosts in every depending container (Phabricator instances, cachet, pad, login) and ensure there is a <correct IP> mysql line.

When a container doesn't want to restart

Try first to see what happens with docker logs <container name>.

If that doesn't work, go to the services section of the grimoire and reprovision it.

Commit a backup Docker image based on the current content with with docker commit <name> <name>-bak. That will allow investigation.