Operations grimoire/Mastodon

From Nasqueron Agora

📕📁📜 Old technical information :: content warning

⌛ This Nasqueron Operations Grimoire page hasn't been updated for a long time.

☣ As our infrastructure evolves quickly, there is a good chance this information is outdated or now inaccurate. Be careful and consider update it.

➡️ To assert the information is still up-to-date or not, you can check the history of the relevant role in our Operations repository.

Logs

Web
docker logs --tail=100 --follow mastodon_web_1 | grep -v status=20 | grep -v Render | grep -v X-Accel-Mapping
Other containers
docker logs --tail=100 --follow mastodon_db_1
docker logs --tail=100 --follow mastodon_sidekiq_1
docker logs --tail=100 --follow mastodon_streaming_1
docker logs --tail=100 --follow mastodon_redis_1

Upgrade procedure

Backup

PostgreSQL
docker exec mastodon_db_1 pg_dump -Fc -U postgres postgres > mastodon-db-`date +%Y-%m-%d`.dump
Redis
Copy the rdb with cp /srv/data/mastodon/redis/dump.rdb dump.rdb.`date +%Y-%m-%d`

Build image

$ cd /opt/mastodon
$ git fetch
$ git rebase origin/master # or a version tag
$ docker-compose build

Deploy new containers

$ docker-compose run --rm web rails db:migrate
$ docker-compose up -d

Sometimes, the database migration is time consuming. In such case:

$ docker-compose run --rm -e SKIP_POST_DEPLOYMENT_MIGRATIONS=true web rails db:migrate
$ docker-compose up -d
$ docker-compose run --rm web rails db:migrate

Maintenance scripts

Lost 2FA

There is currently no interface or Rake task or helper script to reset 2FA.

You can so use the database:

$ cd /opt/mastodon
$ docker-compose run --rm db psql --host=db -U postgres
UPDATE users SET otp_required_for_login=false WHERE email='user@domain.tld';