Operations grimoire/Sentry

Sentry is deployed on docker-002 through the Docker PaaS.

The Sentry DSN or the humans UI can be reached at https://sentry.nasqueron.org

Command line

Sentry provides a command line interface.

To use it, log in to docker-002, then run sentry nasqueron <command>. It will then spawn a new container correctly linked to the database, SMTP server and PostgreSQL database to run it. TODO: update the sentry wrapper to use the sentry network and get the right environment instead.

If you've deployed another instance than for Nasqueron, replace it by the realm name.

The following commands are available in 23.3:

 cleanup       Delete a portion of trailing data based on creation date.
 config        Manage runtime config options.
 createuser    Create a new user.
 devserver     Starts a lightweight web server for development.
 devservices   Manage dependent development services required for Sentry.
 django        Execute Django subcommands.
 exec          Execute a script.
 execfile      Execute a script.
 export        Exports core metadata for the Sentry installation.
 files         Manage files from filestore.
 help          Show this message and exit.
 import        Imports data from a Sentry export.
 init          Initialize new configuration directory.
 killswitches  Manage killswitches for ingestion pipeline.
 migrations    Manage migrations.
 performance   Performance utilities
 permissions   Manage Permissions for Users.
 plugins       Manage Sentry plugins.
 queues        Manage Sentry queues.
 repair        Attempt to repair any invalid data.
 run           Run a service.
 sendmail      Sends emails from the default notification mail address.
 shell         Run a Python interactive interpreter.
 spans         Span utilities
 start         DEPRECATED see `sentry run` instead.
 tsdb          Tools for interacting with the time series database.
 upgrade       Perform any pending database migrations and upgrades.
 write-hashes  Runs span hash grouping on event data in the supplied filename using the...

Kafka and ZooKeeper

• Deployment pitfall: ZooKeeper wants a clean log if you initialise it, if not it will shutdown. Ensure it has always a data volume or an empty log when starting it.

• Kafka has a 20 seconds pause before initializing the topics, to let the cluster correctly start.

To reprovision Kafka and ZooKeeper from scratch, the following works fine:

$ rm -rf /srv/kafka/sentry_kafka /srv/zookeeper/sentry_zookeeper
$ deploy-container zookeeper
$ deploy-container kafka

How to upgrade Sentry?

On Docker Hub or through Dwellers and our local registry, build a new nasqueron/sentry image.

On the Docker engine, perform database migration first, then upgrade the container:

1. Pull the new image: docker pull nasqueron/sentry
2. Run migrations: sentry nasqueron upgrade (should we stop Sentry containers first?)
3. Deploy containers: deploy-container sentry

Deployment of Sentry containers

Sentry containers are currently deployed to docker-002. Pillar configuration and containers definitions are based on the getsentry/self-hosted repository.

The containers ARE NOT directly deployed USING the getsentry/self-hosted repository or the scripts it provides. Containers are regularly managed like any other container through Salt in paas-docker role. Instead, the getsentry/self-hosted repository is used as reference implementation documentation and any change needs to be reflected to our own configuration.

Correspondance repositories

rOPS	getsentry/self-hosted	Sentry version
D2907	4d318761a84c	23.4.0.dev
D2881	06662c6cdc85	23.2.0