Operations grimoire/RabbitMQ: Difference between revisions
From Nasqueron Agora
| (3 intermediate revisions by the same user not shown) | |||
| Line 14: | Line 14: | ||
* To redeploy containers, use <code>salt-call --local state.apply roles/paas-docker/containers/rabbitmq</code> | * To redeploy containers, use <code>salt-call --local state.apply roles/paas-docker/containers/rabbitmq</code> | ||
* To enable a plugin, [https://devcentral.nasqueron.org/diffusion/DRABBITMQ/browse/ | * To enable a plugin, [https://devcentral.nasqueron.org/diffusion/DRABBITMQ/browse/main/Dockerfile edit this Dockerfile] and rebuild image | ||
* To enable a new port for a new protocol or metrics, edit {{ops file|pillar/paas/ | * To enable a new port for a new protocol or metrics, edit relevant file under {{ops file|pillar/paas/}} | ||
* To bind queue or check what happens, use web interface | * To bind queue or check what happens, use web interface | ||
| Line 23: | Line 23: | ||
* Notifications center: intelligent bus to get notifications from GitHub, Phabricator, Jenkins, DockerHub, normalize them and publish them to RabbitMQ | * Notifications center: intelligent bus to get notifications from GitHub, Phabricator, Jenkins, DockerHub, normalize them and publish them to RabbitMQ | ||
* Wearg: IRC bot, read wearg-notifications queue to get notifications to publish them on Libera Chat channels | * Wearg: IRC bot, read wearg-notifications queue to get notifications to publish them on Libera Chat channels | ||
== Troubleshoot == | |||
=== Metrics === | |||
[https://grafana.nasqueron.org/d/Kn5xm-gZk/rabbitmq-overview?orgId=1&refresh=15s Grafana dashboard] | |||
If the ready messages is high, it means a queue is not read. As of August 2024, the only permanent queue is the one for notifications used by [[Wearg]]. If Wearg is offline or disconnected from RabbitMQ, it's expected to have those messages in the queue. | |||
=== Metrics doesn't work anymore === | |||
Base image disable metrics. They renamed once the configuration file to do so. | |||
If metrics disappears, check in the container the /etc/rabbitmq/conf.d/ directory and update [https://devcentral.nasqueron.org/diffusion/DRABBITMQ/browse/main/Dockerfile edit this Dockerfile] accordingly. | |||
As of July 2024, Prometheus plugin is also needed to allow data scraping. | |||
Latest revision as of 16:15, 4 August 2024
RabbitMQ is deployed through Docker.
Clusters
| Cluster name | Management interface | Description |
|---|---|---|
| white-rabbit | https://white-rabbit.nasqueron.org | Main cluster, used for CI and CD purpose |
Procedures
- To redeploy containers, use
salt-call --local state.apply roles/paas-docker/containers/rabbitmq - To enable a plugin, edit this Dockerfile and rebuild image
- To enable a new port for a new protocol or metrics, edit relevant file under rOPS: pillar/paas/
- To bind queue or check what happens, use web interface
Services using RabbitMQ
- white-rabbit, dev vhost
- Notifications center: intelligent bus to get notifications from GitHub, Phabricator, Jenkins, DockerHub, normalize them and publish them to RabbitMQ
- Wearg: IRC bot, read wearg-notifications queue to get notifications to publish them on Libera Chat channels
Troubleshoot
Metrics
If the ready messages is high, it means a queue is not read. As of August 2024, the only permanent queue is the one for notifications used by Wearg. If Wearg is offline or disconnected from RabbitMQ, it's expected to have those messages in the queue.
Metrics doesn't work anymore
Base image disable metrics. They renamed once the configuration file to do so.
If metrics disappears, check in the container the /etc/rabbitmq/conf.d/ directory and update edit this Dockerfile accordingly.
As of July 2024, Prometheus plugin is also needed to allow data scraping.
