Operations grimoire/Anubis
Anubis is a proxy to filter out AI scrapers requests. It will allow sites like DevCentral to stop to serve heavy traffic for LLM model training.
Challenges
We want to stop scraping traffic, but let legitimate traffic pass (e.g. CLI requests from Conduit).
Anubis is used by a lot of similar software forges from open-source projects: Anubis known instances, but it's unclear if someone already succeed to configure it for Phabricator.
It's our current field of investigation by Doba Gui and Dereckson.
At Wikimedia, Andrew Kostka (WMDE) has some experience deploying it, apparently for Wikibase products.
Implementation (by Doba Gui)
Isolation via UNIX Sockets
To avoid port conflicts (especially the default metrics port 9000), every instance is configured to use UNIX sockets for both traffic and metrics:
- Main socket: `/run/anubis/Template:Instance.sock`
- Metrics socket: `/run/anubis/Template:Instance-metrics.sock`
Dynamic Configuration
The Salt states resolve the target application's port from the `docker_containers` pillar. In `anubis_instances`, we define the target:
anubis_instances:
devcentral:
target:
service: phabricator
container: devcentral
Provisioning and Vault
Anubis API and Dashboard keys are provisioned in Vault using a helper script [scripts/fix_anubis_devcentral.sh](cci:7://file:///C:/Users/Gui%20Martinien/Downloads/operations-main/operations-main/scripts/fix_anubis_devcentral.sh:0:0-0:0) and retrieved by Salt during deployment.
Current work
- Doba deployed an experimental run manually on Dwellers to figure how we can build and configure Anubis
- Work to deploy Anubis with Salt: D3908
Troubleshoot
Each site must have its own Anubis instance
If you serve to protect two different domains, you need two Anubis instances, one per domain.
In December, we noticed strange errors about challenges not found trying to fire the same instance for two different targets from the same domain.
