Operations grimoire/Anubis
Anubis is a proxy to filter out AI scrapers requests. It will allow sites like DevCentral to stop to serve heavy traffic for LLM model training.
We're currently deploying it to our infrastructure
Challenges
We want to stop scraping traffic, but let legitimate traffic pass (e.g. CLI requests from Conduit).
Anubis is used by a lot of similar software forges from open-source projects: Anubis known instances, but it's unclear if someone already succeed to configure it for Phabricator.
It's our current field of investigation by Doba Gui and Dereckson.
At Wikimedia, Andrew Kostka (WMDE) has some experience deploying it, apparently for Wikibase products.
Current work
- Doba deployed an experimental run manually on Dwellers to figure how we can build and configure Anubis
- Work to deploy Anubis with Salt: D3908
Troubleshoot
Each site must have its own Anubis instance
If you serve to protect two different domains, you need two Anubis instances, one per domain.
In December, we noticed strange errors about challenges not found trying to fire the same instance for two different targets from the same domain.
