AI scrapers have been bullying the internet into oblivion and there's not much we can do about it. The well-behaved bots will relent when you ask them

Block AI scrapers with Anubis

submited by

Style Pass

2025-01-19 14:30:04

AI scrapers have been bullying the internet into oblivion and there's not much we can do about it. The well-behaved bots will relent when you ask them to, add entries to your robots.txt (even though they should understand the intent behind a wildcard), or block their user agents.

A majority of the AI scrapers are not well-behaved, and they will ignore your robots.txt, ignore your User-Agent blocks, and ignore your X-Robots-Tag headers. They will scrape your site until it falls over, and then they will scrape it some more. They will click every link on every link on every link viewing the same pages over and over and over and over. Some of them will even click on the same link multiple times in the same second. It's madness and unsustainable.

I got tired of this and made a tool to stop them for good. I call it Anubis. Anubis weighs the soul of your connection using a sha256 proof-of-work challenge in order to protect upstream resources from scraper bots. It's a reverse proxy that requires browsers and bots to solve a proof-of-work challenge before they can access your site, just like Hashcash.

If you would like to purchase commercial support for Anubis including an unbranded or custom branded version (namely one without the happy anime girl), please contact me.