Search code, repositories, users, issues, pull requests...

The AI/LLM bot blocker web server, firewall, and robots.txt config generator used in production by the Ichido Search Engine. These configs block known large AI and LLM bots from accessing your site content, while still allowing classical search engines and legitimate users to access content. Supports the following web servers, firewalls, and standards:

In total there are 6 variants of config files, of which you'll only need 2 with the minimal config (1 web server config and 1 robots.txt), or 3 with the full config (1 web server config, 1 robots.txt, and 1 firewall config). The minimal config will block most AI bots with a low false positive rate, and the full config that aggressively blocks AI bots and site scrapers, but will likely have many more false positives (due to blocking of all IP addresses from large cloud vendors that most LLMs operate on). It is recommended for most use cases to use the minimal config.

The config files can be built manually from source, or prebuilt files can be downloaded from Ichido's file server. Minimal config prebuilt files are prefixed with minimal- and full files with full-. Below are instructions for applying the configurations.

Leave a Comment