AI's Wide Shut · Chapter 9
robots.txt for AI crawlers
The book prints a starting point and says new crawlers appear regularly. This is the version kept current. Last checked 13 October 2026.
The file
Save it as robots.txt at the top level of your site, so it's at
yourdomain.com/robots.txt. If you already have one, add these groups to it.
# Block AI training crawlers
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: Bytespider
Disallow: /The first six are the book's, unchanged; the book's own version is on the Chapter 9 code page. The rest were added here.
What's in it
| User-agent | Company | Where it's from |
|---|---|---|
GPTBot | OpenAI | The book |
ClaudeBot | Anthropic | The book |
Google-Extended | The book | |
Applebot-Extended | Apple | The book |
Meta-ExternalAgent | Meta | The book |
CCBot | Common Crawl | The book |
Amazonbot | Amazon | Added here |
Bytespider | ByteDance | Added here |
What it doesn't block
This file blocks crawlers that collect content for training. Several companies use
different names for their AI search features and for pages fetched when someone asks their
assistant a question — for example OpenAI's OAI-SearchBot and
ChatGPT-User, and Amazon's Amzn-SearchBot. Leaving those alone keeps
your site in AI search results, which is the choice the book's version makes. If you'd rather
be out of those too, add their names the same way.
Changes
- 13 October 2026 — added
AmazonbotandBytespider.