The short answer
- OpenAI, Anthropic and Perplexity each publish separate user agents for training, search and user-requested fetches.
- Google-Extended controls use of your content in Gemini. It doesn’t affect Google Search or AI Overviews.
- llms.txt is a proposed convention. It’s cheap to add, but don’t expect it to change results on its own.
Who is crawling
The main AI companies now separate their crawlers by purpose and document them. These are the user agents you write rules for in robots.txt.
| Company | User agent | Used for |
|---|---|---|
| OpenAI | GPTBot | Training |
| OpenAI | OAI-SearchBot | ChatGPT search results |
| OpenAI | ChatGPT-User | Pages a user asks ChatGPT to open |
| Anthropic | ClaudeBot | Training |
| Anthropic | Claude-SearchBot | Search results in Claude |
| Anthropic | Claude-User | Pages a user asks Claude to open |
| Perplexity | PerplexityBot | Perplexity's search index |
| Perplexity | Perplexity-User | Pages a user asks for |
| Google-Extended | Use in Gemini models (a control token, not a separate crawler) | |
| Common Crawl | CCBot | An open dataset widely used for training |
Check each company's documentation before relying on this list, because names and behaviour change.
A sensible default
For most businesses that want customers to find them, the useful position is to allow the search and user-fetch agents, so you can appear and be cited in AI answers, and to decide separately about training.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: *
Allow: /
Sitemap: https://your-business.co.uk/sitemap.xml
Blocking Google-Extended doesn't remove you from Google Search or AI Overviews, which use the normal Googlebot. Block Googlebot and you disappear from both.
What about llms.txt?
llms.txt is a convention proposed in 2024: a Markdown file at the root of your site that lists your most useful pages, each with a one-line description, written for a language model to read. It's quick to write and does no harm. No major answer engine has committed to using it, so treat it as tidy housekeeping rather than a ranking lever.
robots.txt is a request
Well-behaved crawlers follow robots.txt, and nothing forces the rest to. If content must be kept out of AI systems, it shouldn't be public. For everything else, robots.txt is how you state your preference, and the main companies honour it.