Build log/GEO

robots.txt, llms.txt and the AI crawlers: what to allow

AI companies run separate crawlers for training and for live answers. You can block one without disappearing from the other.

The short answer

  • OpenAI, Anthropic and Perplexity each publish separate user agents for training, search and user-requested fetches.
  • Google-Extended controls use of your content in Gemini. It doesn’t affect Google Search or AI Overviews.
  • llms.txt is a proposed convention. It’s cheap to add, but don’t expect it to change results on its own.

Who is crawling

The main AI companies now separate their crawlers by purpose and document them. These are the user agents you write rules for in robots.txt.

CompanyUser agentUsed for
OpenAIGPTBotTraining
OpenAIOAI-SearchBotChatGPT search results
OpenAIChatGPT-UserPages a user asks ChatGPT to open
AnthropicClaudeBotTraining
AnthropicClaude-SearchBotSearch results in Claude
AnthropicClaude-UserPages a user asks Claude to open
PerplexityPerplexityBotPerplexity's search index
PerplexityPerplexity-UserPages a user asks for
GoogleGoogle-ExtendedUse in Gemini models (a control token, not a separate crawler)
Common CrawlCCBotAn open dataset widely used for training

Check each company's documentation before relying on this list, because names and behaviour change.

A sensible default

For most businesses that want customers to find them, the useful position is to allow the search and user-fetch agents, so you can appear and be cited in AI answers, and to decide separately about training.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

Sitemap: https://your-business.co.uk/sitemap.xml

Blocking Google-Extended doesn't remove you from Google Search or AI Overviews, which use the normal Googlebot. Block Googlebot and you disappear from both.

What about llms.txt?

llms.txt is a convention proposed in 2024: a Markdown file at the root of your site that lists your most useful pages, each with a one-line description, written for a language model to read. It's quick to write and does no harm. No major answer engine has committed to using it, so treat it as tidy housekeeping rather than a ranking lever.

robots.txt is a request

Well-behaved crawlers follow robots.txt, and nothing forces the rest to. If content must be kept out of AI systems, it shouldn't be public. For everything else, robots.txt is how you state your preference, and the main companies honour it.

Build yours with the robots.txt builder in the GEO guide.
Written by Flavoureak

We build and run Estimark, CurioKids and CalcStack, and we build web and mobile software for other businesses. Start a project.