Blog

robots.txt for AI Crawlers: GPTBot, ClaudeBot and More

22 November 2026 ยท 2 min read

Short answer AI companies use separate user agents for training and for live search. To appear in AI answers but stay out of training data, allow the search agents (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User) and disallow the training ones (GPTBot, ClaudeBot, Google-Extended, CCBot). Blocking everything also removes you from AI search results.

The main AI user agents

User agentCompanyWhat it does
GPTBotOpenAICollects content that may be used to train models
OAI-SearchBotOpenAIIndexes pages for ChatGPT search results
ChatGPT-UserOpenAIFetches a page when a user's conversation needs it
ClaudeBotAnthropicCollects content that may be used for training
Claude-SearchBotAnthropicIndexes pages for Claude's search results
Claude-UserAnthropicFetches a page when a Claude user asks for it
PerplexityBotPerplexityIndexes pages for Perplexity answers
Perplexity-UserPerplexityFetches a page for a user's question
Google-ExtendedGoogleNot a crawler: a token that controls whether Googlebot's content is used for Gemini training. It does not affect Google Search.
Applebot-ExtendedAppleSame idea for Apple's AI training
CCBotCommon CrawlOpen web archive that many AI datasets are built from

Companies add and rename agents from time to time, so check each one's own documentation once a year.

Option 1: allow everything (best for visibility)

If you want the widest possible reach, a simple file is enough:

User-agent: *
Allow: /
Disallow: /api/

Sitemap: https://example.com/sitemap.xml

Option 2: allow AI search, block training

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

The search and user agents fall under * and stay allowed.

Common mistakes

  • Blocking "all bots" with a security plugin or firewall rule. Cloudflare and some hosts have one-click AI bot blocking. It may be switched on without you knowing; check your dashboard.
  • Forgetting the group rule. A bot obeys only the most specific group that names it. If you write a GPTBot group, the * rules no longer apply to GPTBot.
  • Thinking robots.txt hides pages. It's a request, not a lock. Use authentication for private content.
  • Blocking Googlebot to stop AI Overviews. That removes you from Google Search entirely. Google-Extended doesn't control AI Overviews either.

Check your rules

RankHoot reads your robots.txt and shows which AI and search crawlers are allowed or blocked on your page, along with JavaScript-only content that AI crawlers can't see.

Try it free: robots.txt GeneratorCreate a correct robots.txt in seconds, including what AI crawlers can and can't do on your site.
What Is llms.txt? How to Create One (With Example)22 November 2026 How to Get Your Website on Google Fast (New Site Guide)22 November 2026 How to Find Broken Links on Your Website (Free)22 November 2026

Check your site in 20 seconds

RankHoot runs 60+ SEO, AI search, speed and security checks and gives you copy-paste fixes.

Scan my site