Short answer AI companies use separate user agents for training and for live search. To appear in AI answers but stay out of training data, allow the search agents (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User) and disallow the training ones (GPTBot, ClaudeBot, Google-Extended, CCBot). Blocking everything also removes you from AI search results.
The main AI user agents
| User agent | Company | What it does |
|---|---|---|
GPTBot | OpenAI | Collects content that may be used to train models |
OAI-SearchBot | OpenAI | Indexes pages for ChatGPT search results |
ChatGPT-User | OpenAI | Fetches a page when a user's conversation needs it |
ClaudeBot | Anthropic | Collects content that may be used for training |
Claude-SearchBot | Anthropic | Indexes pages for Claude's search results |
Claude-User | Anthropic | Fetches a page when a Claude user asks for it |
PerplexityBot | Perplexity | Indexes pages for Perplexity answers |
Perplexity-User | Perplexity | Fetches a page for a user's question |
Google-Extended | Not a crawler: a token that controls whether Googlebot's content is used for Gemini training. It does not affect Google Search. | |
Applebot-Extended | Apple | Same idea for Apple's AI training |
CCBot | Common Crawl | Open web archive that many AI datasets are built from |
Companies add and rename agents from time to time, so check each one's own documentation once a year.
Option 1: allow everything (best for visibility)
If you want the widest possible reach, a simple file is enough:
User-agent: *
Allow: /
Disallow: /api/
Sitemap: https://example.com/sitemap.xml
Option 2: allow AI search, block training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml
The search and user agents fall under * and stay allowed.
Common mistakes
- Blocking "all bots" with a security plugin or firewall rule. Cloudflare and some hosts have one-click AI bot blocking. It may be switched on without you knowing; check your dashboard.
- Forgetting the group rule. A bot obeys only the most specific group that names it. If you write a
GPTBotgroup, the*rules no longer apply to GPTBot. - Thinking robots.txt hides pages. It's a request, not a lock. Use authentication for private content.
- Blocking Googlebot to stop AI Overviews. That removes you from Google Search entirely. Google-Extended doesn't control AI Overviews either.
Check your rules
RankHoot reads your robots.txt and shows which AI and search crawlers are allowed or blocked on your page, along with JavaScript-only content that AI crawlers can't see.