AI Crawler Checker
Find out whether ChatGPT, Claude, Perplexity, Gemini and other AI tools are allowed to read a page, and what to change if they aren't.
How to use it
- Paste your homepage or any important page.
- RankHoot reads your
robots.txtand checks the rules for each AI crawler, exactly as the crawler would. - If a crawler you want is blocked, copy the suggested lines into your robots.txt.
What it checks
- Each AI crawler's access: GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended (Gemini), Applebot-Extended and CCBot, plus Googlebot and Bingbot for comparison.
- llms.txt: whether you have the short summary file some AI tools look for.
- noindex: a page marked noindex won't be used by search-based AI answers either.
- Content in the HTML: most AI crawlers don't run JavaScript. If your text only appears after scripts run, they see an empty page.
Blocked by accident is common
Many sites block AI crawlers without meaning to: a security plugin, a template's default robots.txt, or a CDN setting that blocks "AI bots" with one switch. Search-and-answer crawlers like OAI-SearchBot and PerplexityBot are how your pages get cited in AI answers, so blocking them makes you invisible there. Training crawlers like GPTBot are a separate choice: you can block those and still be found. Our robots.txt generator sets that up for you.
Frequently asked questions
What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects pages to train OpenAI models. OAI-SearchBot finds pages to show and cite in ChatGPT search. You can block GPTBot and allow OAI-SearchBot.
Does blocking AI crawlers hurt my Google ranking?
No. Google-Extended only controls use in Gemini and AI training. Googlebot, which ranks you in Google Search, is separate.
My robots.txt allows everything. Why can't AI see my content?
If your page is built with JavaScript and the HTML is nearly empty, most AI crawlers see nothing. The checker flags this as "Needs JavaScript". Pre-rendering or server-side rendering fixes it.
More free tools
Related guides
Check everything at once
RankHoot runs 60+ SEO, AI search, speed and security checks on your page and gives you copy-paste fixes.
Scan my site free