Free tools

AI Crawler Checker

Find out whether ChatGPT, Claude, Perplexity, Gemini and other AI tools are allowed to read a page, and what to change if they aren't.

How to use it

  1. Paste your homepage or any important page.
  2. RankHoot reads your robots.txt and checks the rules for each AI crawler, exactly as the crawler would.
  3. If a crawler you want is blocked, copy the suggested lines into your robots.txt.

What it checks

  • Each AI crawler's access: GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended (Gemini), Applebot-Extended and CCBot, plus Googlebot and Bingbot for comparison.
  • llms.txt: whether you have the short summary file some AI tools look for.
  • noindex: a page marked noindex won't be used by search-based AI answers either.
  • Content in the HTML: most AI crawlers don't run JavaScript. If your text only appears after scripts run, they see an empty page.

Blocked by accident is common

Many sites block AI crawlers without meaning to: a security plugin, a template's default robots.txt, or a CDN setting that blocks "AI bots" with one switch. Search-and-answer crawlers like OAI-SearchBot and PerplexityBot are how your pages get cited in AI answers, so blocking them makes you invisible there. Training crawlers like GPTBot are a separate choice: you can block those and still be found. Our robots.txt generator sets that up for you.

Frequently asked questions

What is the difference between GPTBot and OAI-SearchBot?

GPTBot collects pages to train OpenAI models. OAI-SearchBot finds pages to show and cite in ChatGPT search. You can block GPTBot and allow OAI-SearchBot.

Does blocking AI crawlers hurt my Google ranking?

No. Google-Extended only controls use in Gemini and AI training. Googlebot, which ranks you in Google Search, is separate.

My robots.txt allows everything. Why can't AI see my content?

If your page is built with JavaScript and the HTML is nearly empty, most AI crawlers see nothing. The checker flags this as "Needs JavaScript". Pre-rendering or server-side rendering fixes it.

More free tools

Is Your Website Visible to ChatGPT? A 5-Point Check22 November 2026 How to Get Your Business Recommended by ChatGPT22 November 2026 robots.txt for AI Crawlers: GPTBot, ClaudeBot and More22 November 2026

Check everything at once

RankHoot runs 60+ SEO, AI search, speed and security checks on your page and gives you copy-paste fixes.

Scan my site free