Free tools

robots.txt Generator

Create a correct robots.txt in seconds, including what AI crawlers can and can't do on your site.

AI crawlers
Your robots.txt

How to use it

  1. Choose how AI crawlers should be treated.
  2. List any private paths, like an admin area or checkout pages.
  3. Add your sitemap address, download the file and upload it to your site's root, so it opens at yoursite.com/robots.txt.

AI training vs AI search

AI companies run different crawlers for different jobs. Training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and others) collect pages to build future models. Search and assistant crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot) fetch pages to answer questions right now, and they're how your site gets cited. The middle option blocks the first group and leaves the second alone.

Common mistakes

  • Disallow: / under User-agent: * blocks your whole site from Google. It's often left over from a staging site.
  • Using robots.txt to hide a page from Google. It stops crawling, not indexing. Use a noindex tag for pages that must not appear in search.
  • Blocking CSS or JavaScript files. Google needs them to see your page properly.

After uploading, test it with our AI crawler checker.

Frequently asked questions

Where does robots.txt go?

In the root of your domain: https://yoursite.com/robots.txt. Crawlers only look there, and each subdomain needs its own.

Will blocking GPTBot remove me from ChatGPT?

Not from ChatGPT search. GPTBot is for training. ChatGPT search uses OAI-SearchBot, and browsing uses ChatGPT-User. The middle option blocks training and keeps search.

Do all crawlers obey robots.txt?

The major search engines and AI companies do. It's a polite request, not a lock, so never rely on it to protect private data.

More free tools

robots.txt for AI Crawlers: GPTBot, ClaudeBot and More22 November 2026

Check everything at once

RankHoot runs 60+ SEO, AI search, speed and security checks on your page and gives you copy-paste fixes.

Scan my site free