robots.txt Generator
Create a correct robots.txt in seconds, including what AI crawlers can and can't do on your site.
Heads up: with every AI crawler blocked, ChatGPT search, Claude, Perplexity and Gemini can't read your pages, so they won't recommend or cite you.
How to use it
- Choose how AI crawlers should be treated.
- List any private paths, like an admin area or checkout pages.
- Add your sitemap address, download the file and upload it to your site's root, so it opens at
yoursite.com/robots.txt.
AI training vs AI search
AI companies run different crawlers for different jobs. Training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and others) collect pages to build future models. Search and assistant crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot) fetch pages to answer questions right now, and they're how your site gets cited. The middle option blocks the first group and leaves the second alone.
Common mistakes
Disallow: /underUser-agent: *blocks your whole site from Google. It's often left over from a staging site.- Using robots.txt to hide a page from Google. It stops crawling, not indexing. Use a
noindextag for pages that must not appear in search. - Blocking CSS or JavaScript files. Google needs them to see your page properly.
After uploading, test it with our AI crawler checker.
Frequently asked questions
Where does robots.txt go?
In the root of your domain: https://yoursite.com/robots.txt. Crawlers only look there, and each subdomain needs its own.
Will blocking GPTBot remove me from ChatGPT?
Not from ChatGPT search. GPTBot is for training. ChatGPT search uses OAI-SearchBot, and browsing uses ChatGPT-User. The middle option blocks training and keeps search.
Do all crawlers obey robots.txt?
The major search engines and AI companies do. It's a polite request, not a lock, so never rely on it to protect private data.
More free tools
Related guides
Check everything at once
RankHoot runs 60+ SEO, AI search, speed and security checks on your page and gives you copy-paste fixes.
Scan my site free