Free tool

Robots.txt generator

Decide which AI crawlers may read your site, bot by bot, and get a robots.txt that says exactly that.

  • 100% free
  • No signup
  • Instant results

Write your robots.txt

Step 1 of 3 Choose crawlers

Start from
OpenAI
  • GPTBot Model training
  • OAI-SearchBot AI search index
  • ChatGPT-User Fetch for a user
Anthropic
  • ClaudeBot Model training
  • Claude-SearchBot AI search index
  • Claude-User Fetch for a user
  • anthropic-ai Legacy token
Perplexity
  • PerplexityBot AI search index
  • Perplexity-User Fetch for a user
Google
  • Googlebot Search engine: blocking it drops you from search
  • Google-Extended Model training
Microsoft
  • Bingbot Search engine: blocking it drops you from search
Apple
  • Applebot AI search index
  • Applebot-Extended Model training
Meta
  • Meta-ExternalAgent Model training
  • Meta-ExternalFetcher Fetch for a user
Amazon
  • Amazonbot Model training
ByteDance
  • Bytespider Model training
Common Crawl
  • CCBot Model training

Fetch for a user. Fetches a person asks for: the owner says robots.txt may not apply.

Step 2 of 3 Paths and sitemap

Step 3 of 3 Your robots.txt 

 
Save it as robots.txt at the root of your domain. Then test the live file with the robots.txt checker
 
  
  
 
  
  
 
  
  
 
  
 
  
   
   
    
   
 
 
 
  
 
   How it works  
 
Search, training and user fetches
 
AI companies now run separate crawlers for each job, so you can stay in AI answers without giving your pages to model training.
 
 
 
  
Search crawlers
  
OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot and Applebot build the indexes AI answers cite. Block them and those answers cannot link to you.
  
Training crawlers
  
GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider and CCBot gather data to train models. Blocking them keeps you out of training, not out of search.
  
Why paths repeat
  
A crawler with its own group ignores the * group. So every path you keep private is written again inside each named group, or a named bot would read it.
 
  
 
 
 
  
 
 
Next step
  Watch AI crawler access in Rank Prompt  
The technical audit reads your robots.txt on every check and flags the day a release starts blocking an AI crawler.
 
 
 
Other free tools
  
 
  
 
 
   
  
 
  
 
   Free tool vs. full platform  
 
Go from one snapshot to always-on AI visibility
 
The free tool is a fast, honest starting point. Rank Prompt turns it into a program that tracks, compares, and improves how AI answers talk about you.
 
 
 
 
Right now
 
This free generator
  
     
  • The rules for one site
  • Read back in the browser
  • No watch on your live file
    •  
 
 
 
The full picture
 
Rank Prompt platform
  
     
  • Tracks visibility across ChatGPT, Gemini, Perplexity, Claude, Grok & Google AI Mode (AI Overviews)
  • Runs your full prompt set on a schedule, not one query, once
  • Competitor share-of-voice and rank movement over time
  • Alerts when you gain or lose ground, plus exportable reports
  • AI-optimized content, SEO audits, and citation insights
    •  
  Start free with Rank Prompt   
 
  
 
  
 
 
 
 
 
FAQ
 
Robots.txt Generator FAQ
  
 Still have questions? Contact our team 
 
 
 
  Will blocking GPTBot remove me from ChatGPT answers?   
 
No. GPTBot gathers training data. ChatGPT search uses OAI-SearchBot, and fetches a person asks for use ChatGPT-User. Allow those to stay in ChatGPT answers.
 
 
  Does Google-Extended control AI Overviews?   
 
No. Google says Google-Extended governs Gemini training and grounding, and does not affect Google Search. AI Overviews and AI Mode come from Googlebot's index.
 
 
  Do AI crawlers obey robots.txt?   
 
The major training and search crawlers say they do. Fetches a user triggers, like ChatGPT-User, Perplexity-User and Meta-ExternalFetcher, may not, by their owners' own documentation.
 
 
  Where does the file go?   
 
At the root of the domain, as /robots.txt. Each subdomain needs its own.
 
 
 
 
 
  
  
 
  
Know when a release blocks AI crawlers
 
Rank Prompt checks crawl access, schema and AI readiness on every tracked page, and tells you what changed.
 
 
  Start free    View pricing