Free tool
Free Robots.txt AI Bot Checker
Paste your URL and instantly see which AI crawlers can access your site. We read your live robots.txt and check 15+ bots (GPTBot, ClaudeBot, PerplexityBot, and more) so you know if you're accidentally blocking AI traffic.
Analyze your robots.txt
- 100% free
- No signup
- Instant results
Why AI bots matter
Crawler access in robots.txt is the first gate for training and retrieval use cases.
GPTBot & ChatGPT
OpenAI's crawlers (including GPTBot and related user agents) collect public pages for ChatGPT browsing and model-related use. If they're disallowed, ChatGPT may be less likely to rely on your site as a source.
ClaudeBot
Anthropic uses ClaudeBot and Anthropic-AI to retrieve web content for Claude. Your robots.txt is the first gate for whether those agents may fetch your pages at all.
PerplexityBot
PerplexityBot powers Perplexity's live answers and citations. Blocking it can remove you from those answer panels, sometimes intentionally, sometimes by accident alongside a broad User-agent: * rule.
Each bot follows its own User-agent group when your file has one. A bot without one follows the * group, which we mark as "via *". Consecutive User-agent lines share one group, and the longest matching Allow or Disallow rule wins, as RFC 9309 specifies.
Next step
Audit the rest of your technical setuprobots.txt is one signal. The full audit covers structured data, rendering and Core Web Vitals for AI crawlers.
Other free tools
- Free llms.txt generator
A cheap companion file that some agent and developer tools read.
- Free AI readiness checker
Score your whole site for AI readiness, not just robots.txt.
Go from one snapshot to always-on AI visibility
The free tool is a fast, honest starting point. Rank Prompt turns it into a program that tracks, compares, and improves how AI answers talk about you.
Right now
This free checker
- Checks 15+ AI crawlers in your robots.txt
- Live fetch, no signup
- A one-time read, not monitoring
One-time snapshot
The full picture
Rank Prompt platform
- Tracks visibility across ChatGPT, Gemini, Perplexity, Claude, Grok & Google AI Mode
- Runs your full prompt set on a schedule, not one query, once
- Competitor share-of-voice and rank movement over time
- Alerts when you gain or lose ground, plus exportable reports
- AI-optimized content, SEO audits, and citation insights
What are AI bots and why do they crawl my website?
AI bots (like GPTBot, ClaudeBot, and PerplexityBot) are automated crawlers sent by AI companies to index web content. This data helps train AI models and provide real-time answers. If your robots.txt blocks these bots, AI assistants may not be able to reference or recommend your content when users ask relevant questions.
Which AI bots should I allow in my robots.txt?
At minimum, allow GPTBot (ChatGPT), ClaudeBot (Claude by Anthropic), PerplexityBot, and Google-Extended (Gemini). These cover the major AI platforms where buyers discover brands. Blocking them means your content won't be included in AI-generated recommendations, which is increasingly where purchase decisions start.
How do I unblock AI bots in my robots.txt?
Open your robots.txt file (usually at yoursite.com/robots.txt) and remove any Disallow rules for AI bot user agents. For example, if you see "User-agent: GPTBot / Disallow: /", change the Disallow line to "Allow: /" or remove the block entirely. If the bot isn't mentioned at all, it follows your User-agent: * group, and with no * group it may crawl everything.
Will allowing AI bots hurt my website performance or security?
No. AI bots behave similarly to search engine crawlers like Googlebot. They read your public pages but don't modify anything. Their crawl rate is typically low and won't impact server performance. Allowing them simply ensures your content can be referenced when AI assistants answer relevant questions.
What do "via *" and "no rule" mean in the results?
"Allowed via *" or "Blocked via *" means your robots.txt has no group for that bot, so it follows your User-agent: * group, as crawlers that follow RFC 9309 do. "Allowed, no rule" means there is neither a group for the bot nor a * group, so nothing stops it from crawling. A bot with its own group ignores the * group entirely, so repeat any shared Disallow lines inside each named group.
Go beyond a one-off robots check
Monitor AI visibility over time, compare to competitors, and act on clear recommendations, not just a single static file.