Free robots.txt AI bot checker

Paste your URL and instantly see which AI crawlers can access your site. We read your live robots.txt and check 19 bots (GPTBot, ClaudeBot, PerplexityBot, and more) so you know if you're accidentally blocking AI traffic.

  • 100% free
  • No signup
  • Instant results

Analyze your robots.txt

Why AI bots matter

  1. GPTBot & ChatGPT

    OpenAI's crawlers (including GPTBot and related user agents) collect public pages for ChatGPT browsing and model-related use. If they're disallowed, ChatGPT may be less likely to rely on your site as a source.

  2. ClaudeBot

    Anthropic uses ClaudeBot and Anthropic-AI to retrieve web content for Claude. Your robots.txt is the first gate for whether those agents may fetch your pages at all.

  3. PerplexityBot

    PerplexityBot powers Perplexity's live answers and citations. Blocking it can remove you from those answer panels, sometimes intentionally, sometimes by accident alongside a broad User-agent: * rule.

  4. Each bot follows its own User-agent group when your file has one. A bot without one follows the * group, which we mark as "via *". Consecutive User-agent lines share one group, and the longest matching Allow or Disallow rule wins, as RFC 9309 specifies.

Next step

Other free tools

Go from one snapshot to always-on AI visibility

Right now

This free checker

  • Checks 19 AI and search crawlers in your robots.txt
  • Live fetch, no signup
  • A one-time read, not monitoring

The full picture

Rank Prompt platform

  • Tracks visibility across ChatGPT, Gemini, Perplexity, Claude, Muse, Grok & Google AI Mode (AI Overviews)
  • Runs your full prompt set on a schedule, not one query, once
  • Competitor share-of-voice and rank movement over time
  • Alerts when you gain or lose ground, plus exportable reports
  • AI-optimized content, SEO audits, and citation insights
Start free with Rank Prompt

Robots.txt AI Bot Checker FAQ

Still have questions? Contact our team

What are AI bots and why do they crawl my website?

AI bots (like GPTBot, ClaudeBot, and PerplexityBot) are automated crawlers sent by AI companies to index web content. This data helps train AI models and provide real-time answers. If your robots.txt blocks these bots, AI assistants may not be able to reference or recommend your content when users ask relevant questions.

Which AI bots should I allow in my robots.txt?

At minimum, allow GPTBot (ChatGPT), ClaudeBot (Claude by Anthropic), PerplexityBot, and Google-Extended (Gemini). These cover the major AI platforms where buyers discover brands. Blocking them means your content won't be included in AI-generated recommendations, which is increasingly where purchase decisions start.

How do I unblock AI bots in my robots.txt?

Open your robots.txt file (usually at yourbrand.com/robots.txt) and remove any Disallow rules for AI bot user agents. For example, if you see "User-agent: GPTBot / Disallow: /", change the Disallow line to "Allow: /" or remove the block entirely. If the bot isn't mentioned at all, it follows your User-agent: * group, and with no * group it may crawl everything.

Will allowing AI bots hurt my website performance or security?

No. AI bots behave similarly to search engine crawlers like Googlebot. They read your public pages but don't modify anything. Their crawl rate is typically low and won't impact server performance. Allowing them simply ensures your content can be referenced when AI assistants answer relevant questions.

What do "via *" and "no rule" mean in the results?

"Allowed via *" or "Blocked via *" means your robots.txt has no group for that bot, so it follows your User-agent: * group, as crawlers that follow RFC 9309 do. "Allowed, no rule" means there is neither a group for the bot nor a * group, so nothing stops it from crawling. A bot with its own group ignores the * group entirely, so repeat any shared Disallow lines inside each named group.

Does an allowed bot mean AI crawlers actually visit my site?

No. robots.txt only says who may crawl. To see who did, upload your server’s access log to Rank Prompt: it counts the visits of each AI crawler, checks them against the IP ranges the crawler’s company publishes, and lists the pages they read and the errors they hit. The AI crawler logs guide shows how to export the log.

Go beyond a one-off robots check

Monitor AI visibility over time, compare to competitors, and act on clear recommendations, not just a single static file.

Run my free report View pricing