Integration guide

See which AI crawlers visit your site

An AI assistant can only cite a page its crawler could read. Upload an access log from your server or CDN and see which of 20 AI crawlers visit, which visits are genuine, and which pages fail them.

Start free Explore the feature

What it does

  • Know who reads your site: visits from 20 AI crawlers, GPTBot, ClaudeBot and PerplexityBot among them, each tagged by why it came: training, search index, user request or ads.
  • Tell genuine crawlers from impostors: a visit is verified only when it comes from the crawler company’s own published addresses.
  • Find the pages that answer AI crawlers with a 404 or a server error, the ones that cannot be cited until they are fixed.
  • See which pages each crawler reads, and how its activity moves day by day.
  • Ask Agent Mode or your MCP client what the crawlers did in the last 30 days.

What it cannot do

  • Results cover the logs you upload, so upload a fresh export whenever you want to see the newest days.
  • The Common Log Format has no user agent, so nothing in it can be counted. Switch the log to the combined format.
  • Meta, Amazon, ByteDance, and Cohere publish no address ranges for their crawlers, so those visits are counted but never verified.
  • Pages served from a CDN cache never reach your origin server’s log. Export from the CDN to see those visits too.

Before you start

Plan From Pro

  • Access to your web server’s access logs, or to a Cloudflare zone where you can create a Logpush job.
  • A log from nginx or Apache in the combined format, a Cloudflare Logpush export, or a JSON lines or CSV export that includes the user agent.
  • A file of up to 100 MB, plain or gzipped (.gz). Split a bigger log and upload the parts.
Setup

AI crawler logs setup in 8 steps

  1. Step 1

    Find your access log

    On nginx it is usually /var/log/nginx/access.log; on Apache, /var/log/apache2/access.log on Debian and Ubuntu or /var/log/httpd/access_log on Red Hat. Older days sit next to it, rotated and gzipped; upload those as they are. On managed hosting, look for Access logs or Raw logs in the control panel.

  2. Step 2

    Check the format on nginx

    The default combined format already carries the user agent. If the access_log line names a custom log_format, make sure it keeps $remote_addr, $time_local, "$request", $status and "$http_user_agent" in the combined order, or point it back at combined.

  3. Step 3

    Check the format on Apache

    Log with CustomLog and the combined nickname, not common: the common format has no user agent. Combined is %h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-agent}i".

  4. Step 4

    Or push logs from Cloudflare

    In the Cloudflare dashboard, open your domain’s Logpush page and select Create a Logpush job. Send it to a destination you can download from, such as an R2 bucket, pick the HTTP requests dataset and send at least EdgeStartTimestamp, ClientRequestHost, ClientRequestURI, EdgeResponseStatus, ClientRequestUserAgent and ClientIP. Wait a day, download the files and upload them as they are.

  5. Step 5

    Or export a CSV

    Any tool that exports requests works, from a hosting panel to a log analyzer, as long as the file is JSON lines or a CSV with a header row. Name the columns timestamp or time, path or url, status, user_agent, and ip and host when you have them. Commas, semicolons and tabs all work.

  6. Step 6

    Or connect Cloudflare, Vercel or WordPress

    Skip the uploads: in AI crawlers, Rank Prompt looks at how your site is served and offers the source that fits first. Cloudflare reads every visit with a read-only token, every hour. Vercel gets a log drain on your project that sends each visit as it happens; drains need a Pro or Enterprise team on Vercel, which bills their volume. WordPress sends visits every five minutes through the Rank Prompt Connector plugin.

  7. Step 7

    Upload it in Rank Prompt

    Open Explorer, Search, AI crawlers and select Upload a log. Once the file shows Read under Uploaded logs, the tiles, the chart and both tables fill in.

  8. Step 8

    Read the results

    Start with the error rate and the pages table: a page that answers 404 or 500 to OAI-SearchBot or PerplexityBot cannot be read, let alone cited. Filter the pages by crawler to see what each one reads, and watch the verified share: hits that are not verified may be someone else using the crawler’s name.

Troubleshooting

AI crawler logs: errors and fixes

These are the messages the app shows, word for word, and what fixes each one.

  • This file is over 100 MB. Split it and upload the parts.

    Gzip the log first, which shrinks a text log many times over. If it is still too big, split it by day and upload each part.

  • This file is empty.

    The export came out with no lines, often a log that was rotated a moment before. Pick the rotated file or a longer period.

  • No request in this file could be read. Upload an nginx or Apache access log, Cloudflare Logpush JSON or a CSV with a user agent column.

    The file is in the Common Log Format, has no header row, or lacks a user agent column. Switch the server to the combined format, or export again with the user agent included.

  • The file could not be read. Upload it again.

    The upload broke off or the gzip file is damaged. Upload it again; if it fails twice, unzip it and upload the plain file.

  • The brand's plan no longer includes AI crawlers.

    AI crawlers is on Pro, Agency and Agency Plus. Upgrade the account and upload the log again.

All integrations

FAQ

AI crawler logs integration FAQs

Still have questions? Contact our team

Which AI crawlers does Rank Prompt count?

20 of them, from OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Meta, Amazon, Mistral and more, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Googlebot and Applebot among them. Visits from people are left out.

What does a verified crawler visit mean?

That the visit really came from the company behind the crawler, because its address is one the company publishes. OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Common Crawl, DuckDuckGo, and Mistral publish their addresses. Meta, Amazon, ByteDance, and Cohere do not, so their visits are counted but never verified.

Does Rank Prompt keep my logs or my visitors’ IP addresses?

No. The file is deleted once it is read, and only daily counts per crawler and page are kept, for 13 months. IP addresses are dropped after the check, and visits from people are never stored.

Why are my crawler counts lower than expected?

Pages served from a CDN cache never reach your server, so the origin log misses those visits. Export from the CDN instead, for example with Cloudflare Logpush, which sees every request. A log that covers a few hours also shows only a few hours.

Can visits arrive without uploading a log?

Yes. Connect Cloudflare if it sits in front of your site, Vercel if Vercel serves it, or the Rank Prompt Connector on a WordPress site, and visits arrive on their own. Rank Prompt suggests the one that fits your site.

Which plan do I need?

AI crawlers is on Pro, Agency and Agency Plus, and reading logs costs no credits.

Up and running in a few minutes

Create a free account, follow the steps above and see what the connection adds to your AI visibility.

Start free View pricing