Reading Server Logs for AI Bot Visits

To see AI bot visits, filter your server or CDN access logs for AI user agent tokens such as GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot and Claude-SearchBot, verify the IPs against each company's published lists, then check which URLs they request and what response codes they get. Analytics tools won't show this, because crawlers don't run tracking scripts. This tutorial walks through the process with example commands.

Key takeaways

• Logs are the only reliable view of crawler access.
• Always pair user agent filters with IP verification.
• Response codes matter more than request counts: 403s and 5xx errors mean blocked access.
• Turn findings into a weekly count and a monthly review.

Where do your logs live?

If you use a CDN, its logs matter most: blocked requests may never reach your origin server.

Step 1: Which user agents should you filter for?

Start with tokens documented by the AI companies:

GPTBot|OAI-SearchBot|ChatGPT-User|OAI-AdsBot|PerplexityBot|Perplexity-User|ClaudeBot|Claude-SearchBot|Claude-User|Googlebot|bingbot

Sources: OpenAI, Perplexity, Anthropic, Google, Bing. Google-Extended won't appear, because it has no separate user agent (Google).

Step 2: How do you count visits by bot?

For a standard combined-format log, a quick shell count:

grep -Eo 'GPTBot|OAI-SearchBot|ChatGPT-User|PerplexityBot|Perplexity-User|ClaudeBot|Claude-SearchBot|Claude-User' access.log \
  | sort | uniq -c | sort -rn

And response codes per bot (in combined log format the status is the ninth field):

grep 'OAI-SearchBot' access.log | awk '{print $9}' | sort | uniq -c

Log formats vary, so check your field positions first.

Step 3: How do you verify the bot is real?

A request claiming to be a bot from an unlisted IP is an impersonator. Separate these in your counts, or your analysis will be wrong.

Step 4: Which questions should the logs answer?

1. Can each search crawler reach the site? Mostly 200 responses, few 403s.
2. Which sections do they crawl? Group URLs by folder.
3. Are important pages crawled? Check your top commercial and guide pages.
4. Are they wasting time? Look for parameter URLs, redirects and 404s.
5. Are user-initiated fetchers appearing? ChatGPT-User, Perplexity-User and Claude-User often indicate real users asking about specific pages.

Worked example

Illustrative data, not real. A week of logs for a mid-size site:

Findings: PerplexityBot is mostly blocked by the CDN (fix firewall); OAI-SearchBot hits 60 broken URLs (fix internal links); ChatGPT-User traffic clusters on the pricing page (users are asking about pricing; make sure it's accurate).

Step 5: How do you build a weekly report?

Automate a small table: requests and response codes per verified bot, top 10 URLs per bot and week-on-week change. Alert on:

• Any search crawler's 403 or 5xx share above a threshold you set.
• A bot's requests dropping to zero.
• A large spike from an unverified "bot".

What about privacy?

Access logs contain IP addresses, which can count as personal data under laws such as the GDPR. Keep AI bot analysis within your existing log retention policy, filter to bot traffic early in your pipeline, and share aggregated tables rather than raw logs. Bot IPs from published ranges belong to companies, but your logs also contain human visitors.

How does this connect to GEO results?

Crawl access is a precondition, not a guarantee. Combine log data with your prompt tracking: if a page is never requested by OAI-SearchBot, it's unlikely to be cited in ChatGPT search; if it's crawled regularly but never cited, the issue is more likely content or competition.

Treat logs as the access layer of your GEO measurement stack: they tell you whether you can be seen, while prompt tracking tells you whether you are.

Common mistakes

• Looking at origin logs when the CDN blocks requests upstream.
• Counting unverified user agents as real bots.
• Confusing robots.txt fetches with page crawls; OpenAI may mark robots.txt requests in the user agent.
• Treating crawl volume as a visibility score.

Checklist

• [ ] Access to CDN and origin logs
• [ ] User agent filter list maintained
• [ ] IP verification for each company
• [ ] Weekly counts and alerts
• [ ] Monthly review against prompt tracking

Where to go next

• List of AI crawlers and user agents
• PerplexityBot and Perplexity-User explained
• User agent (glossary)

Sources

• Apache, Log files
• OpenAI, Overview of OpenAI crawlers
• Perplexity, Perplexity crawlers
• Anthropic, Crawler help article
• Google, Verifying Googlebot