How to Configure robots.txt for AI Crawlers

To configure robots.txt for AI crawlers, decide separately whether you want to appear in AI search and whether you allow AI training, then write one robots.txt group per documented AI user agent that reflects those choices. Deploy the file at your domain root, test it, and confirm in server logs that each agent behaves as expected. Budget about an hour, plus a week of log monitoring.

What you need: edit access to robots.txt (or your CMS setting for it), access to server or CDN logs, and a written decision from whoever owns content licensing.

On this page

• Why does this matter for AI visibility?
• Step 1: What does your robots.txt allow today?
• Step 2: Which policy do you want?
• Step 3: How do you write the groups?
• Step 4: What's the precedence trap?
• Step 5: How do you deploy and test?
• How do you read the logs?
• Step 6: How do you keep it maintained?
• Worked example: Ledgerline's robots.txt review
• Common mistakes
• Common questions
• Sources

Updated 31 October 2026. Anything marked "Our view" is MentionX's opinion, not vendor documentation. The worked example uses a fictional company. Always check each vendor's live crawler page before deploying.

Why does this matter for AI visibility?

Answer engines with search switched on fetch and quote live pages. If their search crawler can't reach your pages, they can still talk about you from what they learned in training, but they can't cite your current pricing, docs or comparisons.

From MentionX data. We ran read-only aggregate queries over completed MentionX runs from 22 August to 29 October 2026 (UTC): 2,154 runs across 18 anonymised projects, six engines called through their APIs with web search or grounding on where available. They returned 9,750 citations from 2,152 different domains. Excluding Gemini (whose grounding mostly returns redirect links, so we can't see the final URL), we sorted the 5,917 remaining cited URLs with simple URL patterns:

Limits. A convenience sample of MentionX projects, mostly B2B software. URL patterns are crude: a blog post at /insights/ counts as "everything else". The point is that engines cite many kinds of pages, so a blanket block on a section such as /blog/ or /docs/ can remove pages engines would otherwise quote.

Our view: treat any Disallow that affects public marketing, docs or pricing pages as a visibility decision, not just a technical one, and get sign-off for it.

Step 1: What does your robots.txt allow today?

1. Open https://yourdomain.com/robots.txt.
2. Note every User-agent group and its rules.
3. Look for a blanket block: User-agent: * followed by Disallow: /. That blocks every compliant crawler without its own group.
4. Check your CDN or WAF (Cloudflare, Akamai, Fastly, etc.) for bot rules that block AI agents regardless of robots.txt.
5. Check for subdomains (docs, help, blog). Each has its own robots.txt.

Step 2: Which policy do you want?

Fill in this table with your stakeholders:

User-initiated agents (ChatGPT-User, Claude-User, Perplexity-User) fetch pages when a person asks. Read each vendor's page for how they treat robots.txt.

Step 3: How do you write the groups?

A template for the most common policy, visible in AI search with no AI training:

# --- Search engines ---
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# --- AI search (allow) ---
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# --- AI training (block) ---
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# --- Everyone else ---
User-agent: *
Disallow: /admin/
Disallow: /cart/

Sitemap: https://yourdomain.com/sitemap.xml

To allow training too, change the three Disallow: / lines under AI training to Allow: /.

Step 4: What's the precedence trap?

Under the Robots Exclusion Protocol, a crawler uses the group that matches its user agent most specifically; the * group applies only to crawlers that have no group of their own (RFC 9309). So in the template above, OAI-SearchBot can crawl /admin/ because its own group doesn't disallow it.

Fix: repeat shared disallows in each named group.

User-agent: OAI-SearchBot
Allow: /
Disallow: /admin/
Disallow: /cart/

Also remember that within a group, the most specific (longest) matching path rule wins, and when rules conflict Google uses the least restrictive one (Google: how Google interprets robots.txt).

Step 5: How do you deploy and test?

1. Deploy to the root of each host. File name must be lowercase robots.txt, served as text/plain, returning 200.
2. Purge the CDN cache for /robots.txt.
3. Fetch it: curl -A "GPTBot" https://yourdomain.com/robots.txt and confirm you see the new file.
4. Test rules with Search Console's robots.txt report for Googlebot, and a robots.txt parser library for other agents (for example Python's urllib.robotparser).
5. Monitor logs for a week. Filter by agent; confirm blocked agents stop requesting disallowed paths and allowed agents get 200s.
6. Verify IPs against each vendor's published ranges to discount spoofed bots (OpenAI, Perplexity).

Google says it generally caches robots.txt for up to 24 hours (Google: how Google interprets robots.txt). Other crawlers vary.

How do you read the logs?

Logs are the only way to confirm what crawlers actually did. A simple weekly check is enough for most sites.

1. Filter by user agent. Pull every request whose user agent contains one of the agents in your policy table. Count requests per agent per day.
2. Split by status code. Allowed agents should mostly get 200s (and some 301s or 304s). Lots of 403s or 429s for an agent you allowed usually means a CDN or firewall rule is blocking it.
3. Check paths. Blocked agents should only request /robots.txt and nothing you disallowed. Allowed agents should be reaching the pages you care about, such as pricing, docs and comparisons.
4. Verify identity. For a sample of requests, check the IP against the vendor's published ranges (OpenAI, Perplexity) or, for Google, with a reverse DNS lookup (Google). Requests that fail are spoofed and can be blocked at the firewall without touching robots.txt.

On Apache, the "combined" log format includes the user agent by default (Apache); most CDNs offer the same field in their log exports.

Our view: save the query you use and the counts it gives in the same repo as robots.txt. When someone asks "are we blocking ChatGPT?", you can answer with numbers in minutes.

Step 6: How do you keep it maintained?

• Review the vendor crawler pages quarterly; new agents appear.
• Keep robots.txt in version control with a comment block explaining each decision and its date.
• Add a pre-deploy test that fails if Disallow: / appears under * on production.
• Re-run your AI visibility scan after any change so you can see the effect.

Worked example: Ledgerline's robots.txt review

Ledgerline is a fictional accounts-payable tool for mid-sized companies. Its marketing team notices that ChatGPT answers describe an old pricing plan and never cite the pricing page.

What they find in Step 1:

• The main robots.txt has a sensible * group but no named AI groups.
• The docs subdomain has its own robots.txt from a staging copy: User-agent: * and Disallow: /.
• The CDN has a "block AI scrapers" toggle switched on, added by IT a year earlier.

The decision (Step 2): legal agrees to allow AI search crawlers everywhere public and to block training crawlers on the docs, which contain licensed content from a partner. Training crawlers stay allowed on the marketing site.

The fix:

1. Replace the docs robots.txt with named groups: search agents allowed, GPTBot, ClaudeBot and Google-Extended disallowed, and /internal/ disallowed in every group.
2. Turn off the CDN toggle and replace it with a rule that rate-limits unverified bots instead of blocking named ones.
3. Purge /robots.txt from the CDN cache on both hosts and fetch both files to confirm.
4. Watch logs for a week. OAI-SearchBot and PerplexityBot start getting 200s on the pricing page. GPTBot requests to the docs stop.

A month later, the team re-runs its prompt set. The pricing page now appears among the cited sources for some pricing prompts, and the stale plan shows up less often. Illustrative: the outcome shows the method, not a measured result.

Common mistakes

• Staging robots.txt shipped to production.
• Blocking GPTBot and believing you've left ChatGPT search.
• Blocking Google-Extended to leave AI Overviews (Googlebot controls that).
• CDN "block AI bots" toggles that override a carefully written robots.txt.
• Forgetting subdomains. The docs, help centre and blog on separate hosts each need their own file, and they're often where the most-cited pages live.
• Returning an error for robots.txt. Google treats some server errors on robots.txt as a reason to pause crawling the site (Google). Make sure the file returns a clean 200.
• Blocking Bingbot. Copilot answers are grounded in Bing, so blocking Bing's crawler can cut you out of Copilot as well as Bing search (Microsoft, Bing).
• Writing wildcard rules without testing them. A pattern such as Disallow: /*? can block every URL with a query string, including tracked campaign links and filtered product pages. Test every pattern against a list of real URLs before you deploy.

• Using robots.txt to hide confidential pages; use authentication or noindex for search (Google).

Common questions

Do AI crawlers respect robots.txt? The major vendors say their documented crawlers do (OpenAI, Anthropic, Perplexity). Robots.txt is still advisory, so unknown or spoofed bots may ignore it. Verify IPs and use your firewall for those.

Do I need a separate group for every AI agent? Only for agents you want to treat differently from *. But because a named group replaces *, every named group you add must repeat the disallows that should still apply (RFC 9309).

What about user-initiated agents like ChatGPT-User? They fetch a page because a person asked. Vendors describe how robots.txt applies to them on their crawler pages; read those before blocking, because blocking them can stop people from asking an assistant about your page.

Is there a standard "AI" user agent I can block in one line? No. Each vendor publishes its own agents, and some have several with different jobs. That's why the policy table in Step 2 lists them one by one, and why it needs a quarterly review.

How quickly do changes take effect? Google says it generally caches robots.txt for up to 24 hours (Google). Other vendors differ. Plan for a few days before logs settle.

Does robots.txt remove pages already in AI answers? Not immediately. It stops future crawling by compliant bots. To keep a page out of Google Search, including its AI features, use noindex instead and leave the page crawlable so Google can see the tag (Google).

Sources

• IETF, RFC 9309
• Google Search Central, robots.txt intro; how Google interprets robots.txt; common crawlers; AI features; noindex
• OpenAI, crawlers; Anthropic, crawler FAQ; Perplexity, crawlers; Bing, crawlers; Perplexity, IP ranges
• Google, Verify Google requests; Microsoft, How web search works in Copilot; Apache, Log files