Glossary · Beginner

What Is robots.txt? Definition and Example

Dipuck JonesLast reviewed 2026-10-30

robots.txt is a plain-text file at the root of a website (for example `example.com/robots.txt`) that tells crawlers which parts of the site they may request. It follows the Robots Exclusion Protocol, standardised as RFC 9309 (RFC Editor). It controls crawling, not indexing: to keep a page out of Google, use noindex instead (Google).

What does it look like?

User-agent: *
Disallow: /admin/

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Sitemap: https://example.com/sitemap.xml

How do crawlers read it?

According to Google's documentation (robots.txt spec):

  • A crawler follows the most specific user-agent group that matches it and ignores the others.
  • Within a group, the longest matching path wins. When rules conflict, the least restrictive one applies.
  • Google generally caches the file for up to 24 hours.

Why does it matter for AI visibility?

AI companies publish separate user agents for training and for search, such as OpenAI's GPTBot and OAI-SearchBot (OpenAI). A careless blanket rule can block the very agents that put you in AI answers. Review the file whenever your AI policy changes.

What are the common mistakes?

  • A Disallow: / under User-agent: * left over from staging.
  • Blocking the CSS or JS needed to render pages.
  • Assuming a block removes pages that are already indexed.

How do you test your file?

Open yourdomain.com/robots.txt in a browser to see what's live. Use the robots.txt report in Google Search Console to check how Google fetched it. For each AI agent you care about, read through the groups and confirm which one applies, remembering that a named group overrides User-agent: * completely for that agent.

Questions

Does robots.txt stop a page from being indexed?

Not reliably. A blocked URL can still be indexed without its content if other pages link to it. Use noindex to keep a page out of Google.

How quickly do changes apply?

Google generally caches robots.txt for up to 24 hours.

Can I set rules for AI crawlers?

Yes. Add a group for each user agent, such as GPTBot, OAI-SearchBot, PerplexityBot or Google-Extended.