Glossary · Beginner
What Is an User Agent? Definition and Example
A user agent is the identifier a browser, app or bot sends with every web request to say what it is, through the HTTP User-Agent header. For GEO, user agents matter because AI companies name their crawlers, such as GPTBot, OAI-SearchBot, PerplexityBot and ClaudeBot, and you use those names in robots.txt to decide what each can access.
What does a user agent string look like?
OpenAI documents this example for GPTBot (the version number may change): Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot (OpenAI). The full string includes compatibility text; the part that identifies the bot is the product token, here GPTBot.
String vs robots.txt token
| Term | Where it appears | Example |
|---|---|---|
| User agent string | HTTP request header, your logs | The full Mozilla/5.0 ... GPTBot/1.4 line |
| Product token | robots.txt User-agent: line | GPTBot |
Some tokens have no matching request string. Google-Extended, for example, has no separate HTTP user agent; crawling uses existing Google user agents and the token is used only as a control in robots.txt (Google).
Example
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /This asks OpenAI not to use the site for training while allowing it to appear in ChatGPT search. OpenAI says each setting is independent (OpenAI).
Can you trust the user agent you see?
Not on its own. The header is self-declared, so scrapers can impersonate well-known bots. Google documents verifying Googlebot through reverse DNS lookup (Google), and OpenAI and Perplexity publish IP ranges for their bots. Combine the user agent with IP verification before allowing or blocking traffic at the firewall.
Why it matters for GEO
If your firewall or robots.txt blocks a search crawler by mistake, your pages may not be eligible to appear in that engine's answers. A quick audit of which user agents reach your site, and which you allow, is one of the cheapest GEO checks you can do.
Related
Questions
Where do I see user agents?
In your server or CDN access logs. Each request includes a User-Agent header naming the client.
Can a user agent be faked?
Yes. Anyone can send any string, which is why Google and others publish ways to verify their crawlers.
What's the difference between a user agent string and a robots.txt token?
The string is the full header sent with a request; the token is the short name you write in robots.txt, such as GPTBot.