category
Technical
Technical in MentionX Learn.
Pages
- Guide · PractitionerWhat Is llms.txt? Complete GuideWhat llms.txt is, what goes in it, which AI systems read it, and whether it's worth adding, with MentionX citation data on .md and llms.txt URLs.
- Tutorial · PractitionerHow to Create an llms.txt File (With Example)Step-by-step guide to creating an llms.txt file: choose pages, write the summary, format links, publish, test, and maintain it, with MentionX data.
- Guide · Practitionerllms.txt vs robots.txtllms.txt vs robots.txt: what each file does, how they differ, which setup fits your site, and how to keep them consistent, with MentionX data.
- Guide · PractitionerGPTBot User Agent: Allow or Block?GPTBot explained: what OpenAI's training crawler does, how it differs from OAI-SearchBot, and how to decide whether to allow or block it.
- Tutorial · PractitionerHow to Configure robots.txt for AI CrawlersStep-by-step robots.txt setup for AI crawlers: separate search from training, avoid the precedence trap, test, verify in logs and maintain.
- Guide · PractitionerSchema Markup for AI SearchHow schema markup helps AI search: which schema.org types matter, Organization and sameAs, the match rule, validation and priorities, with MentionX data.
- Guide · PractitionerList of AI Crawlers and User Agents (2026)List of AI crawlers by company and purpose (training, AI search, user fetch), with robots.txt tokens, verification methods and MentionX citation data.
- Guide · PractitionerOAI-SearchBot vs GPTBot vs ChatGPT-UserOAI-SearchBot, GPTBot and ChatGPT-User do different jobs. What each OpenAI agent does, how robots.txt applies, and which settings fit your goals.
- Guide · PractitionerPerplexityBot and Perplexity-User ExplainedPerplexityBot indexes sites for Perplexity's cited answers; Perplexity-User fetches pages for user questions. How to allow, verify and troubleshoot both.
- Guide · PractitionerGoogle-Extended: What It ControlsGoogle-Extended lets you manage Gemini training and grounding use of your content without affecting Google Search or AI Overviews. What it controls.
- Guide · PractitionerClaudeBot and Claude-SearchBotAnthropic runs ClaudeBot for training, Claude-SearchBot for search and Claude-User for user requests. How to manage each in robots.txt.
- Tutorial · PractitionerReading Server Logs for AI Bot VisitsHow to find AI crawler visits in your server or CDN logs, verify them, and turn them into useful GEO signals, with commands and a worked example.