GuidePractitioner6 min readPart of GEO Practitioner

How to Rank in Meta AI: Meta-WebIndexer and What to Allow

How Meta AI finds and cites websites: the Meta-WebIndexer crawler, which Meta crawlers to allow in robots.txt, and how to track what Meta AI says about you.

Dipuck Jones
Written by Founder of MentionX · 13+ years in B2B SaaS growth and SEO
Reviewed
Short answer

To be cited by Meta AI, let Meta's search crawler read your site: Meta says that allowing Meta-WebIndexer helps it cite and link to your content in Meta AI responses ([Meta](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers)). After access, the work is the same as for any AI assistant: clear, accurate pages that answer the everyday questions people ask inside WhatsApp, Instagram, Facebook and Messenger, and a prompt set you run again to see what Meta AI says about you.

Key takeaways
  • Meta runs five documented crawlers. The one tied to Meta AI citations is Meta-WebIndexer.
  • Training and search are separate decisions: Meta-ExternalAgent is the training crawler, Meta-WebIndexer the search one.
  • A blanket rule against Meta's bots also blocks Meta-WebIndexer, which removes a site from Meta AI citations without anyone deciding to.
  • Meta AI questions are casual and local. Consumer and local brands should look first.

Checked 17 November 2026 against Meta's web crawler documentation. "Our view" marks MentionX opinion.

01Where do people use Meta AI?

Meta AI is built into WhatsApp, Instagram, Facebook and Messenger, and is also available as its own app and on the web, with the ability to search the web (Meta). People meet it while chatting or scrolling, so questions tend to be everyday ones: where to eat, what to buy, how to fix something, what a friend just recommended.

That setting decides who should care. A restaurant, a skincare brand or a local service is asked about in Meta AI far more naturally than enterprise software. See Grok, Meta AI and DeepSeek: do smaller engines matter? for how to decide whether to track it at all.

02Which crawlers does Meta use?

Meta documents five (Meta):

Crawler (user agent token)What Meta says it is forrobots.txt
meta-webindexerImproving the quality of Meta AI search resultsNo exception stated
meta-externalagentTraining AI models and improving products by indexing contentUsed in Meta's own robots.txt example
meta-externalfetcherFetching single links a user asked for, including agents completing tasksMay bypass rules, because a user asked for the page
meta-externaladsAdvertising and other business productsNo exception stated
facebookexternalhitLink previews when someone shares your page on Meta appsMay bypass rules for security checks

Two points matter for visibility. First, Meta names Meta-WebIndexer as the crawler behind citations and links in Meta AI. Second, it is a different crawler from the one used for training, so you can allow search and still say no to training.

03What should you allow in robots.txt?

Your goalMeta-WebIndexerMeta-ExternalAgentOthers
Be cited in Meta AI, no objection to trainingAllowAllowAllow
Be cited in Meta AI, opt out of trainingAllowDisallowAllow
Stay out of Meta AI entirelyDisallowDisallowKeep facebookexternalhit so shared links still preview

For the middle row, the file looks like this:

User-agent: meta-webindexer
Allow: /

User-agent: meta-externalagent
Disallow: /

Meta says its crawlers may cache robots.txt for up to 24 hours, so allow a day before checking logs. Which row to choose is a business decision; blocking and allowing AI crawlers covers the pros and cons, and how to configure robots.txt for AI crawlers covers the syntax.

04Why do sites block Meta AI by accident?

Three common ways:

  • A blanket AI bot rule. Lists of "AI crawlers to block" often include every Meta token. Copying one in blocks Meta-WebIndexer along with the training crawler.
  • A firewall or CDN setting. "Block AI bots" switches work on the provider's own list, not yours. Check what yours covers.
  • An old rule. A site that blocked meta-externalagent in 2024 may have used a wildcard that also catches newer tokens.

To check, look in your server or CDN logs for meta-webindexer and the response it received. A 403 or a challenge page means it is being turned away. Meta publishes a way to confirm a request really comes from its network, by listing the IP ranges of its autonomous system (AS32934); it warns that the addresses change often.

05What kind of content suits Meta AI?

Meta's documentation says Meta-WebIndexer analyses content for relevance and accuracy. It does not publish ranking rules, so what follows is method, not a formula.

  • Answer everyday questions directly. "Is it open on Sunday?", "does it work for sensitive skin?", "how long does delivery take?" Put the answer in the first sentence under a clear heading.
  • Keep practical facts in text. Opening hours, prices, locations, sizes, ingredients. Not only in images or a map widget.
  • Make shared links look right. facebookexternalhit builds the preview when someone shares your page in a chat. A correct title, description and image help the person and keep your details consistent.
  • Keep your Meta presence accurate. Your Facebook Page and Instagram profile carry your name, category, hours and address. Make them match your site.
  • Local businesses: follow local GEO. Consistent name, address and phone everywhere is the base.

Our view: do not write separate "Meta AI content". A page that answers a real question well serves every assistant. The Meta specific work is access and accurate profiles.

06How do you track what Meta AI says?

  1. Write 20 to 30 prompts in the casual wording people use in a chat: "good pizza near the station", "best running shoes for flat feet under 100 euros".
  2. Run each several times and record whether you are named, how you are described, and which links appear.
  3. Note whether you were signed in. Meta says answers can be personalised with information people have chosen to share on its products, in some countries.
  4. Repeat monthly and keep the answers, so you can show what changed.

MentionX offers Meta AI as an extra engine you can add, run on the same prompts as your other engines; see pricing. Whatever tool you use, compare Meta AI with itself over time, not with ChatGPT's numbers.

07What if Meta AI gets a fact wrong?

Find the source and fix it there. Check your own site first, then your Facebook Page and Instagram profile, then the outside pages that rank for the question. The method is the same as in how to fix wrong facts about your brand.

08Checklist

  1. Search your robots.txt for "meta" and read every rule.
  2. Decide separately on Meta-WebIndexer (citations) and Meta-ExternalAgent (training).
  3. Ask whoever runs your CDN or firewall what its AI bot setting blocks.
  4. Find meta-webindexer in your logs and confirm it gets a 200.
  5. Make hours, prices and locations plain text.
  6. Match your Facebook Page and Instagram profile to your site.
  7. Run a casual prompt set monthly.

Questions people ask

5 answers

How do I get my website cited by Meta AI?

Allow the Meta-WebIndexer crawler in robots.txt and at your firewall. Meta says allowing it helps Meta cite and link to your content in Meta AI responses.

What is Meta-WebIndexer?

It is the crawler Meta uses to improve Meta AI search results. Its user agent token is meta-webindexer.

Does blocking Meta-ExternalAgent remove me from Meta AI answers?

Meta lists it as a separate crawler, used to train AI models and improve products by indexing content. Meta ties citations in Meta AI to Meta-WebIndexer.

Does Meta AI obey robots.txt?

Meta documents robots.txt rules for its crawlers and says Meta-ExternalFetcher may bypass them because it fetches links a user asked for.

Is Meta AI worth tracking for a B2B company?

Usually later than ChatGPT and Google. People meet Meta AI inside WhatsApp, Instagram, Facebook and Messenger, so consumer and local brands have more reason to track it first.

Sources

  1. Meta for Developers, Meta web crawlersdevelopers.facebook.com
  2. Meta, Introducing the Meta AI appabout.fb.com
Written and reviewed by Dipuck JonesHow we research and testReport an error

Keep learning

All AI engines →