# Meta-WebIndexer

## Quick answer

Meta-WebIndexer is Meta's crawler for search-style indexing that feeds Meta AI's answers, navigating the web to improve the quality of what Meta AI can find and reference. Allowing it helps Meta AI cite and link back to a site's content in its responses, separate from Meta-ExternalAgent's training-focused crawl.

## What it does

Meta-WebIndexer navigates the web specifically to improve the quality of Meta AI's search results, which Meta describes as distinct from Meta-ExternalAgent's broader training and product-indexing purpose. Meta tells site owners directly that allowing Meta-WebIndexer in robots.txt helps Meta cite and link to that content in Meta AI's responses, framing this bot as the one tied to citation and attribution rather than to model training. It follows robots.txt rules, so a site can allow most of its content while disallowing specific sections, like a private directory, that it doesn't want surfaced in Meta AI's answers. Because this bot's role is about appearing in Meta AI's live responses, it behaves more like a search-engine crawler feeding AI answers than a training-data collector.

## Facts

- Operator: Meta
- Robots token: meta-webindexer
- User agent string(s): meta-webindexer/1.1 (+/documentation/sharing/webmasters/web-crawlers); meta-webindexer/1.1
- Purpose: search-index
- Respects robots.txt: yes

## Should you block it?

Disallowing meta-webindexer stops Meta AI from citing and linking to a site in its answers, since Meta directly ties allowing this crawler to that outcome, which is the opposite tradeoff from blocking a training-only bot; a brand wanting to be mentioned when someone asks Meta AI about its category should generally allow this one. Meta also notes that blocking it doesn't retroactively remove content Meta AI may have already indexed, so disallowing meta-webindexer mainly affects future citations rather than erasing a brand's existing footprint inside Meta AI. Because this bot is separate from Meta-ExternalAgent, a site can choose to allow meta-webindexer for AI-answer visibility while still disallowing Meta-ExternalAgent if its concern is specifically about training data rather than about being cited in Meta AI's live responses.

## How to block it

Add this to robots.txt to block meta-webindexer from indexing content for Meta AI's answers, while keeping other paths open if desired:

User-agent: meta-webindexer
Disallow: /private/

Use 'Disallow: /' instead to block it from the entire site.

## FAQ

### What's the benefit of allowing meta-webindexer?

Meta says allowing Meta-WebIndexer in robots.txt helps it cite and link to a site's content in Meta AI's responses.

### Does blocking meta-webindexer remove content already indexed by Meta AI?

No. Blocking it does not retroactively remove content already indexed; it mainly affects whether Meta AI indexes and cites the site going forward.

### Is meta-webindexer the same as Meta-ExternalAgent?

No. Meta-WebIndexer is documented separately for search-style indexing that feeds Meta AI's answers, apart from Meta-ExternalAgent's training and product-indexing purpose.

## Sources

- Meta-WebIndexer navigates the web to improve Meta AI search result quality, and allowing it helps Meta AI cite and link to the site. Source: https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers