# Meta-ExternalFetcher

## Quick answer

Meta-ExternalFetcher retrieves a specific page when a user, or an AI agent acting for a user, asks for it directly, such as an assistant navigating a site to complete a task, rather than crawling proactively like a standing search bot.

## What it does

Meta-ExternalFetcher fetches individual links at the request of a specific user action, supporting features like agentic AI capabilities where an AI assistant needs to navigate a website on someone's behalf to complete a task. Unlike a crawler that proactively visits many pages on its own schedule, this bot activates only when a particular fetch was requested, similar in spirit to Amazon's Amzn-User or Mistral's MistralAI-User. Meta states that because these fetches were requested by the user, Meta-ExternalFetcher may bypass robots.txt rather than always honoring a site's disallow directives. There is no published IP range list for this bot, so confirming traffic is genuine relies on Meta's own documentation rather than an IP check.

## Facts

- Operator: Meta
- Robots token: meta-externalfetcher
- User agent string(s): meta-externalfetcher/1.1 (+/documentation/sharing/webmasters/web-crawlers); meta-externalfetcher/1.1
- Purpose: user-triggered-fetch
- Respects robots.txt: no

## Should you block it?

Because Meta-ExternalFetcher only fetches a page when a person or an AI agent acting for them specifically requested it, often to complete a task like checking information or navigating a site, blocking it mainly affects whether those on-demand, agentic AI requests can succeed on a site. A business that wants AI agents to be able to interact with its public pages on a user's behalf, for example to look up information or complete a simple task, would generally want this fetcher allowed. The complication is that Meta says this bot may bypass robots.txt specifically because the fetch was user requested, so a disallow rule here is not guaranteed to be fully honored the way it would be with a standard crawler. Sites mainly worried about AI training reuse should note that user-triggered fetchers like this one are typically about completing a specific task, not about building a training dataset.

## How to block it

robots.txt can include this rule for Meta-ExternalFetcher:

User-agent: meta-externalfetcher
Disallow: /

Meta says this bot may still bypass robots.txt in some cases because its fetches are triggered by a specific user request.

## FAQ

### What triggers a Meta-ExternalFetcher request?

A specific user action, such as an AI agent navigating a website on someone's behalf to complete a task.

### Will a robots.txt disallow definitely stop Meta-ExternalFetcher?

Not always. Meta says it may bypass robots.txt because its fetches were requested by the user.

### Is Meta-ExternalFetcher used for AI training?

No. Meta describes it as supporting agentic AI capabilities and user-requested fetches, not training data collection.

## Sources

- Meta-ExternalFetcher fetches individual links at a user's request to support agentic AI capabilities, including helping AI navigate websites to complete tasks. Source: https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers
- Meta-ExternalFetcher may bypass robots.txt because it performs fetches that were requested by the user. Source: https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers