Meta-ExternalFetcher
Meta's user-triggered fetch crawler.
Quick answer
Meta-ExternalFetcher retrieves a specific page when a user, or an AI agent acting for a user, asks for it directly, such as an assistant navigating a site to complete a task, rather than crawling proactively like a standing search bot.
What it does
Meta-ExternalFetcher fetches individual links at the request of a specific user action, supporting features like agentic AI capabilities where an AI assistant needs to navigate a website on someone's behalf to complete a task. Unlike a crawler that proactively visits many pages on its own schedule, this bot activates only when a particular fetch was requested, similar in spirit to Amazon's Amzn-User or Mistral's MistralAI-User. Meta states that because these fetches were requested by the user, Meta-ExternalFetcher may bypass robots.txt rather than always honoring a site's disallow directives. There is no published IP range list for this bot, so confirming traffic is genuine relies on Meta's own documentation rather than an IP check.
Facts
Operator
Meta
Robots token
meta-externalfetcher
User agent string(s)
- meta-externalfetcher/1.1 (+/documentation/sharing/webmasters/web-crawlers)
- meta-externalfetcher/1.1
Purpose
User-triggered fetch
Respects robots.txt
No
From Meta's documentation, checked Oct 2026.
Should you block it?
Because Meta-ExternalFetcher only fetches a page when a person or an AI agent acting for them specifically requested it, often to complete a task like checking information or navigating a site, blocking it mainly affects whether those on-demand, agentic AI requests can succeed on a site. A business that wants AI agents to be able to interact with its public pages on a user's behalf, for example to look up information or complete a simple task, would generally want this fetcher allowed. The complication is that Meta says this bot may bypass robots.txt specifically because the fetch was user requested, so a disallow rule here is not guaranteed to be fully honored the way it would be with a standard crawler. Sites mainly worried about AI training reuse should note that user-triggered fetchers like this one are typically about completing a specific task, not about building a training dataset.
How to block it
robots.txt can include this rule for Meta-ExternalFetcher: User-agent: meta-externalfetcher Disallow: / Meta says this bot may still bypass robots.txt in some cases because its fetches are triggered by a specific user request.
Frequently asked questions
What triggers a Meta-ExternalFetcher request?
A specific user action, such as an AI agent navigating a website on someone's behalf to complete a task.
Will a robots.txt disallow definitely stop Meta-ExternalFetcher?
Not always. Meta says it may bypass robots.txt because its fetches were requested by the user.
Is Meta-ExternalFetcher used for AI training?
No. Meta describes it as supporting agentic AI capabilities and user-requested fetches, not training data collection.
Related
- AI crawler directory — every AI crawler's robots.txt token, purpose and facts in one place.
- Free AI crawler checker — check which AI crawlers a domain's robots.txt actually allows or blocks.
- Glossary — definitions of the AI-visibility terms that come up alongside crawler behavior.
Sources
- Meta-ExternalFetcher fetches individual links at a user's request to support agentic AI capabilities, including helping AI navigate websites to complete tasks. Source: https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers (checked Oct 2026)
- Meta-ExternalFetcher may bypass robots.txt because it performs fetches that were requested by the user. Source: https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers (checked Oct 2026)
MarketHQ tracks brand mentions across communities, news, blogs, social and AI answers, and turns them into gap analysis and action plans.