Diffbot-User
Diffbot's user-triggered fetch crawler.
Quick answer
Diffbot-User represents a human user browsing a specific URL through Diffbot's software, acting in direct response to that person's own input. It's a one-off fetch tied to a real request, distinct from Diffbot's separate, proactive crawling that discovers and indexes pages on its own.
What it does
Diffbot-User is the identity Diffbot's systems use when a request is made on behalf of a human user who is browsing a particular URL through Diffbot software, in response to something that person typed or clicked. That makes it different in kind from Diffbot's main crawling activity, which proactively discovers and fetches pages across the web to build Diffbot's own structured data products, without a specific person asking about a specific page in the moment. Diffbot documents this distinction in its robots.txt FAQ so site owners can tell the two kinds of traffic apart and, if they choose, treat them differently. Diffbot doesn't publish a fixed IP range for Diffbot-User in this data, since this traffic is generated on demand rather than on a predictable crawl schedule.
Facts
Operator
Diffbot
Robots token
Diffbot-User
User agent string(s)
- (Mozilla/AppleWebKit style string identifying requests made on behalf of a human user via Diffbot software)
Purpose
User-triggered fetch
Respects robots.txt
Not stated by the operator
From Diffbot's documentation, checked Oct 2026.
Should you block it?
Diffbot-User fetches are tied to a real person's action, so blocking this specific bot mainly affects whether that user's request through Diffbot's software successfully reaches the page, not whether the site shows up in any index or AI answer. A site that wants to support tools built on Diffbot, which extracts structured data from pages for other applications, would leave Diffbot-User allowed so those user-driven requests go through normally. Because this is separate from Diffbot's own proactive crawling, blocking Diffbot-User doesn't stop Diffbot from building its broader data products from the site; a site that specifically wants to opt out of that separate crawling activity needs to address Diffbot's other crawler, not Diffbot-User. The practical question is narrower than most AI-bot decisions: does the site want to support on-demand lookups initiated by someone else's tool.
How to block it
Add this to robots.txt to block Diffbot-User from fetching pages on a user's behalf: User-agent: Diffbot-User Disallow: / This only affects user-triggered fetches through Diffbot's software, not Diffbot's separate, proactive crawling.
Frequently asked questions
Is Diffbot-User the same as Diffbot's main crawler?
No. Diffbot documents Diffbot-User specifically as requests made on behalf of a human user browsing a URL through its software, separate from its proactive crawling.
Does Diffbot-User follow robots.txt?
That isn't specified in Diffbot's published data here, so check Diffbot's own robots.txt FAQ directly for its current behavior before relying on a rule.
Why would Diffbot-User request a page on my site?
Because a person using a tool built on Diffbot's software browsed to a URL on that site, triggering a one-off fetch in response to their input.
Related
- AI crawler directory — every AI crawler's robots.txt token, purpose and facts in one place.
- Free AI crawler checker — check which AI crawlers a domain's robots.txt actually allows or blocks.
- Glossary — definitions of the AI-visibility terms that come up alongside crawler behavior.
Sources
- Diffbot-User is used by requests originating on behalf of a human user browsing a URL using Diffbot software, in response to their input. Source: https://www.diffbot.com/docs/crawl/faq/robots-txt (checked Oct 2026)
MarketHQ tracks brand mentions across communities, news, blogs, social and AI answers, and turns them into gap analysis and action plans.