robots.txt
Quick answer
robots.txt is a plain-text file at a site's root that tells automated crawlers, including search engine bots and the crawlers AI companies use, which parts of the site they may fetch. It cannot stop a crawler that ignores it, but most AI companies that document a crawler publish a name a site can explicitly allow or block in this file.
Why it matters
If a brand's robots.txt accidentally blocks an AI crawler, that engine may never read the brand's own pages directly and ends up relying only on what third-party sites say about the brand instead. That gap is easy to miss because the brand's site still works fine for human visitors and still ranks normally in classic search, so nothing looks broken until someone checks which AI engines actually cite the brand's own pages versus only mentions of it elsewhere. Checking robots.txt is one of the first, cheapest things to rule out when a brand seems invisible to one specific AI engine while visible to others.
How to measure it
Read the file directly and look for each AI crawler's documented user-agent name, noting whether it is allowed or disallowed on the pages that matter most, such as the pricing page, the comparison pages, and the docs. Then compare that against whether the specific engine behind that crawler actually cites the brand's own pages as a source, rather than only surfacing third-party coverage of the brand. A mismatch, blocked in the file but still somehow cited, or allowed but never cited, both point to something worth investigating further.
Example
A brand migrates to a new site platform, and the platform's default robots.txt allows a short list of well-known search bots while blocking everything else by default, including a newer AI crawler that wasn't on the list at the time. The brand's own pages stop appearing as a cited source in that AI engine's answers, even though community mentions of the brand keep showing up, since those live on other sites that robots.txt has no control over.
Frequently asked questions
Does robots.txt stop an AI company from training on a site's content?
It can signal a preference, but only if that AI company's crawler actually respects the specific rule written for it. robots.txt is a documented request, not an enforced technical block.
How is robots.txt different from llms.txt?
robots.txt controls whether a crawler may fetch a page at all. llms.txt is a separate, much less widely supported file some sites add to suggest which pages an AI assistant should treat as the canonical summary.
Can blocking an AI crawler in robots.txt ever help a brand?
Rarely for a product that wants to be found and recommended by AI answers. Blocking is more often used by sites that don't want their content reused for model training, which is a different goal from being visible in AI answers.
Related terms
Related
MarketHQ tracks brand mentions across communities, news, blogs, social and AI answers, and turns them into gap analysis and action plans.