Pay-Per-Crawl
Quick answer
Pay-per-crawl is an emerging access model where a publisher charges an AI company's crawler a fee each time it fetches a page, rather than simply allowing or blocking it through robots.txt. It sits between fully open access and a full block, letting a publisher get paid for content an AI product reads and potentially cites.
Why it matters
If pay-per-crawl becomes widely adopted, which sources an AI engine actually reads could start to depend on which publishers an AI company has paid to crawl, not only on a page's accuracy or relevance. For a brand, that means some of the third-party coverage it has worked hard to earn could quietly stop being readable by a given AI engine, for reasons that have nothing to do with the coverage itself. That is a new kind of gap to watch for, separate from whether the coverage was good or whether the brand's own site is crawlable.
How to measure it
This is early enough that there's no settled, widely used way to measure its effect. A brand can mostly watch whether specific coverage it cares about sits behind a pay-per-crawl arrangement, and check whether an AI engine's answer about the brand changes depending on whether that particular piece of coverage was crawled or not. Keeping a short list of outlets that matter most to a brand, and checking their crawl status periodically, is a reasonable starting habit.
Example
A trade publication where a brand has long-standing, favorable coverage adopts a pay-per-crawl policy. One AI company that doesn't pay for crawls stops reading new articles from that publication, so its answers about the brand gradually fall out of date, drawing on older, already-crawled articles instead, while another AI company that does pay keeps getting the publication's current coverage. The brand's coverage didn't change at all; which AI engines could still read it did.
Frequently asked questions
Is pay-per-crawl the same as blocking a crawler in robots.txt?
No. Blocking denies access entirely. Pay-per-crawl is a paid middle ground where access is allowed in exchange for a fee per fetch, rather than an outright yes or no.
Does pay-per-crawl affect a brand's own site, or only publishers writing about it?
It can apply to either, though the more immediate effect on brand visibility is on the third-party publications an AI engine reads for coverage of that brand, not the brand's own site.
Is pay-per-crawl widely adopted yet?
Not universally. It's an emerging model some publishers and AI companies are experimenting with rather than a standard every site or crawler already follows.
Can a brand do anything if a key publication adopts pay-per-crawl?
Not directly, since the arrangement is between the publisher and the AI company. A brand can note which engines still read that coverage and adjust where it focuses its own outreach.
Related terms
Related
MarketHQ tracks brand mentions across communities, news, blogs, social and AI answers, and turns them into gap analysis and action plans.