AhrefsBot
Ahrefs's other crawler.
Quick answer
AhrefsBot is the web crawler built by Ahrefs. It gathers the backlink and page data that feeds the Ahrefs SEO platform, and that same crawl also powers Yep, an independent search engine Ahrefs runs. It is a commercial SEO data crawler rather than a search engine or AI assistant crawler.
What it does
AhrefsBot crawls publicly available pages across the web to build the link and content database behind Ahrefs' SEO tools, things like backlink reports, keyword rankings, and site audits that marketers and SEO teams rely on. The same crawl data also powers Yep, a separate search engine that Ahrefs operates. AhrefsBot strictly respects robots.txt, honoring disallow and allow rules as well as crawl-delay directives, so a site that sets a crawl-delay should see AhrefsBot slow down rather than ignore the setting. Ahrefs publishes AhrefsBot's IP ranges through a public API endpoint, and site owners can also confirm traffic by reverse DNS lookup, since the hostname for genuine AhrefsBot requests always resolves to ahrefs.com or ahrefs.net.
Facts
Operator
Ahrefs
Robots token
AhrefsBot
User agent string(s)
- Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)
Purpose
Other
Respects robots.txt
Yes
IP ranges
https://api.ahrefs.com/v3/public/crawler-ip-ranges
From Ahrefs's documentation, checked Oct 2026.
Should you block it?
Because AhrefsBot is a commercial SEO crawler rather than a general search or AI assistant crawler, the main tradeoffs are server load and who gets to analyze a site's content, not visibility in AI answers. Letting AhrefsBot crawl means Ahrefs and any customer running reports on a domain, including competitors, can see its backlink profile and page structure, which some site owners would rather keep less visible. Blocking it has a real cost too: tools many marketing teams use for SEO research, including a site's own team, pull from this crawl, so disallowing AhrefsBot can make a domain harder for anyone to audit with those tools. If server load from crawling is the real concern rather than data visibility, a crawl-delay directive is a lighter option than a full disallow, since AhrefsBot already honors that setting.
How to block it
Add this to robots.txt to block AhrefsBot from crawling any page: User-agent: AhrefsBot Disallow: /path-to-disallow/ AhrefsBot respects crawl-delay directives too, so a site that only wants to slow the crawl rather than block it outright can set that instead of a full disallow.
Frequently asked questions
Does AhrefsBot affect a site's AI visibility?
No. AhrefsBot feeds the Ahrefs SEO platform and the Yep search engine. It has no connection to ChatGPT, Claude, Gemini, Perplexity, Grok, or DeepSeek.
Can AhrefsBot's traffic be verified?
Yes. Ahrefs publishes AhrefsBot's IP ranges through a public API endpoint, and genuine traffic also resolves via reverse DNS to ahrefs.com or ahrefs.net.
Does AhrefsBot slow down if I set a crawl-delay?
Yes. Ahrefs says AhrefsBot strictly respects crawl-delay directives in robots.txt, along with disallow and allow rules.
Related
- AI crawler directory — every AI crawler's robots.txt token, purpose and facts in one place.
- Free AI crawler checker — check which AI crawlers a domain's robots.txt actually allows or blocks.
- Glossary — definitions of the AI-visibility terms that come up alongside crawler behavior.
Sources
- AhrefsBot powers the database for both Ahrefs and the independent search engine Yep. Source: https://ahrefs.com/robot/ (checked Oct 2026)
- Both AhrefsBot and AhrefsSiteAudit strictly respect robots.txt disallow/allow rules and crawl-delay directives. Source: https://ahrefs.com/robot/ (checked Oct 2026)
MarketHQ tracks brand mentions across communities, news, blogs, social and AI answers, and turns them into gap analysis and action plans.