SemrushBot
Semrush's other crawler.
Quick answer
SemrushBot is the main crawler Semrush uses to discover and collect web data for its SEO and marketing tools, including its backlink and keyword databases. It's one of several Semrush bots, each covering a different product, and site owners can control them separately.
What it does
SemrushBot is the search bot software Semrush sends out to discover and collect new and updated web data, feeding the backlink, keyword, and domain analytics tools Semrush sells to its own customers. Semrush operates multiple bots under different names, including SiteAuditBot and SplitSignalBot for other products, so disallowing SemrushBot specifically only affects this one data-collection crawler rather than every Semrush tool at once. It follows robots.txt rules by default and supports the non-standard Crawl-delay directive, which Semrush documents as capping its own requests at up to 10 seconds apart, giving a site a way to slow the crawl rather than block it outright. SemrushBot's data ultimately feeds reports that Semrush's paying customers, who may include a site's own competitors, can look up about that site.
Facts
Operator
Semrush
Robots token
SemrushBot
User agent string(s)
- Mozilla/5.0 (compatible; SemrushBot/3~bl; +http://www.semrush.com/bot.html)
Purpose
Other
Respects robots.txt
Yes
From Semrush's documentation, checked Oct 2026.
Should you block it?
SemrushBot isn't a search engine or an AI answer crawler, so blocking it has no effect on how a site ranks in Google or shows up in AI assistant answers; the real considerations here are crawl load and who gets to see data about the site. Allowing SemrushBot means any Semrush customer, which can include direct competitors, can look up backlink and keyword data Semrush has collected about the site, which some site owners are comfortable with and others aren't. Because SemrushBot supports Crawl-delay, a site worried mainly about server load from frequent requests can slow the bot down rather than block it entirely, keeping some data flowing while reducing resource use. A site that wants to limit third-party SEO tools from indexing its backlink profile and keyword rankings would block SemrushBot for that reason, understanding that this is a data-visibility choice, not a search-ranking one.
How to block it
Add this to robots.txt to block SemrushBot from crawling any page: User-agent: SemrushBot Disallow: / To slow it down instead of blocking it outright, use the non-standard Crawl-delay directive, which Semrush caps at up to 10 seconds between requests.
Frequently asked questions
Does blocking SemrushBot affect my Google ranking?
No. SemrushBot is an SEO data-collection crawler, not a search engine crawler, so it has no direct effect on Google or other search rankings.
Can I slow SemrushBot down instead of blocking it completely?
Yes. Semrush supports the non-standard Crawl-delay directive, which it says can take intervals of up to 10 seconds between requests to a site.
Is SemrushBot the same as SiteAuditBot or SplitSignalBot?
No. Semrush operates separate bots for different products; blocking SemrushBot specifically doesn't block SiteAuditBot or SplitSignalBot, which need their own rules.
Related
- AI crawler directory — every AI crawler's robots.txt token, purpose and facts in one place.
- Free AI crawler checker — check which AI crawlers a domain's robots.txt actually allows or blocks.
- Glossary — definitions of the AI-visibility terms that come up alongside crawler behavior.
Sources
- SemrushBot is the search bot Semrush uses to discover and collect new and updated web data. Source: https://www.semrush.com/bot/ (checked Oct 2026)
- SemrushBot supports the non-standard Crawl-delay directive, capped at 10 seconds between requests. Source: https://www.semrush.com/bot/ (checked Oct 2026)
MarketHQ tracks brand mentions across communities, news, blogs, social and AI answers, and turns them into gap analysis and action plans.