Free tool
AI crawler checker
Check which AI crawlers a domain's robots.txt allows or blocks. No sign-up, no credit card.
No sign-up. Checks only https://<domain>/robots.txt.
Want the full facts on each crawler? Browse the AI crawler directory.
Free tool
Check which AI crawlers a domain's robots.txt allows or blocks. No sign-up, no credit card.
No sign-up. Checks only https://<domain>/robots.txt.
Want the full facts on each crawler? Browse the AI crawler directory.