Robots.txt & AI crawler checker

This tool does two jobs. The checker fetches your site's /robots.txt and tells you, in plain language, which of the major AI crawlers it currently allows or blocks - including OpenAI's GPTBot, Google-Extended, PerplexityBot, ClaudeBot, Common Crawl's CCBot, and more. The generator lets you tick the crawlers you want to block, add your sitemap, and copy a ready-to-deploy robots.txt. It's the fastest way to take control of how AI systems use your content.

robots.txt is a plain-text file at your domain root that follows the Robots Exclusion Protocol. It's organized into groups, each beginning with one or more User-agent lines (naming a crawler) followed by Allow and Disallow rules. Disallow: / blocks the whole site for that crawler; an empty Disallow: allows everything. A User-agent: * group is the fallback for any crawler not named specifically. You can also declare your sitemap with a Sitemap: line. For the full spec, see Google's robots.txt documentation.

One critical caveat: robots.txt is voluntary. Reputable crawlers honor it, but it isn't enforced and bad actors can ignore it. To truly block access you need server-side controls. Think of robots.txt as a clear, respected request - not a lock.

This is a strategy decision, not a technical one. There are two distinct kinds of AI crawler, and they deserve different treatment:

• Training crawlers (GPTBot, CCBot, ClaudeBot, Google-Extended, Bytespider) collect content to train models. Blocking them keeps your content out of training data - useful if you sell content or want to protect IP.
• Search/answer crawlers (OAI-SearchBot, PerplexityBot) fetch pages to cite in real-time AI answers. Blocking these means you lose the chance to be cited - and the referral traffic that comes with it.

For most businesses that want visibility, the pragmatic stance is to allow search/answer crawlers (so you get cited and referred) and decide on training crawlers based on how protective you are of your content. If your growth strategy includes being found in AI answers - as it should in 2026 - blocking everything is usually a mistake. Pair your decision with our AI search visibility checker and llms.txt generator.

• Accidentally blocking Googlebot. A stray Disallow: / under User-agent: * can de-index your whole site. Never block normal search crawlers.
• Using robots.txt to hide sensitive pages. It's public - listing a secret path actually advertises it. Use authentication instead.
• Forgetting the sitemap. Add a Sitemap line so crawlers discover all your URLs.
• Wrong location. It must be at the domain root and applies per host/subdomain.
• Confusing crawling with indexing. Disallowing a URL stops crawling, not necessarily indexing; use a noindex meta tag to keep a page out of results.