Robots.txt & AI crawler checker

This tool does two jobs. The checker fetches your site's /robots.txt and tells you, in plain language, which of the major AI crawlers it currently allows or blocks - including OpenAI's GPTBot, Google-Extended, PerplexityBot, ClaudeBot, Common Crawl's CCBot, and more. The generator lets you tick the crawlers you want to block, add your sitemap, and copy a ready-to-deploy robots.txt. It's the fastest way to take control of how AI systems use your content.

robots.txt is a plain-text file at your domain root. It follows the Robots Exclusion Protocol. The file is split into groups. Each group starts with one or more User-agent lines (naming a crawler) followed by Allow and Disallow rules. Disallow: / blocks the whole site for that crawler; an empty Disallow: allows everything. A User-agent: * group is the fallback for any crawler you have not named. You can also declare your sitemap with a Sitemap: line. For the full spec, see Google's robots.txt documentation.

One critical caveat: robots.txt is voluntary. Good crawlers honor it. Nothing makes them, and a bad actor can ignore it. To truly block access you need server-side controls. Think of robots.txt as a clear, respected request - not a lock.

This is a strategy decision, not a technical one. There are two distinct kinds of AI crawler, and they deserve different treatment:

• Training crawlers (GPTBot, CCBot, ClaudeBot, Google-Extended, Bytespider) collect content to train models. Block them and your content stays out of training data. That is useful if you sell content, or if you want to guard your IP.
• Search/answer crawlers (OAI-SearchBot, PerplexityBot) fetch pages to cite in live AI answers. Block these and you lose the chance to be cited, plus the referral traffic that comes with it.

Most businesses want visibility. The sane stance is to allow search/answer crawlers (so you get cited and referred). Then decide on training crawlers based on how closely you guard your content. If your growth strategy includes being found in AI answers - as it should in 2026 - blocking the lot is usually a mistake. Pair your decision with our AI search visibility checker and llms.txt generator.

• Accidentally blocking Googlebot. A stray Disallow: / under User-agent: * can drop your whole site from the index. Never block a normal search crawler.
• Using robots.txt to hide sensitive pages. The file is public, so listing a secret path is a way of advertising it. Put a login in front of it instead.
• Forgetting the sitemap. Add a Sitemap line so crawlers discover all your URLs.
• Wrong location. It must be at the domain root and applies per host/subdomain.
• Confusing crawling with indexing. A Disallow rule stops crawling. It may not stop indexing. Use a noindex meta tag to keep a page out of the results.