Free tools · AI bots checker
AI bots checker.
Robots.txt is a request; the firewall is the reality. This reads your robots.txt for 16 AI user agents, then loads your homepage as GPTBot, ClaudeBot and PerplexityBot to see what they actually get.
What this checks, and why it matters
Every assistant that browses the web sends a named crawler. OpenAI uses GPTBot for training, OAI-SearchBot for ChatGPT search and ChatGPT-User when it opens a page during a live answer. Anthropic, Perplexity, Google, Microsoft, Apple, Meta and Amazon each have their own. A robots.txt rule or a firewall challenge that turns one away means that assistant answers questions about you from other people's pages.
The checker reads robots.txt the way crawlers do: the most specific User-agent group wins, and a bare Disallow: / is a full block. Then it fetches your homepage as three real bot user agents, because a bot-fight rule can block a crawler that robots.txt allows.
Blocking training crawlers is a legitimate choice. Blocking the search and live-answer agents costs recommendations, so decide per bot.
The full picture
This is one check. The scan is all of them, plus what assistants actually say.
A free scan asks ChatGPT your customers' questions, shows who gets recommended instead, runs the technical audit, and drafts the fixes.
Run a free scan