OAI-SearchBot, GPTBot, PerplexityBot, ClaudeBot: which AI crawlers to allow and which to block

Roughly one in five sites we scan is blocking at least one AI crawler without meaning to. Usually it happened in 2023 or 2024, when "block the AI bots" snippets went around and someone pasted one into robots.txt. The snippet did its job. It also removed the business from ChatGPT search answers.

This guide is the reference we wish existed then: what each crawler does, which ones affect whether you get recommended, and the exact rules to use.

Key takeaways

Blocking a training crawler and blocking a search crawler are different decisions. GPTBot trains models; OAI-SearchBot decides whether you appear in ChatGPT search. Most businesses should allow the search and answer crawlers, decide separately about training, and never use a blanket Disallow: /.

How robots.txt works, in one minute

robots.txt is a plain text file at the root of your domain that tells well-behaved crawlers which paths they may fetch. The format is standardised in RFC 9309: a User-agent line names a crawler, followed by Allow and Disallow rules. User-agent: * applies to any crawler without a more specific group. Google's introduction to robots.txt is the clearest practical explanation.

Two things trip people up:

• A crawler uses the most specific group that matches it. If you have a GPTBot group and a * group, GPTBot follows only its own group.
• robots.txt controls crawling, not indexing. A page that's blocked can still be referenced from elsewhere, but the crawler can't read its content, so an AI can't quote your facts from it.

The crawlers that matter for AI visibility

Sources: OpenAI's crawler overview, Perplexity's bot documentation, Anthropic's crawler help article, Google's crawler overview, Common Crawl's CCBot page.

The OpenAI split

OpenAI documents three agents with different purposes. OAI-SearchBot is the one that determines whether your site can appear in ChatGPT search. GPTBot is for training. ChatGPT-User acts on behalf of a user in real time. OpenAI's documentation says sites can allow OAI-SearchBot while disallowing GPTBot, and that the search bot is what you want to allow to be surfaced in search.

If you only remember one thing from this guide: a GPTBot block does not remove you from ChatGPT search, but an OAI-SearchBot block does.

The Google situation

There is no separate crawler for AI Overviews or Gemini grounding. Googlebot fetches your pages once for everything. Google-Extended is a robots.txt token that lets you opt out of your content being used for Gemini training and grounding, without affecting Search. Blocking Google-Extended may keep you out of Gemini answers while leaving you in classic search results. That's a legitimate choice for some publishers. For a business that wants to be recommended, it's the opposite of what you want.

Perplexity and Anthropic

Perplexity documents PerplexityBot for indexing and Perplexity-User for user-triggered fetches. Anthropic documents ClaudeBot and honours robots.txt for it. Both are worth allowing if you want to be findable when someone asks those assistants about your category.

How to check your own site

1. Open https://yourdomain.com/robots.txt in a browser. Look for the user agents above and for Disallow: / under User-agent: *.
2. Check your CDN or WAF. Cloudflare, Akamai and similar services have "block AI bots" toggles that act before robots.txt is ever read. These are easy to enable by accident and invisible in your robots file.
3. Fetch a page as the bot. From a terminal:

curl -A "OAI-SearchBot/1.0" -I https://yourdomain.com/

A 200 means the bot can fetch it. A 403 or a challenge page means it can't, regardless of what robots.txt says.

4. Check the text version. AI crawlers don't execute much JavaScript. If your services, prices or address only appear after a script runs, they aren't there for the bot.

TangentFlow's free scan runs all four checks automatically as part of the technical audit.

A robots.txt you can copy

For a business that wants to be recommended by AI assistants and has no objection to its public pages being used, this is a sensible default:

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /account/

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

If you want to stay visible in AI search but keep your content out of model training, allow the search agents and block the training ones:

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Note that this second configuration is a trade-off. Training data influences what models "know" without searching, so a strict training block can make an assistant less likely to describe you from memory. For most small businesses the search agents are what matter, and the training question is a values decision rather than a visibility one.

Common mistakes we see in scans

• A single Disallow: / under User-agent: * left over from a staging site. It blocks everyone, including Google.
• Blocking GPTBot and assuming that's ChatGPT search. It isn't. See above.
• A CDN bot-fight mode that serves a JavaScript challenge to every non-browser. Crawlers can't solve it, so they see nothing.
• noindex on service pages added by a plugin default. Robots.txt is fine, but the pages are excluded anyway.
• Blocking crawlers from /blog/ or /docs/. For SaaS products, docs pages are frequently the exact pages assistants cite for integration and pricing facts.

What about llms.txt?

llms.txt is a proposed convention: a Markdown file at /llms.txt that summarises your site for language models and links to the pages worth reading. Adoption by the major assistants is not documented, so treat it as low-cost and low-certainty. It takes ten minutes, it can't hurt, and it's a useful exercise in writing down the facts you want AI to know. We include a generated draft in every scan.

FAQ

Will allowing AI crawlers slow down my site? The AI crawlers are modest compared with Googlebot. If a crawler is genuinely hammering your server, use Crawl-delay where supported or rate-limit at the CDN rather than blocking outright.

I blocked everything in 2024. How long until I'm back in answers? Technical unblocks tend to show within days for ChatGPT search and Perplexity, because they fetch live. Gemini follows Googlebot's normal recrawl schedule. Re-check weekly to confirm.

Can I allow the bots on some pages only? Yes. Use Allow and Disallow with path prefixes under each bot's group. Keep service, pricing, location and comparison pages open at minimum.

Do crawlers respect robots.txt? The ones listed here document that they do. Unknown scrapers may not, but those aren't the ones deciding whether ChatGPT recommends you.

Sources

• IETF, RFC 9309: Robots Exclusion Protocol
• Google Search Central, Introduction to robots.txt
• Google Search Central, Overview of Google crawlers and fetchers
• OpenAI, Overview of OpenAI crawlers
• Perplexity, Perplexity crawlers
• Anthropic, Does Anthropic crawl data from the web?
• Common Crawl, CCBot
• Microsoft Bing, Webmaster guidelines
• llms.txt, The /llms.txt proposal