TL;DR: Each AI provider runs two kinds of crawler, and the difference decides whether you can be cited. Training crawlers (GPTBot, ClaudeBot, Google-Extended) collect content for future models; search and retrieval crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) fetch pages to answer questions now, with citations. Blocking a retrieval bot removes you from that engine's answers entirely. Most sites that fail this fail it by accident, either through a stale robots.txt or a Cloudflare toggle that silently overrides a correct file. Check both layers.
What does "not blocking the bots" mean?
It means confirming that the AI crawlers you want can actually reach your pages, at both the robots.txt layer and the CDN layer. This is check 14 of the AI SEO checklist and, in practice, the one the most sites fail without knowing it. You can do every other technical pre-flight check perfectly, write flawlessly structured and sourced content, and still be completely invisible in AI answers because a crawler was blocked three years ago and nobody checked.
robots.txt is a plain text file at your domain root that tells crawlers which paths they may fetch. It was standardized as RFC 9309, but it is a polite request, not an enforced barrier, so it works only for bots that choose to honor it. The major AI crawlers do.
Why does the training-vs-search distinction matter so much?
Because they are separate bots with separate jobs, and blocking the wrong one has opposite consequences. Each major provider runs a fleet split by purpose: a training crawler that collects content for future model versions, a search or retrieval crawler that indexes pages to answer queries, and often a user bot that fetches a page the instant someone asks about it.
The distinction is the whole point of the check. If you block the training crawler, you keep your content out of model training but stay eligible for AI-search citations. If you block the retrieval crawler, you remove yourself from that engine's answers entirely, no matter how good your content is. This means a site can make a deliberate, nuanced choice: opt out of training while staying citable. What you must never do by accident is block the retrieval bots and disappear from AI search without realizing it.
Which crawlers should you know by name?
The ones that decide your visibility, grouped by job:
- OpenAI: GPTBot (training), OAI-SearchBot (powers ChatGPT search results), ChatGPT-User (live fetch when a user clicks a citation).
- Anthropic: ClaudeBot (training), Claude-SearchBot (retrieval for Claude's search), Claude-User (real-time user fetch).
- Google: Google-Extended (opts out of Gemini training without affecting Google Search ranking), separate from Googlebot itself.
- Perplexity: PerplexityBot (retrieval for Perplexity answers).
- Common Crawl: CCBot (an open dataset used to train many models).
One trap worth naming: the old Anthropic strings anthropic-ai and claude-web are deprecated. A robots.txt that blocks only those is not blocking Anthropic's live crawler, which is now ClaudeBot. Rules copied from a 2023 example routinely target names that no longer do anything.
Why is the CDN layer the bigger risk than robots.txt?
Because a firewall can silently override a perfectly correct robots.txt. Cloudflare, and similar CDN and security layers, ship an "AI Scrapers and Crawlers" or "block AI bots" control that treats retrieval bots like OAI-SearchBot and PerplexityBot the same as malicious scrapers. If that toggle is on, your robots.txt can say "allow" all day and the bot still gets a 403 at the edge before it ever reaches your file. A large share of sites block AI crawlers this way without knowing it, which is why the check has two layers, not one.
So verification is a two-step job. Read your robots.txt and confirm the retrieval bots are allowed, then check your CDN or WAF and confirm it is not blocking them at the edge and is not overriding your origin robots.txt. Studies through 2026 have measured meaningful traffic declines for publishers who block AI crawlers, so this is a revenue question, not a theoretical one.
How do you decide your policy?
Decide per category, on purpose, rather than accepting whatever your CMS shipped. The common posture for a brand that wants visibility: allow the search and retrieval bots so you stay citable, and make a deliberate call on the training bots based on whether you want your content in model training. A reasonable middle ground allows everything and keeps a server-level rule ready for any bot that misbehaves.
Two guardrails regardless of posture. Keep sensitive paths (admin, login, checkout, account) blocked for all bots, AI or otherwise. And remember that a one-line mistake here is dangerous in both directions: a stray Disallow can remove your site from Google overnight, so test changes on staging or with Search Console's robots.txt tester before pushing to production.
Allowing AI search vs blocking it by accident
| Signal | Reachable by AI | Accidentally blocked |
|---|---|---|
| robots.txt | Retrieval bots allowed | Copied 2023 file blocking them |
| Deprecated strings | Targets live ClaudeBot | Blocks only anthropic-ai, claude-web |
| CDN layer | Verified not overriding robots.txt | "Block AI bots" toggle silently on |
| Training vs search | Deliberate choice per category | Retrieval bots blocked unknowingly |
| Sensitive paths | Blocked for all bots | Exposed or inconsistently gated |
| Result | Eligible for AI citations | Invisible in AI answers |
How we apply this at HBS
We verify both layers before a site is considered live: robots.txt allows the retrieval crawlers and targets current user-agents rather than deprecated ones, and the Cloudflare or CDN layer is confirmed not to be silently blocking AI bots or overriding the origin file. We audit this quarterly, because the bot landscape shifts fast and a file that was correct last year may be blocking a newly split crawler today. Catching an accidental block is one of the highest-impact fixes in our technical pre-flight, and it is a week-one deliverable in the Advanced SEO Solutions work we ship, because a brand invisible to retrieval bots is invisible in AI search no matter how good everything else is.
The check that quietly decides everything
You can win every other part of this checklist and still lose AI search entirely if a retrieval crawler cannot reach you. Check robots.txt, check the CDN, target the live user-agents, and make your allow-and-block choices deliberately. If you are not sure whether AI engines can actually crawl your site, our Advanced SEO Solutions team can audit both layers, fix the accidental blocks, and confirm you are eligible for citations across ChatGPT, Claude, Perplexity, and Google. Get a free audit and find out whether the engines can reach you at all.
