Which AI Crawlers Should You Allow? A robots.txt Guide

Before an AI system can cite you, it has to be able to read you. That sounds obvious, and yet "can the bots get in?" is the first thing to check on almost every site that's invisible in AI answers. Sometimes robots.txt is blocking the wrong bot. More often, something upstream like a CDN or firewall is quietly turning the bot away before robots.txt is ever consulted.

This post explains the crawlers that matter, what each one does, a sensible default robots.txt, and how to verify that it's all working.

Three jobs, three kinds of bot

The mistake most site owners make is treating "AI crawlers" as one thing. The big AI companies each run separate bots for separate jobs, and blocking the wrong one has very different consequences.

1. Training crawlers collect content to train future models. Blocking them opts your content out of training. It does not remove you from AI search answers.
2. Search crawlers build the index that AI search features retrieve from. Block these and you disappear from that product's answers.
3. User-triggered fetchers visit a page because a person asked the assistant to. Because a human requested the fetch, these often don't follow robots.txt the way crawlers do.

The crawlers to know

User agent names and behavior change, so treat this table as a snapshot and check each vendor's documentation before you make changes. It's accurate as of September 2026.

Sources: OpenAI's bot documentation, Anthropic's crawler help article, Perplexity's bot guide, and Google's AI features documentation.

Two things in that table are worth underlining. First, Google doesn't run a separate crawler for AI Overviews or AI Mode: ordinary Googlebot access, plus being indexed and eligible for a snippet, is the requirement. Second, Google-Extended is a control token, not a ranking lever. Blocking it doesn't help or hurt your search performance.

So which should you allow?

Split the decision in two.

Search and retrieval bots: allow them

If you want to be cited in ChatGPT, Claude, or Perplexity, their search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) need access. For a business that lives on being found, blocking these is like taking your sign down.

Training bots: your call

Allowing GPTBot and ClaudeBot lets your content inform future models, which may help those models describe your business accurately. Blocking them is a legitimate choice if you're a publisher protecting paid content or you object on principle. The important thing is that it's a separate decision from being findable in search.

A sensible default robots.txt

If you want maximum visibility, you don't need to list AI bots at all. The standard "allow everything" file already permits them:

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

That's the entire robots.txt for this website. If you want to opt out of training but stay visible in AI search, list the training bots explicitly:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Each crawler follows the most specific group that matches its name, so the search bots fall through to the * rule and stay allowed. After you change the file, expect a delay: OpenAI says it can take about 24 hours for its systems to adjust.

One caution: robots.txt is a request, not a lock. Well-behaved crawlers honor it. It does nothing to stop a scraper that ignores it, and it isn't a privacy tool. Anything truly private belongs behind a login.

The hidden problem: your firewall says no

Here's what catches people. Your robots.txt can say "welcome" while a layer in front of your site says "403 Forbidden." Content delivery networks and security plugins have started treating AI crawlers as a threat category. Cloudflare, for example, has blocked AI crawlers by default on newly added domains since July 2025, and it offers site-wide controls to change that. Managed hosting, WordPress security plugins, and bot-protection features can do the same.

If you changed CDN or security settings any time in the last couple of years, check the bot settings specifically. Look for a setting named something like "AI bots," "AI scrapers and crawlers," or "bot fight mode," and confirm that the search crawlers you want are allowed.

The JavaScript problem

Even a crawler that gets through can come away with nothing. A study by Vercel and MERJ, published in December 2024, analyzed hundreds of millions of crawler requests and found that none of the major AI crawlers rendered JavaScript. They sometimes downloaded script files but did not run them. Googlebot is the notable exception because it has a full rendering pipeline.

That study is now nearly two years old, and vendors may have changed things since, so verify against your own logs. But the safe assumption holds: if your key content, headings, links, or schema only appear after JavaScript runs, plan for some crawlers seeing a blank shell. The fix is server-side rendering or static generation for anything you want cited. This site is plain static HTML, which is why it's easy to read.

How to check that it's working

1. Read your robots.txt. Visit yourdomain.com/robots.txt. Look for broad Disallow: / rules under User-agent: * left over from a staging site, and for bots blocked by name.
2. View the raw HTML. Run curl -s https://yourdomain.com/your-page/ | grep -i "your key phrase". If the phrase isn't in the raw response, non-rendering crawlers can't see it.
3. Test your firewall. Request a page with a crawler's name as the user agent, for example curl -I -A "OAI-SearchBot" https://yourdomain.com/. A 403 or a challenge page is a red flag. Note that this only tests user-agent rules; some firewalls also verify the crawler's IP range, so a passing test isn't proof.
4. Check your logs. The most reliable evidence is your server or CDN log. Search for OAI-SearchBot, ClaudeBot, and PerplexityBot, and confirm they receive 200 responses on real pages.
5. Confirm Google can render and index the page. Use the URL Inspection tool in Search Console. Being indexed and snippet-eligible is Google's stated requirement for showing up in its AI features.

What crawler access doesn't do

Being crawled is necessary and nowhere near sufficient. A crawler visiting your site tells you the door is open, not that anyone was impressed. Whether you get cited depends on everything downstream: whether the content is worth citing, whether the system understands who you are, and whether the wider web backs you up. The rest of the AI Search Visibility guide covers those layers, and the next post tackles the file everyone asks about: llms.txt.