robots.txt for AI crawlers: which bots you must never block

The only crawlers you can safely block in robots.txt are the ones that collect text for model training: GPTBot from OpenAI, ClaudeBot from Anthropic, Google-Extended from Google. The crawlers that bring you people must stay open: OAI-SearchBot powers search inside ChatGPT, Claude-SearchBot powers Claude's answers, PerplexityBot powers Perplexity, YandexBot builds the Yandex index that Alice and Neuro answer from.

The names look alike, which is why they get mixed up. Most instructions on the internet offer to "block your site from AI" with one list of ten lines, both groups end up in it, and the site disappears from AI answers entirely. New articles will not fix that: the crawler that shows your page to a person simply never arrives.

Checking your own file takes a minute. Open your-site.com/robots.txt in a browser and look for the line Disallow: /. If it sits under User-agent: * with no exceptions, the whole site is closed to everyone. If OAI-SearchBot, Claude-SearchBot, PerplexityBot or YandexBot are listed next to it, the ones that bring customers are exactly the ones you blocked.

Which AI crawlers visit websites?

A crawler is a program that walks through sites and reads pages. Each company runs several, and they come for different reasons.

Numbers come from the companies' own documentation as of 10 August 2026: OpenAI runs four crawlers, Anthropic three, Perplexity two.

The list of names goes stale faster than the guides explaining how to configure them, and you can see it in the wild. On 10 August 2026 we pulled robots.txt from 46 large Russian websites: media, retail, classifieds, banks, SaaS. Thirty-one served the file, the rest answered with an error or a redirect. Six of those 31 name any AI crawler at all. GPTBot appears on six sites, OAI-SearchBot and PerplexityBot on four, Claude-SearchBot on one. Meanwhile Claude-Web and anthropic-ai, neither of which exists in Anthropic's current documentation, show up three times each. Dead names get copied three times more often than Claude's live search crawler.

The most telling file in that sample belongs to a large building-supplies retailer. Fifteen AI agents are listed by name and allowed, Claude-Web among them. Claude-SearchBot never made the list, so it reads the general User-agent: * group, and that group holds Disallow: / plus a whitelist of a dozen sections. Claude's crawler gets a shop window instead of a site, and nobody ever decided that.

One note on the third row, the user-triggered crawlers. Per their own documentation ChatGPT-User and Perplexity-User may ignore robots.txt, because the request came from a live person rather than a scheduled crawl. OpenAI writes "robots.txt rules may not apply"; Perplexity is blunter: "generally ignores robots.txt rules". Anthropic says all three of its crawlers honor robots.txt directives. Blocking this group buys you no protection, it only breaks the scenario where a client pastes a link to you into a chat and asks about it.

What breaks if you block the wrong ones?

By consequence, not by name.

You blocked a training crawler (GPTBot, ClaudeBot, Google-Extended). Your texts will not feed future models. Visibility in search and in AI answers is untouched. Google states it plainly: Google-Extended does not affect a site's inclusion in Google Search and is not used as a ranking signal. That is an owner's decision, not a defect.

You blocked a surfacing crawler (OAI-SearchBot, Claude-SearchBot, PerplexityBot, YandexBot). The site stops showing up in answers. A model will not present what it never read, and it has nothing to cite. In a list of links a page survives on the far positions and still collects a trickle of clicks. An AI answer holds a handful of sources, and there is nothing to pull from if the crawler never arrived.

**You blocked everyone with User-agent: * / Disallow: /.** The most common case by far. That is what robots.txt looks like on a staging copy while the site is being built. Then the site ships with that file attached and lives that way for years.

There is a subtler trap too. Both Google and Yandex read only one group of rules: the most specific one matching that crawler, and the other groups do not apply. Google puts it this way: "Only one group is valid for a particular crawler". Yandex follows the same rule. So a file like this

```text User-agent: Googlebot Allow: /

User-agent: * Disallow: / ```

does not mean "open for Google and mostly open for others". It means "open for Google only, every AI crawler stays outside". People write this when they want to "let the search engine in".

What does a working robots.txt look like?

It is a plain text file in the site root. It opens at https://your-site.com/robots.txt, anyone can read it, no password needed. It has to sit in the root: a file at site.com/folder/robots.txt is ignored. Rules apply only to the host, protocol and port where the file lives, so the subdomain shop.site.com needs its own separate file.

For most businesses the working version is short:

```text User-agent: * Allow: /

Sitemap: https://your-site.com/sitemap.xml ```

The first two lines mean "all crawlers, everything is allowed". The third points to the sitemap, a list of every address on the site, so crawlers do not have to guess.

If you decided not to hand your texts to training but want to stay in answers, add separate groups:

```text User-agent: * Allow: /

User-agent: GPTBot Disallow: /

User-agent: ClaudeBot Disallow: /

User-agent: Google-Extended Disallow: /

Sitemap: https://your-site.com/sitemap.xml ```

Only the training crawlers are blocked here. OAI-SearchBot, Claude-SearchBot, PerplexityBot and YandexBot fall under the general permission and keep surfacing the site.

Technical limits per Google documentation: the file is parsed up to 500 kibibytes, anything beyond that is dropped, and the contents are cached for roughly 24 hours, so an edit will not take effect instantly.

How do you check your own file in a minute?

1. Open your-site.com/robots.txt in a browser. A blank page or a 404 means there is no file. That alone is not a disaster: no file, no rules, crawlers walk freely. But nobody told them where the sitemap is either.
2. Find Disallow: / in the text. A lone slash with nothing after it closes the entire site. Lines like Disallow: /admin close one section, which is normal.
3. Run through the names in the table above. Any crawler from the first group sitting in a block rule needs fixing today.
4. Check that a Sitemap: line exists.

If you want a machine check instead of your own eyes: Google Search Console has a robots.txt report (Settings → robots.txt). It shows which files Google actually fetched, when, and with which errors, and keeps the request history for the last 30 days; on a Domain property it covers the files of the top twenty hosts. Yandex Webmaster has a robots.txt analyzer where you paste a specific URL and see whether it is allowed or blocked.

Changes do not take effect right away. Google's documentation says crawling a new page takes anywhere from a few days to a few weeks, and asking for a recrawl of the same URL several times does nothing.

One file answers one question. The full list of what keeps AI assistants from seeing your site is covered separately: how to check in five minutes whether AI assistants can see your site.

The file is clean, so why are crawlers still blocked?

The second most common place visibility disappears is bot protection at the CDN or hosting layer. It lives entirely outside robots.txt: the file is wide open, the crawler gets a 403 and leaves.

Cloudflare has a dedicated AI Crawl Control section, available on every plan including the free one. On 1 July 2026 Cloudflare replaced the single "block AI bots" switch with three categories: Search (crawlers that index your content to answer questions later and send referral traffic back), Agent (automated activity acting in real time on a person's behalf), Training (crawlers that take content to train a model).

From 15 September 2026 new domains onboarding to Cloudflare get new defaults: Training and Agent are blocked on pages that display ads, Search stays allowed. Any customer can opt out of the new defaults at any point before 15 September. And if you ever pressed "Block AI bots", one more detail matters: multi-purpose crawlers that combine Search and Training also fall under the training block.

A third reason has nothing to do with crawlers: the whole site is rendered by scripts and the page source contains no text. The crawler arrives and sees a blank page. To test this, open the page source; the full walkthrough is in the site-check article.

And a caveat about the mechanism itself. robots.txt controls crawling, not display. Google warns that a URL blocked in robots.txt can still be indexed if other sites link to it, just without a snippet. Removing a page from search requires a noindex meta tag or a password, not a crawl ban.

What should you do before 15 September?

Open your robots.txt and look at it once. That block line usually appeared when the site launched, put there by a contractor or a site builder, and nobody has read it since.

If your site sits behind Cloudflare, open AI Crawl Control and check what is set right now. The defaults change on 15 September, and knowing your own settings in advance is cheaper than untangling them afterwards.

The fix takes five minutes. Without it your articles, structured data and sitemap run empty: there is nobody to read them.

A walkthrough with ready-made prompts (a seven-point site check and the fixes that follow from it) lives in the Echo block of the Academy, three lessons are free: https://academy.agineai.com/block/b12-eho