AI Crawler List 2026: Every Bot + Should You Block Them?

Anthropic last updated its crawler documentation on 7 April 2026. Most AI-crawler lists ranking on Google predate that update, and they agree on something now wrong: that ClaudeBot fetches citations during chat sessions and anthropic-ai is the training crawler.

Anthropic's documentation says the opposite. ClaudeBot is the training crawler. Claude-User fetches pages when a person asks Claude about them, and Claude-SearchBot builds the index behind Claude's search results. Allow ClaudeBot because a list called it the citation fetcher, and you have allowed training for no citation benefit.

Memorising bot names is the wrong exercise. The names churn; the three classes underneath them do not, and the class determines what you lose the moment you write Disallow.

Table of contents

• What are the three types of AI crawler?
• The complete AI crawler list for 2026
• Why do crawler lists disagree about ClaudeBot?
• Should you block GPTBot and ClaudeBot?
• What does Google-Extended actually control?
• How do you block (or allow) AI crawlers in robots.txt?
• Do AI crawlers respect robots.txt?
• What do you do when a crawler overloads your server?
• Does robots.txt carry legal weight?
• Where CrawlForge fits into this

What are the three types of AI crawler?

Every bot below belongs to one of three classes. Getting the class right is the whole job.

The asymmetry matters. Training crawlers affect a model shipping in six months. Search-index and user-triggered crawlers affect whether someone gets your page in an answer this afternoon.

A blanket Disallow: / for everything AI-shaped — what most "block AI bots" snippets do — collapses those three into one decision. That is how sites quietly disappear from AI answers while believing they only opted out of training.

The complete AI crawler list for 2026

Sources: OpenAI's bot documentation, Anthropic's crawler support article, and Momentic's maintained AI crawler list (updated November 2025). Where a vendor contradicts a third-party list, the vendor wins. Tokens match case-insensitively but are reproduced exactly as published.

Which AI crawlers cannot be blocked in robots.txt?

A growing category of AI traffic is not a crawler at all. Agentic browsers drive a real browser for a user, and several are indistinguishable from Chrome traffic in your logs.

If your policy depends on stopping these, robots.txt is the wrong layer. You need IP rules, a WAF, or authentication — each of which blocks the humans using those tools too.

Why do crawler lists disagree about ClaudeBot?

Because the roles were reassigned and third-party lists have not caught up. Anthropic now documents exactly three crawlers — ClaudeBot for training, Claude-User for user-initiated fetches, Claude-SearchBot for search indexing — on a page dated April 2026. Momentic's November 2025 list, one of the better-maintained ones, still describes ClaudeBot as the chat-citation fetcher and anthropic-ai as the primary training crawler.

When a third-party list and vendor documentation disagree, follow the vendor and keep the legacy token anyway. An extra User-agent: anthropic-ai block costs nothing and covers the case where the older behaviour is still live somewhere.

Should you block GPTBot and ClaudeBot?

Blocking training crawlers is a policy decision, not an SEO one. The part worth internalising: blocking GPTBot and ClaudeBot costs zero AI-search visibility today, because ChatGPT search runs on OAI-SearchBot and Claude search on Claude-SearchBot. OpenAI states the settings are independent — allow OAI-SearchBot while disallowing GPTBot — noting only that if both are allowed it may reuse one crawl for both.

So the decision splits cleanly:

• You publish to be found. Allow all three classes. Search and user-triggered bots are the new referral channel; blocking them is a noindex on your fastest-growing discovery surface.
• Your content is the product (documentation you sell, licensed research, a paywalled archive). Block training, allow search and user-triggered fetches: discoverable, but out of the corpus.
• The content is genuinely sensitive. robots.txt is not access control. Use authentication.

What not to do is block everything AI-sounding because a checklist said so. That combination makes you invisible in AI answers while doing nothing authentication would not do better.

What does Google-Extended actually control?

Google-Extended is a robots.txt token, not a crawler. No request in your logs will ever carry it as a user agent. Googlebot fetches; the token controls whether that content may be used for Gemini training and grounding. Two consequences follow, both routinely mishandled:

Blocking Google-Extended does not affect your Google Search ranking. It is a use-restriction on already-crawled content, evaluated separately from indexing.

You cannot opt out of AI Overviews without opting out of Search. They are generated as part of Google Search, from the Search index. The only lever that removes you is disallowing Googlebot — an expensive way to avoid a summary.

How do you block (or allow) AI crawlers in robots.txt?

Here is a complete file for the second case above: search and user-triggered crawlers allowed, training crawlers blocked, expensive endpoints protected. If you are in the first case — publishing to be found — move the section 3 groups up into section 1 and allow them too. That is what CrawlForge's own robots.txt does, training crawlers included. Serve it at /robots.txt on every host and subdomain you own — robots.txt is per-origin, so blog.example.com needs its own copy.

# 1. AI search crawlers — allow these or you vanish from AI answers User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: DuckAssistBot User-agent: YouBot User-agent: Amazonbot Disallow: /search Disallow: /cart/ Disallow: /*/compare/ Allow: / # 2. User-triggered fetchers — a human asked for this page User-agent: ChatGPT-User User-agent: Claude-User User-agent: Perplexity-User User-agent: MistralAI-User Allow: / # 3. Training crawlers — opt out of the corpus User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: CCBot Disallow: / User-agent: cohere-ai Disallow: / # 4. Everyone else, including Googlebot and Bingbot User-agent: * Disallow: /search Disallow: /cart/ Disallow: /*/compare/ Allow: /

Two syntax notes. Every User-agent line needs at least one Allow or Disallow after it, or the group does nothing. And path wildcards (*, $) are honoured by major crawlers but not universally, so plain prefixes are safest for rules you cannot afford to have ignored.

The grouping rule that breaks most robots.txt files

The most common mistake in AI-crawler configs, and RFC 9309 is worth reading to internalise it: a crawler obeys exactly one group — the most specific one whose user-agent line matches it. It does not merge that group with User-agent: *.

So section 4's Disallow: /search above protects Googlebot and every unnamed bot. It does not apply to OAI-SearchBot, which has its own group in section 1 — which is why section 1 repeats those lines. Put your expensive-endpoint rules only under User-agent: *, add named AI-bot groups, and you have exempted every AI crawler from your crawl traps.

Do AI crawlers respect robots.txt?

The reputable ones do, with three real caveats.

User-triggered fetches are explicitly outside it. OpenAI documents that because ChatGPT-User actions are initiated by a user, robots.txt rules may not apply. That is the same reasoning that lets your browser fetch a page your robots.txt disallows.

Changes take time to land. OpenAI puts robots.txt propagation at roughly 24 hours, so expect a day of continued traffic before concluding a bot is misbehaving.

It is voluntary. robots.txt is a request, not an enforcement mechanism. Anthropic is explicit that robots.txt is its supported opt-out and that IP blocking "may not work correctly or persistently guarantee an opt-out" — precisely because blocking IPs stops the crawler reading the robots.txt carrying your opt-out. If content must not be fetched, put it behind authentication.

How do you verify a crawler is genuine?

A user-agent string is an unauthenticated header. Anyone can send ClaudeBot/1.0, and plenty do — it is the cheapest way to inherit the reputation of a bot you allowlisted. Verify against published IP ranges instead:

These are JSON files in the format Google and Bing publish, so one fetch-and-match routine covers all of them. A request claiming to be GPTBot from an address outside those ranges is not GPTBot.

What do you do when a crawler overloads your server?

Block-or-allow is a false binary. Most pain attributed to AI crawlers is a rate problem, not an access problem.

A September 2025 r/webdev thread titled "ClaudeBot is hammering my server with almost a million requests in one day" — roughly 1,800 upvotes and over 200 comments — describes ClaudeBot hitting one site about 881,000 times in 24 hours. On the GitLab forum in May 2026, administrators describe returning 403 to a user-agent regex covering GPTBot, ClaudeBot and others — not to opt out of AI, but because crawlers were hitting expensive routes such as /-/blame.

That is the general shape of the problem: crawlers follow every link, including ones that trigger a database-heavy diff render. Three fixes, in order of preference:

1. Disallow the expensive routes, not the bot. Blame views, diff endpoints, faceted-search permutations, calendar pages with infinite date ranges. Remember the grouping rule — repeat those lines in every named group.
2. Slow it down. Anthropic supports the non-standard Crawl-delay extension, so User-agent: ClaudeBot followed by Crawl-delay: 1 is honoured. Support elsewhere is inconsistent; use WAF rate rules keyed on the verified IP ranges above.
3. Report it. Anthropic asks that suspected bot malfunctions go to its support channel with the affected domain. A million requests a day from one origin is usually a fixable bug on their side.

Does robots.txt carry legal weight?

Increasingly, yes — indirectly. The EDPB's Guidelines 03/2026 — adopted 7 July 2026, in public consultation until the end of October 2026 — treat robots.txt and ai.txt signals as legally relevant to whether scraping for AI training satisfies the GDPR legitimate-interests balancing test. A machine-readable opt-out that a scraper ignored is now evidence in that assessment — a real change from robots.txt as pure etiquette. Full picture: is web scraping legal for AI in 2026.

Where CrawlForge fits into this

The disclosure is owed: CrawlForge is a web scraping platform, which puts us on the other side of this table. Our 28 tools fetch pages for AI agents, respecting robots.txt and rate limits when they do.

That is also the point. The policy you write for ChatGPT-User and Claude-User governs every AI agent built on a scraping API — including the one a prospective customer points at your docs while evaluating you.

If you decide to be readable by AI, make it cheap for them. A generated llms.txt gives agents a machine-readable map of the pages you want read, cutting the crawl volume needed to understand your site. If you run an MCP server yourself, the fetching side has its own hazards — see SSRF in MCP servers.

---

Building agents that read the live web? Start free with 1,000 credits — robots.txt compliance and rate limiting on by default. See the llms.txt generator or how CrawlForge compares to other scraping MCP servers.