robots.txt for AI Crawlers: GPTBot, ClaudeBot & More | Adviora

AI Crawler Access Audit: GPTBot, ClaudeBot, Google-Extended and PerplexityBot

Somewhere in the last two years, someone on your team added a block of rules to robots.txt for AI crawlers. It was probably a reasonable reaction to a legal email, a board question, or a headline about models training on published work. It was probably also copied from a blog post, applied to one host, and never reviewed again.

That single decision is now one of the most consequential technical settings on your site, and almost nobody has audited it. The reason is a distinction the copy-paste blocks do not make: every major AI provider runs separate user agents for training and for retrieval. Blocking the retrieval agent removes you from the answer surface — ChatGPT search answers, Claude's search results, Perplexity's citations — while doing nothing about the training exposure you were actually worried about.

This article maps every agent that matters, using only the providers' own published documentation, gives you a decision framework for the training-versus-visibility trade-off, and then walks the audit procedure end to end — including the two layers above robots.txt where access usually breaks. Adviora runs this same audit continuously inside the GEO & AEO Visibility module.

Why AI Crawler Access Matters in 2026

For twenty years, robots.txt governed one thing: whether a search engine could index you. The trade-off was simple and the answer was almost always yes. AI assistants broke that simplicity by introducing a second, genuinely contested question — may this content be used to train a model? — and then by routing their answer products through entirely different agents than their training pipelines.

The result is a file that now encodes two separate business positions at once, usually without anyone noticing. A block written to express “do not train on our content” routinely also expresses “do not cite us”, because the copied snippet listed every AI-sounding user agent under one Disallow.

The cost is invisible in every standard report. Search Console covers Google Search clicks, impressions, CTR and average position; it does not tell you that Perplexity stopped citing your comparison pages in March. That gap is the reason Adviora's GEO & AEO Visibility module pairs per-platform citation coverage with an AI crawler access audit — the second explains the first.

The Distinction Everyone Misses: Training Bots vs Visibility Bots

There are three functional categories of AI user agent, and they carry completely different consequences.

• Training crawlers collect content to train foundation models. Disallowing them signals that your content should not be used for model training. Examples: GPTBot, ClaudeBot, Google-Extended.
• Retrieval and indexing crawlers build the index the assistant searches when answering a live question. Disallowing them removes you from that answer surface. Examples: OAI-SearchBot, Claude-SearchBot, PerplexityBot.
• User-initiated fetchers visit a page because a human asked the assistant to. Examples: ChatGPT-User, Claude-User, Perplexity-User. robots.txt rules may not apply to these, since a person triggered the request.

Read that third category carefully. It is why robots.txt should be understood as a published statement of policy rather than a security boundary. If you need enforcement, you need the network layer; if you need a clear, honoured expression of intent to the major providers, robots.txt is the right instrument.

The audit that follows is really an exercise in separating these three categories for each provider and confirming that what your file says matches what your organisation actually decided.

OpenAI: GPTBot, OAI-SearchBot, OAI-AdsBot and ChatGPT-User

OpenAI publishes four agents with clearly separated purposes.

• GPTBot — crawls content to train generative AI foundation models. Disallowing it signals that your content should not be used for model training.
• OAI-SearchBot — powers ChatGPT's search features. Blocking it means your site will not appear in ChatGPT search answers, although it may still show as a navigation link.
• OAI-AdsBot — validates the safety and relevance of submitted ad landing pages. Its data is not used for model training, so blocking it is an advertising-operations decision, not a content-licensing one.
• ChatGPT-User — user-initiated fetches, including GPT Actions. robots.txt rules may not apply because a human triggered the request.

OpenAI publishes IP ranges for each agent at openai.com/gptbot.json, /searchbot.json, /adsbot.json and /chatgpt-user.json. Use those to verify identity, because user-agent strings are trivially spoofed and a meaningful share of traffic claiming to be GPTBot is not.

The single most common misconfiguration in the wild is a rule that disallows both GPTBot and OAI-SearchBot. That combination is almost never what the business intended: it protects nothing extra and it guarantees absence from ChatGPT search answers.

Anthropic: ClaudeBot, Claude-User and Claude-SearchBot

Anthropic publishes three crawlers with the same training-versus-retrieval split.

• ClaudeBot — crawls content used for training.
• Claude-User — fetches pages on behalf of a user who directed Claude to a specific source.
• Claude-SearchBot — indexes content to improve search result quality.

Control is via standard robots.txt directives — for example a User-agent: ClaudeBot group with Disallow: / — and Crawl-delay is supported. Anthropic states that its bots “respect ‘do not crawl' signals by honoring industry standard directives in robots.txt.”

Two operational details matter here and are widely missed. First, Anthropic explicitly warns that rules must be applied to every subdomain you want opted out; a rule on www does nothing for blog, docs, help or shop hosted separately. Second, Anthropic discourages blocking by IP, because a bot that cannot reach your site cannot read your robots.txt and therefore cannot honour it. The published IP list at claude.com/crawling/bots.json is for verification, not for building a blocklist.

Perplexity: PerplexityBot and Perplexity-User

Perplexity runs two agents, and the training question does not arise in the way people assume.

• PerplexityBot — surfaces and links websites in Perplexity search results. It is explicitly not used for AI model training.
• Perplexity-User — visits pages when a user's request requires it, and generally ignores robots.txt because a real person initiated the request.

Because PerplexityBot is a visibility crawler rather than a training crawler, disallowing it is close to a pure loss: you forfeit citations and links in Perplexity answers and you prevent nothing you were worried about. If your original block was written to address training exposure, PerplexityBot almost certainly does not belong in it.

Perplexity documents three control layers: robots.txt, WAF rules in Cloudflare or AWS combining user agent and IP, and IP allowlisting from the published list at perplexity.com/perplexitybot.json. The middle option is worth noting, because a WAF rule written by a security team is exactly the kind of block a marketing team never finds.

Google: Googlebot, Google-Extended, GoogleOther and Google-InspectionTool

Google's crawler family is larger and the naming is less intuitive, so this is where most audits produce a wrong conclusion.

• Googlebot (desktop and mobile) — crawls for Google Search. Blocking it removes you from Search, and therefore from AI Overviews and AI Mode, which are downstream of the Search index.
• Google-Extended — controls whether crawled content may be used for training Gemini models and grounding in Gemini Apps. Google states: “Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal.”
• GoogleOther — a generalist crawler used by product teams for one-off crawls and internal research; alongside GoogleOther-Image and GoogleOther-Video.
• Google-InspectionTool — powers the URL Inspection tool and Rich Results Test; it does not affect rankings, and blocking it breaks your own diagnostics.
• Specialist crawlers including Googlebot-Image, Googlebot-Video, Googlebot-News, Google StoreBot and Google-CloudVertexBot.

Google's crawlers respect robots.txt and their identity can be verified by reverse DNS lookup. The practical takeaway is that Google is the one provider where the training opt-out is genuinely separable from search visibility with a documented guarantee — Google-Extended is the lever, Googlebot is untouched, and blocking Googlebot to express an opinion about AI is self-harm.

Adviora's Technical SEO Governance module checks crawl and index directives per page, while the GEO & AEO Visibility module reports the agent-level access matrix across all four providers.

Deciding Your Policy: A Business Decision, Not a Technical One

The audit tells you what your site currently says. It cannot tell you what it should say — that is a content-licensing and brand-exposure judgement that belongs to legal, editorial and marketing together. Use a simple frame.

1. Separate the two questions explicitly: may this content train models, and may this content be retrieved and cited? Answer them independently, in writing.
2. Segment by content type, not sitewide. Proprietary research, subscriber content and licensed material often justify a training block; marketing pages, documentation and support content almost never do.
3. Price the visibility side honestly. If your buyers research in assistants, absence from the answer surface is a demand-generation cost, not a neutral position.
4. Check whether a licensing conversation exists. A training block is a negotiating position only if someone is negotiating; otherwise it is just absence.
5. Record the decision, the owner and the review date as comments in robots.txt itself, so the next person does not inherit an unexplained block.
6. Re-review on a fixed schedule and after any legal, CDN or CMS change.

Most organisations that run this properly end up in the same place: allow every retrieval and search agent, make a considered, segment-specific decision about training agents, and document why. The teams in trouble are the ones who never separated the questions.

The Audit Procedure: Seven Checks

Work through these in order. Each one catches a failure the previous one cannot see.

7. Fetch the live robots.txt over HTTP, not the version in your repository. Deployed files drift from source control more often than anyone expects, and edge rewrites can serve a different file entirely.
8. Repeat for every subdomain — www, blog, docs, help, shop, careers, regional hosts and any marketing microsite. Rules do not inherit across hosts, and Anthropic calls this out explicitly.
9. Build an agent-by-agent matrix rather than reading the file top to bottom. List GPTBot, OAI-SearchBot, OAI-AdsBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Googlebot, Google-Extended, GoogleOther and Google-InspectionTool, and mark each allowed or disallowed. Watch for a wildcard User-agent: * group that a later specific group unexpectedly overrides.
10. Verify real traffic against the published IP lists — openai.com/gptbot.json, claude.com/crawling/bots.json and perplexity.com/perplexitybot.json — and reverse DNS for Google. This tells you which agents actually visit, and exposes impostors.
11. Read server and CDN logs for the same agents. An agent that is allowed in robots.txt but absent from logs is being stopped somewhere else.
12. Inspect the WAF, CDN and bot-management layer. Managed bot rules, rate limits, JavaScript challenges and country blocks all sit above robots.txt and leave no trace in it. This is the most common cause of a clean file with no crawler traffic.
13. Check snippet-level controls. Even with crawling allowed, a nosnippet or low max-snippet rule — in a meta tag or an X-Robots-Tag header — removes the page from AI Overviews, because Google requires snippet eligibility for AI features. Check templates as well as individual pages.

Steps six and seven are where experienced teams find their real problem. A perfectly written robots.txt sitting behind a bot-management rule is the modern equivalent of a beautiful sitemap on a noindexed site. The Technical SEO Audit Anatomy article covers the crawl and index layer these checks sit on top of.

A Sample robots.txt Block

The example below expresses one common, deliberate position: allow everything that affects visibility, disallow training crawlers on a defined proprietary path only, and document the decision in the file. Adapt it — do not copy it as policy.

• # AI crawler policy — owner: Head of Digital — reviewed: [date] — next review: [date]
• # Visibility agents: allowed everywhere.
• User-agent: OAI-SearchBot / Allow: /
• User-agent: Claude-SearchBot / Allow: /
• User-agent: PerplexityBot / Allow: /
• User-agent: Googlebot / Allow: /
• # Training agents: allowed except proprietary research.
• User-agent: GPTBot / Disallow: /research/ / Disallow: /members/
• User-agent: ClaudeBot / Disallow: /research/ / Disallow: /members/
• User-agent: Google-Extended / Disallow: /research/ / Disallow: /members/
• # Note: ChatGPT-User, Claude-User and Perplexity-User are user-initiated and may not honour these rules.

Write each User-agent line and its directives on separate lines in the real file — the slashes above are only to keep the example compact on this page. Keep one group per agent rather than combining agents you may later want to treat differently, and put the review date in a comment where the next auditor will actually see it.

The Layers Above robots.txt

Three layers can override the intent in your file, and none of them are visible to a marketer reading robots.txt.

• Bot management at the CDN. Managed rule sets classify unknown agents as bots by default; an AI crawler may be challenged or blocked before your origin ever sees it.
• Origin controls. Rate limits, IP reputation lists, geographic restrictions and authentication walls all deny access without any robots directive involved.
• Snippet and index directives. nosnippet, data-nosnippet, max-snippet and noindex operate after crawling and control whether the content can be shown, which is what AI Overviews eligibility depends on.

There is a fourth consideration that is not a layer at all: user-initiated agents largely bypass robots.txt by design. ChatGPT-User, Claude-User and Perplexity-User exist because a person asked for a specific page. Treat robots.txt as the public statement of your policy and the network layer as the enforcement mechanism, and stop expecting one to do the other's job. Adviora's GEO & AEO Visibility module reports the access matrix and the snippet-directive findings together, because a fix at one layer is worthless while another still blocks.

Best Practices

1. Never write a blanket AI block. Decide per agent, per provider, per content segment.
2. Allow OAI-SearchBot, Claude-SearchBot and PerplexityBot unless you have a specific, documented reason not to.
3. Use Google-Extended for the Gemini training decision and leave Googlebot alone.
4. Apply your rules to every subdomain, and add a subdomain check to your launch checklist.
5. Verify agents against published IP lists and reverse DNS rather than user-agent strings.
6. Do not block AI crawlers by IP — a bot that cannot read robots.txt cannot honour it.
7. Audit the WAF, CDN and bot-management layer whenever robots.txt looks correct but logs are empty.
8. Sweep templates for nosnippet, data-nosnippet, max-snippet and noindex directives.
9. Record owner, rationale and review date as comments inside robots.txt.
10. Re-audit after every CMS migration, CDN change, security review and legal update.

The Future of AI Crawler Control

The direction of travel is more granularity, not less: providers keep splitting agents by purpose because site owners keep demanding the ability to say yes to citation and no to training. Expect more named agents, more published IP lists, and more pressure for machine-readable licensing signals that go beyond a binary Disallow. Expect too that user-initiated retrieval keeps growing, which will steadily reduce how much any robots directive actually governs. The durable practice is not a clever file — it is a documented, reviewed policy, verified at every layer, on every host.