AI Crawler Robots.txt Guide: 2026 AI Search & Bot Rules

AI Crawler Robots.txt Guide: Complete Search vs Training Directives

Quick answer: To maximize AI search visibility without donating your proprietary content to foundation model training, separate search crawlers from training bots in your robots.txt. Allow search and discovery bots like Googlebot, bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot, and Meta-WebIndexer. Separately disallow training and dataset collection tokens like GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, and CCBot. Always test your configuration at the CDN and WAF level, as user-agent inheritance and server firewall rules can override robots.txt directives.

---

How to configure robots.txt for AI crawlers in 8 steps

1. Identify your business objective: decide whether you want AI search citations, model training opt-out, or total bot exclusion.
2. Separate search indexers (e.g., OAI-SearchBot, Claude-SearchBot) from model training crawlers (e.g., GPTBot, ClaudeBot).
3. Maintain unrestricted access for Googlebot and bingbot to protect organic rankings and AI Overviews/Copilot eligibility.
4. Add specific Disallow: / blocks for training tokens like Google-Extended, Applebot-Extended, meta-externalagent, and CCBot.
5. Repeat all global path restrictions (e.g., /admin/, /private/) inside every explicit User-agent block to prevent rule inheritance leakage.
6. Verify your syntax against RFC 9309 standards and never place noindex directives inside robots.txt.
7. Configure WAF and CDN firewalls using verified IP JSON feeds to prevent spoofed user-agent scraping.
8. Inspect server access logs and monitor Google Search Console and AI citation tracking dashboards to verify compliance.

---

1. What is an AI Crawler?

An AI crawler is an automated software client that visits web pages for artificial intelligence applications, including search indexation, foundation model pre-training, real-time user-directed retrieval, or synthetic data extraction.

The industry phrase "AI crawler" is often treated as a single monolith, but leading AI providers operate four distinct functional categories of bots:

1. AI Search & Indexing Crawlers: Autonomous web spiders that index content specifically to generate real-time citations and source links in conversational search engines (e.g., OAI-SearchBot for ChatGPT Search, Claude-SearchBot for Claude, PerplexityBot for Perplexity).
2. Model Training & Dataset Crawlers: Scrapers that download massive web corpora to train, fine-tune, or reinforce future foundation models (e.g., GPTBot, ClaudeBot, CCBot, Bytespider).
3. User-Requested Retrieval Fetchers: On-demand clients triggered when an end-user explicitly submits a URL or prompt asking an AI assistant to summarize or analyze a specific live page (e.g., ChatGPT-User, Claude-User, Perplexity-User).
4. Policy Control Tokens: Standalone identifiers that do not crawl independently, but instruct existing search infrastructure whether crawled assets may be used for AI model training or grounding (e.g., Google-Extended, Applebot-Extended).

Understanding this four-tier distinction is essential: blocking a training crawler (GPTBot) does not prevent your site from being cited in AI search, provided you allow the corresponding search crawler (OAI-SearchBot). For a complete system on structuring your content for citation extraction once crawlers are allowed, read our 2026 AI Citation Playbook and our foundational How to Do GEO Guide.

---

2. What is OAI-SearchBot? (ChatGPT Search vs. GPTBot)

OAI-SearchBot is OpenAI's dedicated search crawler used to discover, crawl, and index web pages specifically for ChatGPT Search (formerly SearchGPT), surfacing direct citations and source links in AI responses.

Unlike GPTBot, which is used exclusively for training foundation models, OAI-SearchBot is not used to train OpenAI's generative models. OpenAI explicitly engineered OAI-SearchBot to allow webmasters to participate in ChatGPT Search traffic without opting their intellectual property into AI model training.

Comparison: OpenAI User Agents

How to allow OAI-SearchBot while blocking GPTBot

To earn citations in ChatGPT Search while preventing OpenAI from scraping your articles for model training, add the following directives to your robots.txt:

# Allow ChatGPT Search to index and cite content
User-agent: OAI-SearchBot
Allow: /

# Block OpenAI from using content for AI model training
User-agent: GPTBot
Disallow: /

According to official OpenAI documentation (developers.openai.com/api/docs/bots), OAI-SearchBot respects standard RFC 9309 rules. If you disallow OAI-SearchBot, your site will not appear in ChatGPT Search results, though third-party navigational links discovered via Bing may occasionally surface.

---

3. OpenAI GPTBot & OAI-SearchBot Official Documentation

OpenAI publishes clear specifications for its web crawlers. Both bots can be filtered by User-Agent header and verified by cryptographic reverse DNS or published IP blocks.

1. User-Agent Headers

• OAI-SearchBot full string: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)
• GPTBot full string: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)
• ChatGPT-User full string: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ChatGPT-User/1.0; +https://openai.com/bot)

2. IP Range Verification

OpenAI publishes machine-readable CIDR IP blocks. You should use these JSON endpoints to configure your CDN firewall rules (Cloudflare, AWS CloudFront, Fastly) rather than relying solely on User-Agent headers, which can be easily forged by malicious scrapers:

• SearchBot CIDRs: https://openai.com/searchbot.json
• GPTBot CIDRs: https://openai.com/gptbot.json
• ChatGPT-User CIDRs: https://openai.com/chatgpt-user.json

3. Directory-Specific Rules for OpenAI

If you run a paywalled publication or proprietary documentation portal, you can selectively block OAI-SearchBot from premium paths while keeping your public blog crawlable:

User-agent: OAI-SearchBot
Disallow: /premium/
Disallow: /subscriber-only/
Allow: /blog/
Allow: /guides/

User-agent: GPTBot
Disallow: /

---

4. Anthropic ClaudeBot & Claude-SearchBot Documentation

Anthropic operates three distinct crawlers for Claude (support.claude.com):

1. Claude-SearchBot: The search crawler that indexes content to provide real-time web citations inside Claude chat and Claude 3.5/3.7 Sonnet web search tools.
2. ClaudeBot: The data-collection spider that harvests training corpora for Anthropic foundation models.
3. Claude-User: The user-directed fetcher that opens specific links when a user instructs Claude to "read this URL."

Anthropic Robots.txt Configuration

To allow Claude search citations while blocking model training data collection:

# Allow Claude Search Indexer
User-agent: Claude-SearchBot
Allow: /

# Block Anthropic Model Training Scraper
User-agent: ClaudeBot
Disallow: /

# User-directed live fetching (Optional: Disallow if paywalled)
User-agent: Claude-User
Allow: /

Anthropic Crawl-Delay Support

Unlike Googlebot (which ignores Crawl-delay), Anthropic officially supports the non-standard Crawl-delay directive. If Claude-SearchBot causes server load spikes, you can throttle request intervals:

User-agent: Claude-SearchBot
Crawl-delay: 5

Anthropic also publishes its IP ranges and respects standard RFC 9309 rules. Changes to your robots.txt are generally picked up within 24 to 48 hours.

---

5. Googlebot vs. Google-Extended (The AI Overview Truth)

The most frequent mistake in Generative Engine Optimization (GEO) is blocking Google-Extended under the false belief that it disables Google AI Overviews.

Googlebot (Core Search & AI Overviews)

Google officially states that AI Overviews and Google AI Mode are built directly into Google Search. Their crawling and indexing eligibility is governed strictly by Googlebot (Google Crawler Documentation).

If you block Googlebot:

# DANGER: This removes your site from both classic Google Search AND AI Overviews
User-agent: Googlebot
Disallow: /

Your pages will drop out of Google's organic index entirely, destroying both traditional SERP rankings and AI citations.

Google-Extended (Gemini & Vertex AI Control)

Google-Extended is a policy token used by webmasters to control whether their content may be used for:

• Training Google's foundation models (Gemini series)
• Grounding in standalone Gemini Apps (outside of Google Search)
• Grounding with Vertex AI Search & Conversation APIs

Google explicitly documents that Google-Extended has zero impact on Google Search indexing, organic rankings, or eligibility for AI Overviews. It does not have an independent HTTP crawler string; it is a policy flag respected by Google's systems.

To opt out of Gemini training while keeping full Google Search and AI Overview visibility:

# Safe: Opts out of Gemini training without affecting Google Search or AI Overviews
User-agent: Google-Extended
Disallow: /

---

6. Microsoft Bingbot & Copilot

Microsoft Copilot is grounded directly on the Bing search index. Microsoft's Webmaster Guidelines confirm that bingbot is the primary crawler for both Bing search and Microsoft Copilot grounding (Bing Webmaster Guidelines).

To maintain Copilot citation visibility, ensure bingbot has unrestricted crawl access:

User-agent: bingbot
Allow: /

Bing Advanced Preview & Snippet Directives

If you wish to remain indexed on Bing but restrict how Copilot extracts or summarizes your content, use HTML meta tags and HTTP headers rather than robots.txt:

• NOINDEX: Prevents page indexing in Bing and Copilot.
• NOARCHIVE: Prevents cached storage and Copilot training use.
• NOCACHE: Limits Copilot to title, URL, and snippet preview.
• NOSNIPPET or data-nosnippet: Prevents text excerpts from appearing in search or Copilot summaries.

---

7. Perplexity, Meta, Apple & ByteDance AI Bots

Perplexity (PerplexityBot & Perplexity-User)

Perplexity operates PerplexityBot for search indexation and Perplexity-User for user-directed queries (docs.perplexity.ai). Perplexity states that PerplexityBot is purely for search indexing and does not train foundational weights.

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

Note on WAF security: In August 2025, Cloudflare reported instances of undeclared scraping activity attributed to Perplexity infrastructure bypassing robots files. Always combine robots.txt with WAF rate limiting and reverse IP verification.

Meta AI (Meta-WebIndexer & meta-externalagent)

Meta categorizes its crawlers into distinct roles:

• Meta-WebIndexer: Indexes public content to power Meta AI web search across WhatsApp, Instagram, and Facebook.
• meta-externalagent: Collects web data for training Meta's Llama models.
• meta-externalfetcher: Performs user-requested link summaries.

# Allow Meta AI Search citations
User-agent: meta-webindexer
Allow: /

# Block Meta Llama model training
User-agent: meta-externalagent
Disallow: /

Applebot & Applebot-Extended

Applebot powers search in Apple Intelligence, Siri, and Safari Spotlight. Applebot-Extended is the dedicated training policy token:

# Retain Apple Intelligence search citations, opt out of training
User-agent: Applebot-Extended
Disallow: /

ByteDance (Bytespider) & Common Crawl (CCBot)

Bytespider is ByteDance's aggressive scraper used for Doubao and AI training. CCBot is Common Crawl's open-source dataset harvester. Neither provides direct search traffic to western websites:

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

---

8. Complete 2026 AI Crawlers & AI Bots Directory

This master directory contains the verified user-agent tokens, primary purposes, and official documentation links for all major AI web clients:

---

9. 5 Copy-Paste Robots.txt Recipes for 2026

Recipe 1: Maximum AI Search & Citations + 100% Training Block (Recommended)

This configuration gives your brand maximum visibility in Google AI Overviews, ChatGPT Search, Claude, Perplexity, Copilot, and Meta AI while prohibiting all foundation model training scraping.

# Baseline rules for all standard crawlers
User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /checkout/
Disallow: /private/

# Explicitly allow AI Search Indexers
User-agent: OAI-SearchBot
Disallow: /admin/
Disallow: /api/
Disallow: /checkout/
Disallow: /private/
Allow: /

User-agent: Claude-SearchBot
Disallow: /admin/
Disallow: /api/
Disallow: /checkout/
Disallow: /private/
Allow: /

User-agent: PerplexityBot
Disallow: /admin/
Disallow: /api/
Disallow: /checkout/
Disallow: /private/
Allow: /

User-agent: meta-webindexer
Disallow: /admin/
Disallow: /api/
Disallow: /checkout/
Disallow: /private/
Allow: /

# Block AI Foundation Model Training Scrapers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

# XML Sitemap declaration
Sitemap: https://imvasa.dev/sitemap.xml

Recipe 2: Full Open Indexing (Allow Search and AI Training)

Use this recipe if you want unrestricted distribution and actively want your content included in training datasets and AI knowledge bases.

User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /checkout/
Allow: /

Sitemap: https://imvasa.dev/sitemap.xml

Recipe 3: Total AI Bot Lockdown (Block All AI Search and Training)

Use this if you run a proprietary subscription or data business and do not want any AI company indexing or summarizing your content, while preserving classic Google and Bing blue-link rankings.

User-agent: *
Disallow: /admin/
Disallow: /api/

# Block AI Search Bots
User-agent: OAI-SearchBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: Claude-SearchBot
Disallow: /

User-agent: Claude-User
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Perplexity-User
Disallow: /

User-agent: meta-webindexer
Disallow: /

# Block AI Training Bots
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

Sitemap: https://imvasa.dev/sitemap.xml

Recipe 4: Directory-Level Research Protection

Allow AI search engines to index your general marketing pages and blog, but block AI crawlers from accessing proprietary whitepapers, datasets, or subscriber research.

User-agent: *
Disallow: /admin/

User-agent: GPTBot
Disallow: /research-reports/
Disallow: /datasets/

User-agent: ClaudeBot
Disallow: /research-reports/
Disallow: /datasets/

User-agent: Google-Extended
Disallow: /research-reports/
Disallow: /datasets/

User-agent: OAI-SearchBot
Disallow: /research-reports/
Disallow: /datasets/
Allow: /

Sitemap: https://imvasa.dev/sitemap.xml

---

10. How Robots.txt Matching Works (RFC 9309 Rules)

To avoid critical indexing leaks, you must follow the formal Robots Exclusion Protocol standardized under RFC 9309:

1. The Specificity Inheritance Trap

Crawlers evaluate robots.txt by finding the most specific matching User-agent block. Once a bot matches a specific block, it completely ignores the User-agent: * block.

# FLAWED CONFIGURATION:
User-agent: *
Disallow: /admin/
Disallow: /private-database/

User-agent: OAI-SearchBot
Allow: /

In this flawed configuration, OAI-SearchBot matches User-agent: OAI-SearchBot and completely ignores the /admin/ and /private-database/ restrictions. To fix this, always repeat your baseline private directories in every specific user-agent declaration.

2. Path Matching and Character Length

RFC 9309 specifies that when conflicting Allow and Disallow directives match a URL, the rule with the longest character path length wins.

User-agent: *
Disallow: /guides/
Allow: /guides/ai-crawler-robots-txt-guide

Here, /guides/ai-crawler-robots-txt-guide is 38 characters long, while /guides/ is 8 characters long. The longer Allow rule takes precedence.

3. Case Sensitivity & Host Boundaries

• Paths are strictly case-sensitive. Disallow: /Private/ will not block requests to /private/.
• robots.txt applies strictly to the exact protocol and subdomain where it is hosted. A file at https://example.com/robots.txt does not govern https://subdomain.example.com/ or http://example.com/.

4. The Noindex Trap

Never add Noindex: /path/ inside robots.txt. Google and RFC 9309 do not support noindex directives in robots files. If you block a URL in robots.txt, crawlers cannot access the page to read HTML meta tags or HTTP X-Robots-Tag: noindex headers, meaning the URL may still be indexed and surfaced as an anchor-text-only result.

---

11. Testing, IP Verification & WAF Security

Step 1: Live Status Header Test

Verify that your robots.txt returns an unredirected 200 OK status with Content-Type: text/plain. You can run an instant diagnostic with our Free Technical SEO Audit Scanner or follow our 12-Step Technical SEO Audit Checklist to verify server response headers, status codes, and bot accessibility:

curl -I https://imvasa.dev/robots.txt

Step 2: Test Specific User-Agent Emulation

Test how your CDN edge or origin server responds when an AI bot requests a page:

# Test OAI-SearchBot response
curl -I -A "OAI-SearchBot" https://imvasa.dev/guides/ai-crawler-robots-txt-guide

# Test GPTBot response
curl -I -A "GPTBot" https://imvasa.dev/guides/ai-crawler-robots-txt-guide

Step 3: Implement IP Whitelisting on Cloudflare / AWS WAF

Because user-agent strings are trivially spoofed, implement firewall rules that match both the User-Agent header and the provider's verified ASN or IP ranges:

• Cloudflare Managed Rules: Enable Cloudflare's native "AI Scrapers and Crawlers" managed rule set.
• Reverse DNS Verification: For Googlebot and Bingbot, verify that incoming requests resolve to *.googlebot.com, *.google.com, or *.search.msn.com.

---

Frequently asked questions

What is OAI-SearchBot?

OAI-SearchBot is OpenAI's official search crawler designed to crawl and index web pages for ChatGPT Search. It is separate from GPTBot, does not train foundation models, and is required if you want your website cited with direct clickable source links in ChatGPT responses.

What is the difference between GPTBot and OAI-SearchBot in robots.txt?

GPTBot is OpenAI's model training crawler used to collect datasets for training generative AI models. OAI-SearchBot is OpenAI's search engine crawler used to index web pages for real-time citations in ChatGPT. You can block GPTBot while allowing OAI-SearchBot to earn search traffic without training models.

Where can I find the official OpenAI OAI-SearchBot documentation?

Official documentation for OpenAI crawlers is available at developers.openai.com/api/docs/bots. OpenAI also publishes verified IP addresses for OAI-SearchBot at https://openai.com/searchbot.json and GPTBot at https://openai.com/gptbot.json.

How do I configure Anthropic ClaudeBot in robots.txt?

To block Anthropic's training crawler while maintaining search visibility, declare User-agent: Claude-SearchBot with Allow: / and User-agent: ClaudeBot with Disallow: /. Documentation is available at support.claude.com.

Does blocking Google-Extended hurt my Google search rankings?

No. Google has stated on record that Google-Extended is purely a training and Gemini Apps grounding control. It does not affect Google Search crawling, organic rankings, or eligibility for Google AI Overviews and AI Mode.

What is the best robots.txt configuration for GEO (Generative Engine Optimization)?

The optimal GEO configuration allows all search indexing crawlers (Googlebot, bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot, Meta-WebIndexer) while blocking commercial model-training scrapers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, CCBot, Bytespider).

Can I use Crawl-delay for AI crawlers?

Anthropic (Claude-SearchBot), Common Crawl (CCBot), and Amazonbot support the Crawl-delay directive. Googlebot and Bingbot do not support Crawl-delay in robots.txt; use your CDN rate limiter or Google/Bing webmaster tools to manage crawl rates.

Does robots.txt guarantee that AI bots won't scrape my content?

No. robots.txt is an advisory protocol respected by reputable companies (OpenAI, Google, Microsoft, Anthropic). Rogue scrapers and third-party bots often ignore it. For complete protection of sensitive data, use server authentication, session tokens, or WAF IP filtering.

---

Official Sources & Documentation

1. RFC 9309: Robots Exclusion Protocol: https://www.rfc-editor.org/rfc/rfc9309.html
2. OpenAI Official Crawler Documentation: https://developers.openai.com/api/docs/bots
3. OpenAI OAI-SearchBot IP Feed: https://openai.com/searchbot.json
4. OpenAI GPTBot IP Feed: https://openai.com/gptbot.json
5. Anthropic Web Crawler Documentation: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
6. Google Search Central: AI Features and Your Website: https://developers.google.com/search/docs/appearance/ai-features
7. Google Search Central: Overview of Google Crawlers: https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
8. Microsoft Bing Webmaster Guidelines: https://www.bing.com/webmasters/help/webmaster-guidelines-30fba23a
9. Perplexity Crawler Specifications: https://docs.perplexity.ai/docs/resources/perplexity-crawlers
10. Meta Web Crawlers & User-Agents: https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers
11. Applebot & Applebot-Extended Guidelines: https://support.apple.com/en-us/119829
12. Common Crawl CCBot Specifications: https://commoncrawl.org/faq