robots.txt file showing green allowed AI search bots like OAI-SearchBot and PerplexityBot on one side, and red blocked training crawlers like GPTBot and Bytespider on the other

Blog

robots.txt for AI Crawlers: Which Bots to Allow and Which to Block (2026)

BalochDev 7 min read

In 2026 there are two families of AI bot, and they require opposite decisions. Training crawlers scrape your content to build AI model weights — you may want to block these for IP or privacy reasons. Search and citation crawlers fetch your pages at query time to power live AI answers — blocking these makes you invisible to ChatGPT, Perplexity, and Gemini. The single most common and most damaging AI-visibility mistake is blocking citation crawlers alongside training crawlers, usually through an old "block everything" rule left in a robots.txt from 2023. CapstonAI's Q1 2026 vendor benchmark (a self-reported cohort study, not independent research) found 41% of B2B sites in its sample still block at least one major AI bot.

This is the complete configuration guide. Copy-paste blocks are at the end.

The two-family distinction you must understand first

Every major AI company now runs at least two separate crawlers with separate user-agent strings. Treating them as one is the root cause of most misconfiguration:

(Table place holder)

Three critical points from this table:

1. Google-Extended ≠ Googlebot. Blocking Google-Extended does not block your site from appearing in Google AI Overviews. AI Overviews use the standard Googlebot. Google-Extended specifically controls whether your content enters Gemini model training. Never block Googlebot under any circumstances.

2. GPTBot ≠ OAI-SearchBot. OpenAI documents three distinct crawlers: GPTBot for training, OAI-SearchBot for search indexing, and ChatGPT-User for real-time browsing. You can block GPTBot to prevent training use while still allowing OAI-SearchBot and ChatGPT-User for citation visibility.

3. Anthropic runs the same three-way split. Anthropic formally documented this structure in February 2026: ClaudeBot handles training data collection, Claude-User fetches pages when a person asks Claude a direct question, and Claude-SearchBot indexes content specifically to improve Claude's search-style answers. Blocking ClaudeBot opts out of training without affecting citation visibility — the same logic as OpenAI's split.

4. Bytespider's dominance has faded, but it still needs handling separately. HAProxy's own site-traffic analysis in late 2024 found Bytespider accounted for nearly 90% of the AI crawler traffic it saw — a widely-cited figure at the time. That picture has shifted: Cloudflare Radar data through mid-2026 shows Bytespider's share of AI crawler traffic has fallen to roughly 5–10%, while ClaudeBot has surged to become one of the largest AI crawlers on the web. What hasn't changed is Bytespider's behavior — it's still known for frequently ignoring robots.txt disallow rules, so server-level blocking (WAF, rate limiting) remains the more reliable control regardless of its current traffic share.

The strategic decision: what do you actually want?

Before writing a single directive, answer two questions:

Do you want AI to cite your content in live answers? If yes — allow the search/citation crawlers: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, Claude-User.

Do you want your content used to train AI models? If no — block the training crawlers: GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider. This is a separate, independent decision. A site can make a nuanced choice: opt out of being used to train models while remaining eligible to be cited in AI answers — block GPTBot but allow OAI-SearchBot, for example.

Most businesses should allow citation crawlers. The citation case is strong on its own logic — you cannot appear in an AI answer if the bot that builds that answer's index can't reach your pages. The training decision is a separate IP and brand policy call, worth making deliberately rather than by default.

The recommended configuration for most businesses

# BalochDev recommended robots.txt — 2026
# Goal: allow AI search citations, make deliberate decision on training

User-agent: *
Allow: /

# === SEARCH / CITATION CRAWLERS — ALLOW ===
# These power live AI answers. Blocking = invisible to that engine.

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

# === TRAINING CRAWLERS — YOUR DECISION ===
# Block these if you don't want content used for AI model training.
# Blocking does NOT affect live AI citations.

# User-agent: GPTBot
# Disallow: /

# User-agent: ClaudeBot
# Disallow: /

# User-agent: Google-Extended
# Disallow: /

# User-agent: CCBot
# Disallow: /

# === BLOCK AT SERVER LEVEL (ignores robots.txt) ===
# User-agent: Bytespider
# Disallow: /

# Sitemap reference
Sitemap: https://yourdomain.com/sitemap.xml

Uncomment the training crawler blocks if you want to opt out of training data use.

What happens if you block citation crawlers

The consequences are immediate and precise. Blocking retrieval crawlers removes your site from AI search answers immediately. If OAI-SearchBot cannot access your site, ChatGPT Search will not cite you. If PerplexityBot is blocked, your content does not appear in Perplexity answers. These aren't degraded results — you simply aren't in the citation pool.

How to check your current status

Step 1: Visit yourdomain.com/robots.txt in a browser. Search for each user-agent string above. Any Disallow: / under a citation crawler name is blocking you.

Step 2: Check your CDN. robots.txt alone isn't enough — see The 5-Minute Reachability Check for the Cloudflare layer specifically.

Step 3: Check your server logs. Allowing bots in robots.txt does not prove they are visiting. If a major citation bot stops appearing in your logs for two or more weeks, investigate — a CDN rule, WAF challenge, or stale cache can silently block a bot that robots.txt still shows as "allowed."

Step 4: Use curl to simulate a bot fetch — curl -A "GPTBot" https://yourdomain.com/robots.txt — a 403 means a server-level block that robots.txt alone can't fix.

The agentic browser caveat

Real-time agent fetches are a gray zone. When a person asks ChatGPT or Perplexity to summarize a specific URL, the tool fetches that page on the user's behalf. Providers often treat this as user-directed access rather than crawling, so standard robots.txt crawl rules may not apply the same way. Additionally, agentic browsers like ChatGPT Atlas and Perplexity Comet drive real Chrome sessions on behalf of users, sometimes sending standard Chrome signatures rather than a distinct bot user-agent. Don't try to block agent browsers by user-agent pattern alone — you risk blocking real human visitors in the process.

Frequently asked questions

Will blocking GPTBot hurt my ChatGPT citations? No. GPTBot is for training. ChatGPT Search uses OAI-SearchBot — a separate user-agent. Blocking GPTBot opts you out of training data but does not remove you from ChatGPT Search answers.

Does blocking Google-Extended affect my Google rankings? No. Google-Extended, GPTBot, and the rest are separate from Googlebot. You can block every AI training crawler and remain fully visible in Google Search.

How often should I update robots.txt? At least quarterly. The bot landscape moves fast: OpenAI split off OAI-SearchBot in 2024, Anthropic formally split off Claude-SearchBot and documented the full three-bot structure in February 2026, and crawler market share (as with Bytespider's decline and ClaudeBot's rise) shifts meaningfully within a single year.

Is robots.txt enough to block bots I don't want? For compliant bots (OpenAI, Anthropic, Google), yes — all three companies state they respect robots.txt directives. For non-compliant ones (Bytespider, some scraper networks), no — add server-level blocking via WAF rules or Cloudflare.

This guide is maintained by BalochDev, an AI-first software development studio. We build products — and the sites that sell them — to be found by both people and machines. Last updated 13 August 2026.

Sources & further reading