Cloudflare edge network diagram showing GPTBot, ClaudeBot, and PerplexityBot blocked in red on one side, while OAI-SearchBot and PerplexityBot pass through allowed in green on the other

Blog

Is Cloudflare Blocking AI Bots From Your Site? How to Check and Fix It (2026)

BalochDev 7 min read

Since 1 July 2025, Cloudflare has let every customer — including the free tier — block AI crawlers with a single toggle, and new domains onboarding after that date have gotten a version of that block by default. Cloudflare sits in front of roughly 20% of all websites, so this is one of the most widespread silent AI-visibility killers on the web. A site can rank on Google's first page, publish excellent content, and implement every GEO best practice — and still be completely invisible to ChatGPT, Perplexity, and Claude because the block happens at the CDN, not at the page level. Your Google Search Console will show no crawl errors. Your robots.txt will look correct. You won't know unless you check.

This is the check, and the fix.

Why Cloudflare is a special case

robots.txt is a permission file — it tells crawlers what they may access. But a permission file only works if the crawler can reach your server to read it. Cloudflare sits between the crawler and your server. If Cloudflare blocks the crawler, your robots.txt is never consulted, your content is never fetched, and no citation is possible. The block is invisible from the inside.

One clarification worth getting right, since it's easy to conflate: the feature responsible for this is Cloudflare's "Block AI bots" managed rule, part of its broader bot-management suite — not "Bot Fight Mode," a separate, older, generic bot-challenge feature that targets obvious automated traffic and doesn't specifically distinguish AI crawlers. Since July 1, 2025, "Block AI bots" has been available to every customer, including Free plans, and Cloudflare enabled it as the default for newly onboarding domains at that time.

That default is changing again. Cloudflare announced in July 2026 that starting September 15, 2026, newly onboarded domains will sort AI traffic into three categories — Search, Agent, and Training — with Training and Agent bots blocked by default on pages that display ads, while Search crawlers remain allowed by default. If your domain predates September 2026, you may still be running under the older, blunter default. Check now.

The two-minute check

Method 1 — curl simulation:

curl -I -A "OAI-SearchBot" https://yourdomain.com
curl -I -A "PerplexityBot" https://yourdomain.com
curl -I -A "ClaudeBot" https://yourdomain.com

A 403 Forbidden response means Cloudflare is blocking that bot at the edge. A 200 OK means it's getting through.

Method 2 — Cloudflare dashboard: Log into Cloudflare, pick your zone, navigate to Security → Bots. Look for "Block AI bots" in the configuration panel. If the toggle is on, every bot in the list gets a hard block at the edge.

Then navigate to AI Crawl Control — Cloudflare's tool for monitoring and managing AI bot access, rebranded from its earlier beta name "AI Audit" when it reached general availability in August 2025. The Crawlers tab shows a table of which AI services have requested your content, broken down by user agent, robots.txt compliance, and which sections of your site they hit. If the table is empty and you've been live for weeks, the block is shutting the door before any crawler makes it in.

Method 3 — Server logs: Search your access logs for OAI-SearchBot, PerplexityBot, ClaudeBot, GPTBot. If none appear, they're being blocked before they reach your origin.

The fix: three-layer configuration

Cloudflare now sorts AI crawlers into three categories. Understanding them prevents the mistake that breaks regular Google rankings:

  • Search — crawls to power AI citations (OAI-SearchBot, PerplexityBot, Claude-SearchBot). Allow these.

  • Agent — fetches pages in real time for user-triggered AI actions. Allow these.

  • Training — scrapes content to train models (GPTBot, ClaudeBot, CCBot). Your decision.

One critical warning: multi-purpose crawlers that serve both Search and Training functions can get caught by category-level blocking rules in ways that aren't always obvious. If you block the Training category broadly via Cloudflare's category controls, double-check that you haven't inadvertently affected a crawler your site depends on for standard search visibility. This is the footgun that can quietly break rankings. Use the per-bot WAF rules below rather than the broad Training category block if you want precision.

The recommended Cloudflare configuration:

Step 1 — Turn off the blanket "Block AI bots" toggle in Security → Bots if it's enabled.

Step 2 — In AI Crawl Control, set Search crawlers to Allow and Agent crawlers to Allow. Set Training crawlers based on your policy.

Step 3 — If you want granular control, add WAF custom rules. Example to allow OAI-SearchBot explicitly:

Rule: (http.user_agent contains "OAI-SearchBot") → Action: Allow
Repeat for PerplexityBot, Claude-SearchBot, ChatGPT-User

Step 4 — Verify with curl after saving. Allow up to 5 minutes for rules to propagate globally.

The Cloudflare robots.txt override

Verify that "Manage your robots.txt" inside Cloudflare is disabled so your origin server's file takes precedence. Cloudflare can serve its own robots.txt override from the edge, which may differ from what's on your server. Check that the file your crawlers actually see matches what you intended.

What about Cloudflare's pay-per-crawl?

Cloudflare's AI Crawl Control includes pay-per-crawl functionality, which lets you charge AI crawlers per request instead of blocking them outright — using the HTTP 402 "Payment Required" status code as the mechanism. It's moved from private beta toward broader availability through 2026, with Cloudflare reporting more than a billion AI-crawler 402 responses served daily across its network. Still, for most small and mid-sized businesses, allowing AI search crawlers freely remains the right call while the monetization tooling and market norms continue to mature.

Frequently asked questions

My Google rankings are fine — why would Cloudflare be a problem? Googlebot is typically handled differently from AI crawlers in Cloudflare's default rule sets, and Google's crawler behavior (including how patiently it renders JavaScript) differs from most AI crawlers. Your ranking data can look clean while AI crawlers see nothing — these are separate visibility problems with separate causes.

Does the Cloudflare block affect Google AI Overviews? Yes and no. Google AI Overviews primarily use standard Googlebot, which is typically not caught by the AI bot blocking rules. But double-check your WAF rules aren't inadvertently catching Googlebot-adjacent traffic.

Is this problem specific to Cloudflare? No — any CDN, WAF, or DDoS protection layer can block AI crawlers. Fastly, AWS CloudFront, and Akamai all have similar bot-management features that need the same review. Cloudflare is just the most widely deployed, sitting in front of roughly a fifth of the web, and has one of the most impactful defaults.

How do I know if my site was affected by the default AI-bot-blocking change? Check your AI Crawl Control dashboard. If OAI-SearchBot, PerplexityBot, and ClaudeBot show zero requests in the past 30 days despite your site being live and indexable, a default block is the most likely cause.

This guide is maintained by BalochDev, an AI-first software development studio. We build products — and the sites that sell them — to be found by both people and machines. Last updated 13 August 2026.

Sources & further reading