CDNs and Cloudflare
A content delivery network (CDN) serves your site from servers close to each visitor, making it faster and more resilient. Cloudflare is the most widely used, and its caching, security and bot settings can help or quietly harm both SEO and AI visibility.
  • A CDN mainly helps SEO through faster responses (TTFB and LCP) and better uptime.
  • Check that Bot Fight Mode and firewall rules never block or challenge Googlebot or Bingbot.
  • Cloudflare has blocked AI crawlers by default on new sites since July 2025. Decide deliberately which ones you allow.
  • Use the CDN's logs, not your origin's, to see what crawlers actually received.
  • Test optimisation features such as Rocket Loader before trusting them with your rendering.

A CDN is a network of servers around the world that store copies of your pages and files and serve them from the location nearest each visitor. Cloudflare also sits in front of your site as a reverse proxy, so every request, from people and bots alike, passes through it first. That makes it powerful for performance and security, and it means its settings decide what search engines and AI crawlers are allowed to see.

How a CDN helps SEO

  • Speed: cached responses cut Time to First Byte, which feeds directly into Largest Contentful Paint.
  • Crawl capacity: fast, healthy responses encourage Googlebot to crawl more. Slow or erroring origins make it back off.
  • Resilience: traffic spikes and attacks are absorbed before they reach your server, so pages stay up.
  • Modern protocols: HTTP/2, HTTP/3, TLS 1.3 and Brotli compression come largely for free.

Caching sensibly

  • Cache static assets (images, CSS, JavaScript, fonts) for long periods, with versioned file names.
  • Cache HTML where you can, using cache rules and purging on publish. Cloudflare only caches HTML if you tell it to.
  • Never cache personalised, logged-in or basket pages.
  • Make sure redirects, 404s and canonical headers are cached correctly, not frozen in an outdated state.

The Cloudflare settings that matter most

Bot Fight Mode and firewall rules

The free Bot Fight Mode challenges traffic that looks automated. Verified bots such as Googlebot and Bingbot are meant to pass, but aggressive custom rules, country blocks or rate limits can still catch them. A crawler served a challenge page or a 403 sees no content at all. Review Security Events for blocked verified bots and add skip rules where needed.

AI crawler controls

Since July 2025, new Cloudflare domains block known AI crawlers by default, and existing sites can toggle the same "Block AI bots" setting. Cloudflare can also manage your robots.txt for you, adding AI rules and its Content Signals policy, which lets you state separately whether content may be used for search, for AI answers (ai-input) or for training (ai-train). Cloudflare has also announced stricter defaults from 15 September 2026, blocking mixed-use crawlers on pages that carry ads unless you change the setting.

The catch is that "AI bots" is not one thing. Blocking training crawlers such as GPTBot or CCBot is a content decision. Blocking search and user-triggered agents such as OAI-SearchBot, ChatGPT-User, Claude-SearchBot or PerplexityBot removes you from those assistants' answers. Check exactly which agents your setup blocks, and remember Cloudflare can override what your own robots.txt says.

Crawler Hints

Crawler Hints tells participating search engines via IndexNow when content changes, which helps Bing (and the assistants built on it) pick up updates quickly. It is worth switching on.

Optimisation features

  • Rocket Loader defers JavaScript and can break rendering or analytics. Test before enabling.
  • Email address obfuscation rewrites HTML and injects a script. Harmless for most sites, but it shows up in rendered source.
  • Early Hints and image optimisation (Polish, resizing) can improve LCP.
  • Always Online serves a cached copy when your origin is down, which is better than an error.

Common mistakes

  • Googlebot or Bingbot hitting challenge pages after a new security rule.
  • Leaving AI blocking on by default when you want to be cited by AI search.
  • A robots.txt managed in two places, with Cloudflare's version silently winning.
  • Stale cached HTML still showing old titles, canonicals or noindex tags after a fix.
  • Cached redirects from a past migration that no longer apply.
  • Analysing origin logs only, when most requests never reach the origin.

How to test

  • URL Inspection in Search Console: confirm Google gets a 200 and the real content, not a challenge page.
  • Fetch robots.txt in a browser to see the version Cloudflare actually serves.
  • Security Events and the AI Crawl Control dashboard show which bots were blocked, challenged or allowed.
  • curl -I shows the cf-cache-status header (HIT, MISS, DYNAMIC) for caching checks.
  • Ask an assistant to read a page on your site to confirm its live fetcher gets through.

The GEO angle

For many sites, Cloudflare is now the single biggest factor in AI visibility, because it decides at the edge which AI agents get in. A site can have perfect content and still be invisible to ChatGPT or Perplexity because a default it never chose is turned on. Make the allow and block decision explicitly, write it down, and recheck it whenever Cloudflare changes its defaults.

Related guides