Bot Verification and AI Bot Governance
User agent strings are easy to fake, so real crawlers should be verified by reverse DNS, published IP ranges or cryptographic signatures. Bot governance is deciding which verified bots get access, and making sure your robots.txt, CDN and firewall all agree.
  • Never trust a user agent string on its own. Verify with reverse and forward DNS, or match the IP against the operator's published ranges.
  • Google, Bing, OpenAI and Anthropic publish verification details. Refresh IP lists automatically.
  • Web Bot Auth lets bots and agents sign requests (RFC 9421), and ChatGPT agent already does.
  • Group bots into training, search, user-triggered and agent traffic, then decide access per group.
  • Your CDN may be blocking AI crawlers by default. Check it matches your robots.txt.

Anyone can send a request that claims to be Googlebot or GPTBot. Scrapers do it all the time to slip past rate limits, and the reverse problem is just as common: genuine AI crawlers being blocked by security rules nobody remembers setting. Verification tells you who is really knocking. Governance is the policy that decides who gets in, applied consistently across robots.txt, your CDN and your firewall.

Why the user agent is not enough

The user agent is a header the client writes about itself. It is useful for reporting and for robots.txt, which relies on honest bots, but it proves nothing. Log analysis based on user agents alone will overcount Googlebot and miscount AI crawlers, and allowlists keyed on user agents let spoofed traffic straight through.

Method 1: reverse and forward DNS

Look up the hostname for the requesting IP, check that it belongs to the operator's domain, then look up that hostname and confirm it returns the original IP. Google documents this in verifying Google crawlers and fetchers:

 host 66.249.66.1
# 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
host crawl-66-249-66-1.googlebot.com
# crawl-66-249-66-1.googlebot.com has address 66.249.66.1

Valid Google hostnames end in googlebot.com, google.com or googleusercontent.com, depending on the type of crawler or fetcher. Bing uses search.msn.com for Bingbot. DNS checks are fine for spot checks but slow at log scale.

Method 2: published IP ranges

Most major operators publish machine-readable IP lists, which are faster to match in bulk:

  • Google publishes separate JSON files for common crawlers such as Googlebot, special-case crawlers such as AdsBot, user-triggered fetchers and user-triggered agents.
  • OpenAI publishes one file per bot, covering GPTBot, OAI-SearchBot and ChatGPT-User, linked from its crawler overview.
  • Anthropic lists its crawlers and source IP information in its crawler help article.
  • Bing, Perplexity and others publish ranges in their webmaster or crawler documentation.

These lists change, so fetch them on a schedule rather than copying them into a config file once.

Method 3: cryptographic signatures

Web Bot Auth is a newer approach, built on RFC 9421 HTTP Message Signatures and being developed through an IETF draft whose authors come from Cloudflare and Google. The bot signs each request with a private key and sends Signature, Signature-Input and Signature-Agent headers. The public keys sit at a well-known path on the operator's domain, so any site can check the signature without trusting an IP address. Agents running in shared cloud browsers, which have no stable IPs, can finally prove who they are.

OpenAI's ChatGPT agent allowlisting guide confirms that ChatGPT agent signs its requests this way, and Cloudflare lists it in its directory of signed agents. Most sites will not verify signatures themselves. Their CDN or bot management product does it for them.

Governance: decide by category

Lumping every AI bot together leads to bad decisions. Group them by what the fetched content is used for:

  • Training crawlers such as GPTBot and ClaudeBot collect content that may be used to train models. Blocking them has no immediate visibility cost, but it is a long-term bet about how models will know your brand.
  • Search and index crawlers such as OAI-SearchBot and Claude-SearchBot build the indexes that AI search answers draw on. Block these and you will generally drop out of that assistant's search results.
  • User-triggered fetchers such as ChatGPT-User and Claude-User fetch a page because a person asked about it. Operators note that robots.txt may not apply to these in the same way, since a human made the request.
  • Agents browse, click and fill in forms on a user's behalf, sometimes inside the user's own browser.

Two details trip people up. Google-Extended controls whether content is used for Gemini models and grounding, but it does not affect Google Search, so it does not remove you from AI Overviews. And Anthropic's documentation warns that blocking its IP addresses is not a reliable opt-out, because it stops the crawler reading the robots.txt rule you actually want it to follow.

The CDN layer

Since July 2025, new Cloudflare domains have been asked about AI crawler access during setup, with blocking as the default, and existing customers can block in one click. Cloudflare's AI Crawl Control shows which crawlers visit and lets you allow or block each one, and its pay per crawl beta can return HTTP 402 to bots that have not agreed terms. Its managed robots.txt can also add the Content Signals Policy, which expresses preferences for three uses: search, ai-input and ai-train.

 User-Agent: *
Content-Signal: search=yes, ai-train=no
Allow: /

Content signals are a statement of preference, not enforcement, and compliance is voluntary. Akamai, Fastly, Vercel and others offer verified bot directories of their own. Whatever you use, the point is consistency: your CDN should not quietly block what your robots.txt allows.

Write a bot policy

A one-page policy saves endless debate. For each category, record whether it is allowed, which user agents are covered, where the rule is enforced (robots.txt, CDN, firewall) and who owns the decision. Review it every quarter, because new bots appear constantly.

Common mistakes

  • A firewall or CDN rule that blocks crawlers the robots.txt allows.
  • Allowlists matched on user agent alone, which let spoofed bots in.
  • Hard-coded IP lists that went stale months ago.
  • Rules on the main domain but not on subdomains that host key content.
  • Challenge pages or CAPTCHAs served to verified search crawlers.
  • Blocking Google-Extended and expecting AI Overviews to disappear.

How to test

  • Segment log files by verified and unverified traffic for each bot. The gap shows how much spoofing you get.
  • Check the status codes verified AI bots receive. A high rate of 403s or challenges means something is blocking them.
  • Review your CDN's bot analytics and its list of blocked crawlers.
  • Fetch robots.txt from each subdomain and compare it with the CDN settings.

The GEO angle

You cannot be cited by an assistant that cannot fetch your pages, and you cannot measure AI visibility from logs full of impostors. Verified, category-based access rules let you stay visible in AI search and answers while making a deliberate choice about training, instead of leaving it to a security default.

Further reading

Related guides