Discussion starter: an AI search question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.
- AI Search
- Crawling
What share of AI crawler traffic in your logs is verified against published IP ranges, and how much is spoofed user agents?
Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.
1 reply
-
adminAdmin
The share varies too much between sites to give a meaningful general figure, but the method for finding yours is straightforward, and the answer is usually eye-opening. Several AI companies publish the IP ranges their crawlers use, or support reverse DNS verification, so you can split log requests that claim an AI crawler user agent into verified and unverified. Unverified requests using AI crawler names are not rare; scrapers borrow well-known user agents to look legitimate or to get past rules written for specific bots. There is also a third group that is harder to see: crawling by AI systems that do not declare themselves at all. Knowing the split matters because it decides whether your rules are controlling AI companies or just scrapers wearing their names.
How to measure it
- Collect the published ranges for each provider you care about. OpenAI, for example, publishes IP lists for its crawlers; Google publishes its crawler IP ranges, which moved to a new location in March 2026; other providers publish lists or document reverse DNS checks. Where a provider publishes nothing, treat its user agent as unverifiable.
- Refresh the lists regularly. Ranges change, and an outdated list will misclassify genuine requests.
- Join your logs to the ranges, ideally at the CDN or edge, where all requests are visible.
- Classify each request: verified (claims an AI crawler and comes from its ranges), unverified (claims an AI crawler but comes from elsewhere), and other.
- Look at behaviour for the unverified group: request rates, paths, headers and whether it respects robots.txt.
What you are likely to find
Group Typical characteristics Response Verified AI crawlers Declared user agent, published ranges, usually follow robots.txt Control through robots.txt and provider settings Spoofed AI user agents AI crawler names from hosting or residential IPs, often aggressive Block or challenge at the edge; robots.txt will not stop them User-triggered fetchers Requests made because a user asked an assistant to read a page Usually low volume; generally not governed by robots.txt in the same way as crawlers Undeclared crawling Generic browser user agents with crawler-like patterns Detect by behaviour; bot management tools help Why spoofing happens
Well-known crawler names carry an assumption of legitimacy, and some sites allow them through rules that block unknown bots. Scrapers use those names to get the same treatment. There have also been public disputes about AI companies crawling without declaring themselves: in 2025 Cloudflare said it had observed an AI answer engine using undeclared crawlers to get around blocks, which the company disputed. Whatever the specifics, the lesson for site owners is the same: a user agent is a claim, not an identity.
Hypothetical example:
A publisher sees heavy traffic from user agents claiming to be several AI crawlers and blocks them all in robots.txt. Traffic barely changes. Joining edge logs to published IP lists shows that about a third of the requests come from the providers’ real ranges and stop after the robots.txt change; the rest come from cloud hosting ranges, ignore robots.txt and continue. The publisher adds an edge rule that challenges any request claiming an AI crawler user agent from outside the provider’s published ranges. Load falls sharply, and the real crawlers’ behaviour is now controlled by robots.txt as intended.
Turning the analysis into rules
- Allow or disallow verified crawlers through robots.txt, by provider and purpose.
- At the edge, challenge or block requests that claim a crawler identity but fail verification.
- Rate-limit high-volume unverified traffic regardless of user agent.
- Keep Googlebot and Bingbot verification rules separate and carefully tested, so search crawling is never caught by AI rules.
- Review monthly: new crawlers appear, and ranges change.
What verified crawler traffic tells you
Once verified requests are separated out, they become useful data rather than noise. Which of your pages do verified AI search crawlers fetch most often? Which do user-triggered fetchers request, since those reflect questions real users asked an assistant about your content? How does that compare with your top organic landing pages? Pages that assistants read often but that receive little organic traffic may be answering questions in AI products that never become search visits, which is worth knowing when you judge a page’s value. And a sudden drop in verified search-crawler requests after a configuration change is an early warning that you may have blocked something you meant to allow.
Report the numbers internally
A simple monthly report showing verified AI crawler requests by provider, spoofed requests, and their share of total bot load helps decisions about blocking, licensing and infrastructure cost. It also puts claims about “AI companies scraping our site” on a factual footing: some of the load is the AI companies, following your rules; some of it is not them at all.
Watch for this:
Blocking by user agent alone punishes the crawlers that identify themselves honestly and does nothing to the ones that lie. Verification is what makes AI crawler rules effective.
Verify against published ranges or reverse DNS, classify requests into verified, spoofed, user-triggered and undeclared, control verified crawlers with robots.txt, handle the rest at the edge, and refresh the lists regularly. The share of spoofed traffic on your site is the number that tells you whether your current rules are doing anything at all.
Need help with this on your own site?
Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.
Add a reply Cancel reply
Replies are open to approved members. Log in or apply to join.