Discussion
  • Crawling
  • Technical SEO

AI crawler requests now exceed Googlebot on some sites. When server cost is the constraint, how do you decide which bots to allow, throttle or block?

Published 1 reply

Discussion starter: a technical SEO question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.

Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.

1 reply

  • Niraj Raut
    Niraj Raut
    Original poster

    Decide bot by bot on value against cost, using verified data. Search engine crawlers are essential and should be served well. AI search crawlers and user-triggered fetchers deserve access if the platforms send referrals, citations or visibility you value. Training crawlers return nothing directly, so they are the natural candidates to block or throttle when cost is the constraint. Unverified and spoofed agents should be blocked or challenged regardless. Before making decisions, measure requests and bandwidth per verified bot and referrals per platform, because the bots causing the most load are often not the ones you expect. And reduce the cost of serving bots at all by caching at the edge and closing crawl traps, which lowers load for every crawler, including Googlebot.

    A decision framework

    Bot type Value Default decision
    Search engine crawlers (verified) Essential for search visibility Allow; optimise serving
    AI search crawlers (verified) Citations and referrals in AI search products Allow if the platform sends value; monitor
    User-triggered fetchers A user asked for your page Allow, unless content must not be extracted
    AI training crawlers No direct return Block or throttle if cost matters; a policy decision
    Mixed-use crawlers Search plus training Use provider controls where available; weigh search value
    Unverified or spoofed agents None Block or challenge at the edge

    Measure before deciding

    1. Verify bots against published IP ranges or reverse DNS, and separate spoofed traffic.
    2. Measure requests, bandwidth and origin load per verified bot.
    3. Measure referrals and conversions from each AI platform in analytics.
    4. Identify which URLs each bot requests; crawl traps and parameter URLs often account for much of the load.

    Tools for each decision

    • robots.txt for compliant crawlers, with separate rules by user agent and purpose.
    • CDN and firewall rules for rate limiting and blocking, especially for spoofed traffic.
    • Provider controls: Cloudflare added a “Disallow AI Training” option in September 2026 aimed at mixed-use crawlers, alongside a full block option that can also remove such crawlers from search features; check what each option does before enabling it.
    • Edge caching so repeated bot requests are served cheaply without reaching the origin.
    • Crawl trap fixes such as capping calendars and controlling faceted URLs, which cut wasted requests from every crawler.

    Hypothetical example:

    A publisher’s hosting costs rise as AI crawler requests exceed Googlebot’s. Verified log analysis shows that a third of “AI crawler” traffic is spoofed, most verified AI crawling comes from two training crawlers, and a large share of all bot requests go to tag and date archive pages. The publisher blocks spoofed agents at the edge, disallows the training crawlers in robots.txt, keeps AI search crawlers and user-triggered fetchers allowed because they send measurable referrals, noindexes and caps the archive pages, and caches article pages at the edge. Bot-related origin load falls substantially, Googlebot’s crawl is unaffected, and AI referrals continue.

    Watch for this:

    Broad rules that block “all bots” or “all AI” often catch search crawlers or AI search crawlers you wanted to allow. Test rules against verified crawler requests, and monitor search crawl stats and AI referrals after every change.

    Cost per request is worth reducing anyway

    Much of the cost of serving bots comes from pages that are expensive to generate and not cached: search result pages, filtered listings, personalised pages and uncached product pages. Improving edge caching for anonymous traffic, returning lightweight responses for obviously invalid URLs, and making sure bots cannot reach expensive dynamic pages through crawlable links reduces cost per request for every visitor. Sites that do this often find that bot load stops being a budget problem before any blocking is needed, which keeps options open for AI platforms that may matter more later.

    Revisit decisions regularly

    The landscape changes: new crawlers appear, platforms change what their crawlers do, referral patterns shift and licensing opportunities emerge. Review bot policies quarterly with fresh data on load, referrals and platform changes. A training crawler you block today might be part of a licensing arrangement next year, and a search crawler that sends little now might send more later.

    Verify bots, measure load and value per bot, serve search crawlers well, allow AI search and user-triggered access where it returns value, block or throttle training crawlers when cost is the constraint, block spoofed traffic, reduce serving cost with caching and crawl trap fixes, and review the policy regularly.

    Need help with this on your own site?

    Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.

    See SEO services

Add a reply

Replies are open to approved members. Log in or apply to join.