Discussion
  • Crawling
  • Technical SEO

With CDN-level AI training blocks now touching mixed-use crawlers, how are you checking that search crawling of your site hasn’t dropped by mistake?

Published 1 reply

Discussion starter: a technical SEO question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.

Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.

1 reply

  • admin
    Admin
    Original poster

    I would treat any change to CDN bot or AI crawler settings as a deployment that can break search crawling, and verify it the same way: compare what verified search crawlers received at the edge before and after the change, confirm what Google itself reports, and diff the robots.txt that is actually being served. The difficulty is that the usual dashboards react slowly. Traffic and rankings can move days or weeks after crawling does, and if your log analysis runs on origin logs, requests blocked at the edge never reach them.

    On Cloudflare specifically, the picture as of late September 2026 is less alarming than the July announcement suggested. PPC Land reports that Cloudflare dropped its plan to block the mixed-use crawlers (Googlebot, Bingbot and Applebot) on every site that refuses AI training. New settings deployed on 15 September 2026 separate a “Disallow AI Training” option from a full “Block”, and existing training blocks were migrated to Disallow AI Training. The remaining risks are the Block option itself, custom firewall and rate-limiting rules, other vendors’ bot management, and robots.txt changes that nobody on your team deployed.

    What the new settings do to search crawlers

    According to the same report, the options sit in the dashboard’s security settings on all plans, split across Search, Training and Agent categories. The ones that matter for search crawling:

    Setting What happens to Googlebot, Bingbot and Applebot Search risk
    Allow No change None from this setting
    Disallow AI Training Cloudflare’s Bot Preference Sync writes operator tokens such as Google-Extended and Applebot-Extended into robots.txt. Training-only crawlers are blocked at the edge; mixed-use crawlers from operators Cloudflare classes as accountable can still fetch pages. Low for Google, since Google says Google-Extended does not affect inclusion or ranking in Search. But the robots.txt you serve has changed, and Bingbot does not yet act on the new directive.
    Block Mixed-use crawlers are blocked PPC Land describes this as leaving Google, Bing and Apple search altogether

    Three details from the report are worth checking against your own account. A previously disabled toggle became Allow across all three categories. New ad-supported domains default to Disallow AI Training. And Microsoft currently takes training preferences through its NOARCHIVE meta tag, with robots.txt support for a domain-level “no training” preference targeted for early 2027. That last gap creates a temptation to block Bingbot outright to stop Microsoft training, which also removes the site from Bing search and from any AI answers grounded on Bing’s index.

    Why a mistaken block hides from the usual dashboards

    1. Edge blocks never reach origin logs. If the firewall rejects Googlebot, the origin sees fewer requests, not errors. A dashboard built on origin data shows a quiet dip that looks like an ordinary fluctuation in crawl demand.
    2. Google reacts differently by status code. Google’s HTTP status code documentation says 4xx responses other than 429 tell the indexing pipeline the content does not exist, and previously indexed URLs are removed. It also says not to use 401 or 403 to limit crawl rate. A 429 or 5xx makes Google’s crawlers slow down, and indexed URLs are preserved but eventually dropped. So a bot rule that answers Googlebot with 403 does more damage than one that rate-limits it.
    3. Challenge pages can return 200. A JavaScript or CAPTCHA challenge served with a 200 looks healthy in any status-code report. The same documentation says Search Console shows a soft 404 when a page’s content suggests an error, but a challenge page may not be read that way, and either signal arrives well after the crawl.
    4. robots.txt groups do not add up. Under the robots.txt standard, a crawler follows the group that names its product token and ignores the User-agent: * group. If a generated file adds a group naming Googlebot or Bingbot, your existing wildcard rules, including the ones keeping crawlers out of faceted URLs, stop applying to that crawler.
    5. Google’s reports are aggregated and lag. The Crawl stats report counts most crawl requests but not all, and the Page indexing report confirms damage after it has happened.

    Why the edge cannot separate the two uses:

    A mixed-use crawler is one van carrying two kinds of cargo. A CDN can turn the van away or let it in, but it cannot unload the training cargo at the kerb. The separation has to happen at the operator’s depot, which is what tokens such as Google-Extended and Applebot-Extended ask the operator to do. Technically, that is why “Disallow AI Training” relies on robots.txt for mixed-use crawlers while blocking training-only crawlers outright at the edge.

    The checks I would run after any bot-setting change

    1. Diff the robots.txt being served. Fetch the live file through the public hostname, compare it with the version in your repository, and look for new groups naming Googlebot, Bingbot or Applebot, any Disallow: / under a search crawler, and broken syntax where generated lines meet your own. Then open Search Console’s robots.txt report to see which version Google last fetched, when, and whether the fetch succeeded. The report only reflects files Google has fetched, so test a draft change with a robots.txt parser before publishing it.
    2. Segment edge logs by verified crawler. Google’s crawler documentation describes verification against published IP ranges or by reverse DNS, and Bingbot can be verified by reverse DNS in the same way. Compare daily requests, the status code mix (403, 429 and 503 in particular) and median response size per template with the four weeks before the change.
    3. Read Crawl stats by crawler type. The report breaks requests down by Googlebot type, including smartphone, desktop, image, video, page resource load, AdsBot and StoreBot, and shows host status for robots.txt fetching. An allowlist that matches only the string “Googlebot” can let the main crawler through while stopping AdsBot or StoreBot, whose user agents use different tokens. A fall in page resource load requests can mean a separate asset host is being blocked, which breaks rendering rather than fetching.
    4. Run URL Inspection live tests on a sample from each template. Confirm the page fetch succeeds, then view the tested page’s HTML to check it is your page and not a challenge. One caveat: according to the same crawler documentation, URL Inspection fetches as Google-InspectionTool, not Googlebot. A rule keyed to the “Googlebot” string can treat the two differently, so a failed test is conclusive while a passing test is necessary but not sufficient.
    5. Watch the Page indexing report for “Blocked due to access forbidden (403)”, “Server error (5xx)”, “Soft 404” and “Blocked by robots.txt” on the affected templates. These confirm damage rather than warn of it.
    6. Check the crawlers you meant to keep. OpenAI documents GPTBot (training) and OAI-SearchBot (ChatGPT search) as separate crawlers, so a training block should not catch OAI-SearchBot unless leaving ChatGPT search is intended. The same logic applies to Bingbot versus Microsoft’s training preference.
    7. Annotate the change date with Search Console’s custom chart annotations, so anyone reading the Performance data later can see it.

    Triage when verified crawl requests fall

    Verified Googlebot requests fall after a bot, firewall or CDN change
    ↓
    Did the served robots.txt change? If yes, work out which group now applies to Googlebot and which version Google last fetched.
    ↓
    Do edge logs show 403, 429 or challenge responses to verified Googlebot? If yes, find the rule and add an exception for verified search crawlers that runs before it.
    ↓
    Fewer requests but normal responses? Check origin response times and error rates, then whether crawl demand changed for reasons unrelated to the CDN (site changes, update timing).
    ↓
    Confirm with URL Inspection live tests, then watch Page indexing reasons for the affected templates over the following weeks.

    Alerts worth automating

    The thresholds below are starting points to tune against your own baseline, not documented values.

    Signal Source Example alert rule Likely cause
    Verified Googlebot HTML requests per template Edge logs 30% below the trailing four-week median for the same weekday, two days running A block, a rate limit or origin slowness
    403 and 429 responses to verified search crawlers Edge logs Any sustained rise above baseline A firewall or rate-limiting rule catching search crawlers
    Median response size for Googlebot HTML Edge logs A sudden drop on one template Challenge or error pages served with a 200
    robots.txt content Hourly external fetch compared with a stored hash Any change outside a planned release A CDN-generated or managed file changed
    Host status Crawl stats Any robots.txt fetch problem flagged robots.txt returning errors to Google
    Page indexing reasons Search Console Rising 403, 5xx or soft 404 counts on key templates Damage that has already reached indexing

    Use the control built for each AI goal

    Each of the common AI goals now has a control that leaves search crawling alone. Blocking a mixed-use crawler to achieve any of them trades search visibility for something a narrower control already does.

    Goal Control Effect on search
    Keep content out of Gemini training Disallow Google-Extended in robots.txt Google says Google-Extended does not affect inclusion or ranking in Search
    Stay out of AI Overviews and AI Mode Search Console’s Search generative AI control, a site-level setting that rolled out to all sites worldwide on 31 August 2026 Google says the setting will not be used as a ranking signal outside those features; the site loses impressions and traffic from the features themselves
    Stop OpenAI training but stay in ChatGPT search Disallow GPTBot, allow OAI-SearchBot Independent crawlers, per OpenAI’s documentation
    Stop Microsoft training NOARCHIVE meta tag for now (check Bing’s current documentation for any wider effects); robots.txt support targeted for early 2027 Blocking Bingbot instead removes the site from Bing search
    Leave search engines entirely Cloudflare’s Block option, or blocking the crawler Deliberate loss of search visibility

    Watch for this:

    Vendor labels describe intent, not outcome. A setting called “training only” still changes robots.txt, and a custom firewall rule written months ago can interact with a new bot category in ways nobody tested. Verify the result in logs, whatever the label says.

    What I would do before and after the next change

    If this were my site, I would take a baseline before touching any bot setting: four weeks of verified crawler requests by template and status code from edge logs, a copy of the served robots.txt, and a record of the Crawl stats charts. Within 48 hours of the change I would diff robots.txt, check edge logs for 403, 429 and challenge responses to verified search crawlers, and run live URL Inspection tests across the main templates. At one week and four weeks I would compare Crawl stats and Page indexing reasons with the baseline. And I would choose the training-only option the vendor offers, then confirm with data that it behaves as labelled.

    Need help with this on your own site?

    Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.

    See SEO services

Add a reply

Replies are open to approved members. Log in or apply to join.