Discussion
  • Crawling
  • Technical SEO

Does serving proper 304 Not Modified responses measurably increase crawling of fresh URLs on large sites, or is the gain too small to justify the dev work?

Published 1 reply

Discussion starter: a technical SEO question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.

Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.

1 reply

  • admin
    Admin
    Original poster

    There is little public evidence that 304 responses on their own increase crawling of new URLs, and Google does not promise that they will. Google documents 304 support as a way to save your server’s bandwidth and resources. Any gain in fresh URL crawling is indirect, and it only appears when two things are true at once: Google already wants to crawl more of your site than it does, and your server’s speed is what holds it back.

    On a large site with slow, uncached HTML and a high share of recrawls on pages that have not changed, that combination is plausible and the work can pay for itself. On a site that serves cached HTML quickly, or where new URLs are crawled late because they are hard to find, 304s are mostly a bandwidth saving. You can tell which kind of site you have from logs and the Crawl stats report before anyone writes code. The dev cost varies just as widely, from a configuration change to a real engineering project, and the difference comes from how your stack decides that a page has not changed.

    What Google documents, and what it does not promise

    Google’s crawl budget guide for large sites lists 304 support among its crawl efficiency practices: when a page has not changed since Google last crawled it, a 304 tells Google to reuse the version it already has, which saves your server bandwidth and resources. Notice where the benefit lands. The sentence describes savings on your side, not extra crawling.

    The route from savings to more crawling runs through what the same guide calls the crawl capacity limit. Google describes it as a cap on the total time your server spends holding connections open for Google, counting both the number of parallel connections and how long they last. If the site responds consistently and response times stay stable or improve, the limit goes up. If it slows down or returns server errors or rate-limiting signals, the limit goes down. The guide also says that if Google can load and render your pages faster, it might be able to read more content from your site. That “might” sits alongside crawl demand, which the guide ties to perceived inventory, popularity and staleness. Faster responses raise the ceiling; they do not create demand.

    Google’s overview of its crawlers and fetchers covers the implementation side. Its crawling infrastructure supports ETag with If-None-Match, and Last-Modified with If-Modified-Since. Individual crawlers and fetchers may or may not use caching, depending on the product they serve: Googlebot supports it when recrawling, while Storebot-Google only supports it in certain conditions. For Google’s crawlers, Google recommends ETag over Last-Modified because ETags avoid date formatting problems.

    Important distinction:

    A 304 does not remove a crawl request. Googlebot still asks for the URL; the answer is shorter and cheaper to produce. New URLs only gain if the time saved lets Google raise its capacity limit and Google has unmet demand to spend it on.

    Capacity-bound or demand-bound: the check that decides the value

    Picture Googlebot as a courier with a set amount of time at your depot each day. A 304 is a quick “nothing new here” at the door instead of unloading a full parcel. If the courier’s day is cut short because your depot is slow, quicker stops leave time for more addresses, including new ones. If the courier already leaves early because there are only so many addresses it cares about, quicker stops change nothing. In technical terms, 304s can help a capacity-bound site and do little for a demand-bound one.

    Evidence to inspect Points to capacity-bound (304s can help) Points to demand-bound (304s will not move discovery)
    Crawl stats: host status Server connectivity problems, or crawl requests falling when response times rise Clean host status, crawl requests steady whatever the response time
    Crawl stats: average response time High for HTML and rising under load Low and flat, with HTML served from cache
    Crawl stats: By purpose Refresh requests dominate while new URLs wait days for their first visit Discovery requests arrive soon after publication
    Logs joined to CMS change dates A large share of refresh crawls land on pages unchanged since Googlebot’s previous visit Most recrawled pages have changed anyway (prices, stock, listings), so they would return 200
    Where new URLs sit Linked from frequently crawled hubs and listed in accurate sitemaps, yet still crawled late Deep in pagination, orphaned, or buried among parameter and filter URLs

    The last row matters most for this question. If new URLs are hard to reach, the constraint is the path to them, not the server. Consolidating duplicate URLs, fixing soft 404s and keeping sitemaps and their lastmod values current all appear on the same Google list, and linking new items from pages Googlebot already visits frequently is the practitioner addition. Together they work on the route to new URLs and on wasted crawling, which 304s do not touch.

    Where the dev work actually goes

    The cost of “supporting 304s” splits along three setups.

    • HTML already cached at the CDN or web server. Validators may already exist. The work is making sure ETag or Last-Modified headers survive the edge and reach the crawler, and that conditional requests get a 304 instead of a full 200. This can be a small change.
    • Dynamic pages rendered per request. Hashing the finished HTML to build an ETag saves bandwidth but not server work, because the page is still rendered before the comparison. It also fails if the HTML contains anything that changes per request (tokens, timestamps, rotating recommendations): the hash never matches, so no 304 is ever sent. The version that saves server time computes the validator before rendering, from a content version number or a trustworthy last-changed timestamp.
    • Several servers behind a load balancer. ETags built from server-local file metadata can differ between machines, so a conditional request only matches when it happens to reach the machine that issued the ETag, and you carry the cost without getting most of the 304s.

    The expensive failure is a false 304:

    If the validator ignores something that changed, Google keeps the old version. On a product page that could be a stale price or stock status. On a category or hub page it works directly against your goal: if the “new listings” module changed but the validator only tracks the category description, Google is told nothing changed and misses the links to your freshest URLs. Hub pages either need their link sets included in the validator, or they should be left out of 304 handling.

    That risk is also a strong argument for starting narrow. Stable leaf pages (older articles, long-lived product or listing detail pages, archives) are where refresh crawls are most likely to find nothing new, and where a missed change costs least.

    What I would check before approving the work

    1. Do your HTML responses carry validators at all? Googlebot can only send If-None-Match if you previously sent an ETag. If HTML responses have neither ETag nor Last-Modified, there is nothing to be conditional on yet.
    2. Is Googlebot sending conditional requests, and how does your server answer? Standard access log formats do not record these request headers. Log If-None-Match and If-Modified-Since for verified Googlebot requests for a week or two, alongside the status returned. Look for the pattern where validators go out, conditional requests come back, and the server still returns a full 200.
    3. How much refresh crawling hits unchanged pages? Join the logs to your CMS’s last-changed timestamps. If most Googlebot refreshes land on pages that did change, the ceiling on savings is low.
    4. Is capacity the binding constraint? In Crawl stats, compare host status, average response time and total crawl requests over the same period, and look at the By purpose split.
    5. How slow is fresh URL crawling today? From logs, measure the time between a URL going live and Googlebot’s first request, by template. Without that baseline you cannot show that anything improved.

    Hypothetical example:

    A classifieds site has 6 million listing URLs, adds 25,000 listings a day, and renders listing pages at the origin with a median response time above one second. Logs show most Googlebot requests are refreshes, the majority of them on listings unchanged since the previous fetch, and Crawl stats shows host status problems during traffic peaks. New listings take two to four days to receive their first Googlebot request, even though they appear in sitemaps and on category pages. This site is a reasonable candidate: capacity looks binding, and cheap 304s on older listings could plausibly free room for discovery. A news publisher serving cached HTML in 150 milliseconds, whose new articles are fetched within minutes, has nothing comparable to gain.

    How to measure it if you build it

    A clean split test is hard because the capacity limit applies to the host. Time saved on test URLs can raise crawling of control URLs on the same host, which shrinks the measured difference and can hide a real effect. An interrupted time series on the whole host is the more honest design here.

    • Hypothesis: answering unchanged refresh crawls with 304 lowers Googlebot’s average response time, and Google spends part of the freed capacity on discovery.
    • Primary metric: median hours from publication to first Googlebot request for new URLs, by template.
    • Secondary metrics: the 304 share of Googlebot responses, average response time and total crawl requests in Crawl stats, the Discovery share under By purpose, and the share of new URLs indexed within a fixed window (sampled with URL Inspection).
    • Duration: several full weekly publishing cycles before and after the change, because publishing volume and crawling both follow weekly patterns and Google adjusts the capacity limit based on how the site responds over time. Four weeks either side is a sensible minimum for a site publishing thousands of URLs a day; lower-volume sites need longer to collect enough new URLs for a stable median.
    • Confounders: releases that change response time, CDN configuration changes, swings in publishing volume and Google-side changes. Keep a dated change log so you can separate them.

    The result that should change the decision: if 304s become a large share of Googlebot responses and response time drops, but total crawl requests, the Discovery share and time to first crawl stay flat, the site was demand-bound. Keep the 304s as a cost saving, stop extending them, and move the effort to URL inventory, internal linking and sitemaps. If crawl requests rise and new URLs are reached sooner, extend the rollout to more templates, starting with the next most stable one.

    My practical position

    Treat 304 support as crawl efficiency hygiene with a conditional upside, not as a discovery project. If validators already exist and the work is mostly configuration, do it on stable leaf templates and measure. If it needs application changes, approve it only after logs show three things: capacity is the limit, a large share of refresh crawling lands on unchanged pages, and fresh URLs are easy to find but still crawled late. If any of those fails, the same engineering time is likely to do more for fresh URL crawling when spent on the paths Googlebot uses to find new pages.

    Need help with this on your own site?

    Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.

    See SEO services

Add a reply

Replies are open to approved members. Log in or apply to join.