Discussion
  • Crawling
  • Technical SEO

Service workers and PWA caching: have you seen Googlebot get stale or app-shell-only HTML because of them, and how did you catch it?

Published 1 reply

Discussion starter: a technical SEO question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.

Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.

1 reply

  • admin
    Admin
    Original poster

    The service worker itself is an unlikely cause, but stale or shell-only HTML is a real problem on PWAs. Googlebot fetches pages with plain HTTP requests and renders them without keeping state between page loads, so there is normally no installed service worker sitting between Google and your server. When Google ends up with stale or shell-only HTML on a PWA, the cause is more likely one of the layers that ship alongside the service worker: server routes that return the same app shell for every URL, CDN or prerender caches, render-time API calls that fail for Googlebot, or Google’s own aggressive caching of JavaScript files.

    Where service workers do cause real damage is in diagnosis. They sit between the person testing and the server, so a developer’s browser can show cached content that Googlebot never receives, or hide a stale page that Googlebot does receive. That makes the tester’s own browser an unreliable witness in any “Googlebot sees stale content” investigation on a PWA.

    Why a service worker normally cannot serve Googlebot

    A service worker is installed in a particular browser after a previous visit, and it intercepts that browser’s requests within its scope. Google’s documentation describes two behaviours that make that path unlikely for Googlebot:

    • Googlebot “uses HTTP requests to retrieve content from your server”, and does not support other connection types such as WebSockets or WebRTC.
    • Rendering is stateless: “Local Storage and Session Storage data are cleared across page loads. HTTP Cookies are cleared across page loads.”.

    Google’s documentation does not address service workers directly, so the conclusion that a service worker cache does not persist between Google’s renders is an inference from that stateless model, not a documented statement. It is a strong inference, but you should confirm it on your own site with the checks further down rather than treat it as settled. Seeing Googlebot request /sw.js in your logs does not contradict it: a page that calls register() during rendering will fetch the script, and that is not the same as serving pages from a cache.

    Useful analogy:

    A service worker is like a photocopy of the menu you took home last month. Googlebot asks the kitchen directly for its menu. If you judge what the kitchen hands out by looking at your photocopy, you will diagnose the wrong problem. Technically, the “photocopy” is the Cache Storage and routing logic registered in your browser profile, and the “kitchen” is whatever your server, CDN or prerender service returns to a fresh HTTP request.

    Where stale or shell-only HTML actually comes from

    Layer Can it affect Googlebot? How it goes wrong Where you see it
    Service worker cache Unlikely Serves an old shell or old pages to returning users, including the person testing Browser DevTools, Application panel
    Server SPA fallback Yes Every route returns the same index.html with a 200; content only appears after JavaScript runs Raw HTML via curl; “Soft 404” and duplicate clusters in Search Console
    CDN or full-page cache Yes Long HTML TTLs, stale-while-revalidate, stale-if-error, or a cache key that ignores variation Response headers such as Age and cache status
    Prerender or dynamic rendering cache Yes Bot-only snapshots refreshed on a schedule, or captured before data loaded Compare the bot response with the user response and the snapshot date
    Google’s resource cache Yes Outdated JavaScript or CSS used against fresh HTML Rendered HTML in URL Inspection; failed resources in its details
    Render-time API calls Yes API blocked by robots.txt or bot protection, rate limited or slow, so the render finishes with an empty shell Rendered HTML, console messages, server and firewall logs

    The resource cache row deserves attention because it is documented and easy to miss. Google says: “Googlebot caches aggressively in order to reduce network requests and resource usage. WRS may ignore caching headers.” Its recommended fix is content fingerprinting, making “a fingerprint of the content part of the filename, like main.2bb85551.js“. If your bundle is always called app.js, Google can render new HTML with old code. A related failure mode, reasoned from how bundlers work rather than documented by Google: an old runtime asks for code-split chunks that a deploy has deleted, the chunks return 404, and the render stops at the shell.

    Google also describes the app shell model directly: the initial HTML “does not contain the actual content and Google needs to execute JavaScript before being able to see the actual page content”. That dependence is where “shell-only” results come from. And if the server returns a non-200 status, Google says “rendering might be skipped”, so an error status on a shell page can leave Google with only the empty template.

    How to catch it

    1. Fetch the raw HTML like a crawler. Use curl with and without a Googlebot user agent, and read the headers as well as the body. Is the main content in the HTML? What do Age, Last-Modified and the CDN cache status say? Keep in mind that a spoofed user agent is not treated the same as real Googlebot by every CDN or firewall.
    2. Compare Google’s two views in Search Console. URL Inspection’s “View crawled page” shows what Google stored from its last crawl. “Test live URL” shows the rendered HTML, a screenshot, resources that failed to load and JavaScript console messages. Google points to the URL Inspection Tool and the Rich Results Test for testing how it crawls and renders a URL. Google’s public cached-page link was retired in 2024, so these are the reliable views.
    3. Add a build marker you can read in the rendered HTML. Write the build ID into the server HTML, and have the JavaScript bundle write its own build ID into the page when it runs. If URL Inspection shows HTML build 412 alongside JavaScript build 409, Google rendered fresh HTML with stale code. If both show 409 after you deployed 412, the HTML itself is stale, which points at a cache.
    4. Read the logs for verified Googlebot. Verify requests against Google’s published crawler IP ranges, then look for 404s on JavaScript chunks after deploys, 403 or 429 responses on API endpoints, and HTML requests served from cache long after a publish.
    5. Look for the index-level footprint. Identical app shells tend to cluster. On a JavaScript-dependent site, a rise in “Duplicate, Google chose different canonical than user”, “Soft 404” or “Crawled - currently not indexed”, or search results showing the same generic title across many URLs, is a strong hint that Google stored shells.
    6. Only then test in your own browser, with the service worker out of the way. In Chrome DevTools, use the Application panel’s service worker options (“Bypass for network” or “Update on reload”), or a fresh browser profile. Check whether navigator.serviceWorker.controller is set to confirm whether a worker controls the page you are looking at.
    Fetch raw HTML with a crawler user agent
    ↓
    Is the main content in the raw HTML?
    ↓
    Yes: is it current? If not, check CDN, full-page and prerender caches
    ↓
    No: does URL Inspection’s rendered HTML contain it?
    ↓
    Rendered HTML empty or stale: check failed resources, API responses and build markers

    A worked scenario

    Hypothetical example:

    A ticketing PWA reports that Google is showing last week’s prices in snippets. The developer checks the site and also sees last week’s prices, so the team blames the service worker and removes it. Snippets stay stale.

    Working through the steps shows two separate problems. The developer’s browser was stale because the service worker used a cache-first strategy for page navigations. Googlebot was stale for a different reason: the server-rendered HTML, prices included, was cached at the CDN for 24 hours with no purge on price change. Removing the service worker fixed only the first problem. Purging product pages on price change fixed the second. Same symptom, two causes, and only one of them involved Google.

    Fixes that address the real causes

    • Put primary content and metadata in the initial HTML. Server-side rendering or static generation for the title, canonical, robots directives and main content removes the dependence on a successful render. The app shell can still load on top.
    • Fingerprint JavaScript and CSS filenames, as Google recommends, and keep previous builds’ hashed assets available for a while after each deploy so an older runtime does not break.
    • Return real status codes for unknown routes. A server fallback that answers every path with a 200 shell creates soft 404s. For client-side routing, Google suggests redirecting with JavaScript to a URL that returns a server 404, or adding a noindex robots meta tag.
    • Make HTML caching deliberate. Short TTLs with revalidation for HTML, purge on publish or price change, and a conscious decision about stale-if-error (serving an old page during an outage is often better than a 5xx, but know that it happens).
    • Keep render dependencies reachable. Do not disallow API or script paths in robots.txt, and let verified Google crawlers through bot protection and rate limits.
    • Watch HTML size. Googlebot fetches up to 2MB of an HTML file, and bytes beyond that are not processed. Large inlined state objects in app-shell frameworks can push content past the cut-off.
    • Treat bot-only prerendering as a stopgap. Google calls dynamic rendering “a workaround and not a recommended solution”, and its snapshot cache is another layer that can go stale.
    • Fix the service worker for users, not for Google. A network-first strategy for navigations, with the cached shell as a fallback when offline, stops returning users and testers seeing stale pages.

    What would change this conclusion

    No controlled public test exists that I can point to showing a service worker cache influencing Google’s rendered output. If you ever see URL Inspection’s rendered HTML contain a build marker that exists only in an old service worker precache, and nowhere on your origin, CDN or prerender layer, that would be real evidence against the stateless model and worth reporting with the logs attached. Until then, the efficient order is: rule out the server, the caches, the render dependencies and Google’s resource caching first, and treat the service worker as a problem for your users and your test process.

    Need help with this on your own site?

    Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.

    See SEO services

Add a reply

Replies are open to approved members. Log in or apply to join.