Discussion starter: a technical SEO question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.
- JavaScript SEO
- Technical SEO
A client-rendered app returns 200 for every route, even dead ones. Is real 404 handling at the edge worth a sprint, or do soft-404 classifications cope?
Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.
1 reply
-
adminAdmin
Soft 404 classification copes in the sense that Google usually keeps obviously dead pages out of the index eventually. It does not cope in the sense of stopping crawling, saving rendering, or helping anything that is not Googlebot. Whether real 404s at the edge are worth a sprint depends mostly on two things: whether your route space is bounded, and how many dynamic entities (products, listings, profiles, jobs) expire each month.
A 60-route marketing app with no user-generated URLs can live with Google’s documented client-side fallback. An app where
/item/:idrenders a shell for any ID, and thousands of IDs expire every month, should return real status codes. And it is often less than a sprint: matching unknown route patterns at the edge is small work. The expensive part is checking whether a specific dynamic ID still exists, and that part is only worth building where churn is high.What “coping” looks like from Googlebot’s side
Googlebot requests/item/88213(deleted last month) and gets HTTP 200 with the app shell↓The URL is queued for rendering, as every 200 response is↓Rendering runs your JavaScript and the API call returns “not found”↓Google decides what the rendered page is: Soft 404, a duplicate of other shells, or a thin page it may index↓The URL stays known and keeps being recrawledGoogle’s crawl budget guide is direct about the last step: “Soft 404 pages will continue to be crawled, and waste your budget.” By contrast, “a 404 status code is a strong signal not to crawl that URL again”. The JavaScript SEO documentation adds that when the status code is non-200, “rendering might be skipped”. So a real 404 saves both the recrawl and the render. A soft 404 costs both, every time.
Where relying on soft 404 detection falls short
- It is a classifier, so it misfires both ways. A dead route that renders a friendly “this item is gone, here are similar items” state may not look like an error, so it gets indexed as a thin page or grouped with duplicates. Meanwhile live but thin pages (empty filter results, sold-out products with little content) can be flagged as Soft 404 when you wanted them indexed.
- It pollutes your diagnostics. When dead routes fill the Soft 404 bucket in the Page indexing report, real thin-content problems on live pages get lost in the noise. The same happens in “Duplicate without user-selected canonical” when many dead URLs render an identical shell.
- It only exists at Google. Many non-Google crawlers, including several AI crawlers, are widely reported not to execute JavaScript. They see a 200 app shell for every dead URL. Link checkers, uptime monitors, CDN caches and your own analytics treat those URLs as valid too.
- Unbounded route spaces multiply the problem. If
/anything/at/allreturns 200, then a relative link bug, a scraper’s malformed link or a typo creates a new crawlable URL, and each one needs rendering before Google can dismiss it.
Evidence to pull before you commit the sprint
- Run
curl -s -o /dev/null -w "%{http_code}" https://yoursite.com/definitely-not-a-page-123and the same for a deleted entity URL. Confirm both return 200. - Crawl Stats report, by response: if 404 responses are near zero on a site where content changes daily, dead URLs are hiding behind 200s.
- Page indexing report: size and trend of “Soft 404”, “Duplicate without user-selected canonical” and “Crawled - currently not indexed”. Sample the URLs. Are they dead routes or live pages?
- Server or CDN logs: what share of Googlebot requests hit URLs that no longer correspond to a live entity, and how often it returns to them.
- Churn rate: how many dynamic entities are removed per month, and whether their URLs are linked externally or remain in old sitemaps.
- URL Inspection on a few dead URLs: whether Google has indexed any of them, and what the rendered HTML shows.
When the sprint is worth it
Condition Soft 404 detection plus client-side fallback is probably enough Real 404s at the edge are worth the sprint Route space Fixed set of routes known at build time Catch-all or pattern routes that render for any path or ID Entity churn Content rarely removed Listings, jobs, products or profiles expiring every week Scale Hundreds to low thousands of URLs Tens of thousands upwards, or many URLs in “Discovered - currently not indexed” Search Console Small, stable Soft 404 count made of genuinely dead pages Growing Soft 404 or duplicate counts, or dead URLs appearing as indexed Logs Dead URLs are a small share of Googlebot hits Dead URLs take a large share of crawl activity Channel dependence Organic search is a minor channel Organic is a primary acquisition channel for the dynamic pages Google’s crawl budget guide is aimed at sites with over a million pages changing weekly, or over 10,000 changing daily. Below that, crawl waste alone rarely justifies engineering time. The better arguments at smaller scale are index hygiene, clean diagnostics and correct behaviour for non-Google systems.
It is usually three jobs, not one
Tier What it does Typical effort What it misses 1. Client-side fallback (documented by Google) When the router or API says “not found”, redirect with JavaScript to a URL like /not-foundthat the server answers with a 404, or inject<meta name="robots" content="noindex">Hours to a couple of days First response is still 200 and still needs rendering. Google’s crawl budget guide notes noindex pages are still requested. Non-rendering crawlers see nothing. 2. Edge route matching Generate a manifest of valid route patterns from the router at build time. An edge function returns 404 (with the SPA’s not-found view as the body) for paths matching no pattern A few days on most edge platforms Valid patterns with dead IDs, such as /item/882133. Edge entity check For dynamic routes, check the ID against a key-value store or a fast lookup and return 404 or 410 when it does not exist The sprint: data sync, cache rules, latency budget, failure handling Only as accurate as the data the edge can see Tier 2 also fixes the most common root cause. Static hosting setups often map every missing file to
index.htmlwith a 200 so that deep links to valid client routes work. Simply changing that mapping to return 404 would break every deep link, which is why the edge needs to know which routes are valid rather than which files exist.If you are already planning server-side rendering or prerendering for key templates, fold status handling into that project instead. A server that renders the page already knows whether the entity exists.
The failure mode that matters more than soft 404s
A false 404 on a live page is worse than a soft 404 on a dead one. If a new listing is published but has not yet reached the edge store, or the lookup times out, a strict implementation tells Google a real page is gone. Build it to fail open:
- If the lookup errors or times out, serve the 200 shell and let the client-side fallback handle it.
- Give 404 responses a short cache lifetime, so a newly created entity is not hidden behind a cached 404.
- Write to the edge store as part of publishing, not on a delayed batch job.
- Alert on sudden changes in 404 rate per template, and watch “Not found (404)” in Search Console for URLs that should be live.
Whether you use 404 or 410 for removed entities matters much less than getting existence right. Google’s guidance treats both as signals for permanently removed pages.
Hypothetical example:
A client-rendered jobs board has 30,000 live ads and removes about 12,000 a month. Every expired ad URL still returns 200, is linked from aggregators, and renders “this job has closed”. Over a year, dead URLs outnumber live ones several times over, and each one must be rendered before Google can classify it. Tier 2 alone would not help, because the paths are valid patterns. Tier 3 is the fix. Compare a SaaS marketing app with 80 fixed routes and no user content: tier 1 plus tier 2 covers it in a few days, and tier 3 is unnecessary.
How to confirm it worked
- Rerun the curl checks on dead routes, dead IDs and a sample of live URLs. Live URLs must still return 200.
- Use URL Inspection live tests on a few dead URLs to confirm Google sees the 404.
- Over the following weeks, expect the 404 share in Crawl Stats to rise, then the Soft 404 count in Page indexing to fall as URLs are recrawled.
- In logs, check that Googlebot revisits dead URLs less often over time, and that crawl of live pages holds steady or improves.
- Confirm indexed counts for live templates do not drop. If they do, look for false 404s before anything else.
If this were my app, I would run the curl test and the log check today. If the route space is open-ended, I would ship tiers 1 and 2 immediately, since they are cheap and remove the junk-URL problem. I would only commit the full sprint for tier 3 if logs or Search Console show dead dynamic URLs taking a meaningful share of crawling or appearing in the index, because that is where the engineering cost and the false-404 risk both sit.
Need help with this on your own site?
Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.
Add a reply Cancel reply
Replies are open to approved members. Log in or apply to join.