Discussion starter: a technical SEO question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.
- Crawling
- Technical SEO
On a static site with 80k pages rebuilt nightly, how do you stop a failed build from quietly publishing empty or broken pages to Googlebot?
Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.
1 reply
-
adminAdmin
Treat publishing as a gate, not a side effect of the build. A nightly rebuild of 80,000 pages should only replace the live site when the new build passes checks that compare it with the last good build: page counts, content size per template, required elements, and the health of the data it was built from. If any check fails, the previous build stays live and someone gets alerted. Most “empty page” incidents on static sites are not rendering bugs. They were data fetches that failed quietly, returned empty results, and let the templates render perfectly valid pages with nothing in them. The fix starts there.
Where silent failures come from
- Swallowed data errors. An API call times out, the code catches the error and returns an empty array, and every product list renders “No products found”.
- Partial data. A paginated export stops after page three, so 70% of items are missing and their pages are either removed or rendered empty.
- Schema drift. The CMS renames a field; the template reads
undefinedand prints nothing, or prints the word “undefined”. - Environment problems. Missing secrets or wrong base URLs in the build environment, producing canonicals or links pointing at staging.
- Mass deletions. A filter bug excludes a large category, and the build treats those pages as removed.
Each of these produces HTML that looks structurally fine. That is why checks based only on “did the build exit successfully” miss them.
Checks that catch them
Check What it catches Typical rule Data source validation before rendering Swallowed errors, partial exports Fail the build if any source returns an error, or if row counts fall more than a set percentage below the last successful run Page count by template versus last good build Mass deletions, missing sections Block if any template’s count changes by more than a threshold you set from normal daily variance Content size per template Empty pages, stripped content Block if the median text length of a template falls sharply, or if more than a small share of pages fall below a minimum Required elements Schema drift, broken templates Every page has a title, a self-referencing canonical on the production host, an H1, and for products a price and availability; JSON-LD parses Forbidden strings Undefined values, debug output No “undefined”, “null”, “NaN”, “[object Object]”, “lorem ipsum” or staging hostnames in rendered HTML Internal link integrity Broken navigation, orphaned sections No links to pages that do not exist in the build; category pages link to at least a minimum number of items Change volume Anything unexpected Block if the share of pages whose main content hash changed overnight is far above the normal range Set thresholds from your own history. Run the checks in reporting mode for a couple of weeks, look at the normal variation, and set limits just outside it. Too tight and the team learns to override alerts; too loose and the gate is decoration.
Deployment design that makes the gate enforceable
Fetch and validate data; fail on errors or suspicious counts↓Build to a new, versioned output location↓Run content checks against the last good build↓All checks pass?↓Yes: switch the live pointer atomically and keep the previous build for instant rollback. No: keep the old build live and alert the ownerAtomic deploys matter. Most static hosts and CDN setups can publish a new build to a separate location and switch traffic in one step. If yours uploads files over the live site one by one, a failed or interrupted deploy leaves a mixture of old and new pages, and there is no clean rollback. Keep at least the last few good builds so rollback is a pointer change, not a rebuild.
Watch for this:
Do not regenerate sitemaps from the new build until it passes the gate, and do not update
lastmodon pages whose content did not change. A broken build that also publishes a sitemap announcing 80,000 fresh modifications is inviting Googlebot to come and fetch the broken pages quickly.Handling removals safely
Removing pages is where a static build can do the most damage, because a missing file becomes a 404 without anyone deciding it should. Two safeguards help:
- Keep an explicit list of intentionally removed URLs, generated from the data source’s own “deleted” or “discontinued” state, and compare it with the set of pages missing from the build. Pages missing without being on the list block the deploy.
- Cap the number of removals per night. If more than the cap would disappear, require a human to approve. Genuine bulk removals are rare and planned.
After publishing: confirm what Googlebot receives
- Run a scripted fetch of a fixed sample of URLs from each template after every deploy, checking status codes, content length and required elements on the live site, not on the build output.
- Watch server or CDN logs for verified Googlebot requests and the status codes and response sizes they received. A sudden drop in average response size is an early sign of empty pages.
- Use Search Console’s Page indexing and Crawl stats reports as lagging confirmation. They will not catch a problem the same night, but they show whether Google saw something unusual.
If broken pages did go live
Roll back first, then investigate. A short exposure, measured in hours, usually does limited damage because Googlebot recrawls only a fraction of a large site in that time. Pages it did fetch while empty may be treated as soft 404s or dropped for a while, and will recover as they are recrawled with full content. Longer exposures, or ones where a sitemap pushed Google to recrawl everything, can take weeks to wash out. After the fix, check a sample of affected URLs with URL Inspection and consider resubmitting the sitemap so the corrected pages are fetched sooner.
Hypothetical example:
A travel site rebuilds 80,000 destination pages nightly from a supplier API. One night the API returns rate-limit errors for half the requests, and the build code treats them as “no hotels”. The content-size check sees median text length on hotel list pages fall by more than half and the page-count check sees nothing unusual, because the pages still exist. The gate blocks the deploy, keeps yesterday’s build live, and alerts the on-call developer, who adds retry logic. Without the content check, 40,000 pages saying “No hotels available” would have gone live.
Reduce the blast radius over time
If the platform allows it, move from full nightly rebuilds to incremental builds that regenerate only pages whose data changed. Fewer pages rebuilt means fewer pages at risk each night and faster checks. Whatever the build model, the principle stays the same: the live site should only change when the new version has proven it is at least as complete as the one it replaces.
Need help with this on your own site?
Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.
Add a reply Cancel reply
Replies are open to approved members. Log in or apply to join.