Discussion starter: a technical SEO question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.
- Crawling
- Internal Linking
- Technical SEO
The ‘Crawled - currently not indexed’ count jumped after a redesign with no content change. First check: rendering, internal links, or quality re-evaluation?
Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.
1 reply
-
adminAdmin
Of the three, check rendering first, but spend a few minutes before that on a step that is none of them: find out which URLs moved into “Crawled - currently not indexed”. A redesign can change the URL set without changing any content, and the example list shows quickly whether you are looking at pages that used to be indexed or at new URLs the template started exposing. Internal links come second. “Quality re-evaluation” is the conclusion you reach once both are clean, not the starting hypothesis. Two kinds of redesign change that order, covered at the end: one that changed URLs, and one that only restyled otherwise identical HTML.
The reason for that order is that “no content change” describes the CMS, not what Google received. Google indexes what it fetched and rendered. A redesign can leave every product description untouched in the database and still move it into a tab that loads on click, behind a blocked API call, below a large block of inline script, or under several screens of new boilerplate. From Google’s side, those pages changed. Rendering is also quick to prove or rule out, while a quality explanation can only be reached by eliminating the others.
What the status means after a redesign
Google’s Page indexing documentation defines it briefly: “The page was crawled by Google but not indexed. It may or may not be indexed in the future; no need to resubmit this URL for crawling.” The same page says not to expect every URL on a site to be indexed. So the status records a decision made after crawling, which puts it downstream of both rendering and internal linking.
A redesign changes every template at once, so each page Google recrawls after launch is evaluated in its new form, and each recrawl is a fresh indexing decision. That is why a template change can show up as a sharp rise in this report instead of scattered movement. Three different things can sit behind the rise: new URLs the template exposes, previously indexed pages that Google now receives differently, and marginal pages that were holding on and got a fresh decision.
Step zero: find out which URLs moved
- Export the examples and group them by pattern: template, directory, parameters, new paths. The list is limited to 1,000 rows and is not guaranteed to show every URL in the status, so treat it as a sample.
- Confirm the timing on a sample. URL Inspection’s “Last crawl” field shows when Google last crawled the page, and everything the tool reports comes from that version. If a URL’s last crawl predates the launch, the redesign did not produce its status.
- Compare with the indexed total. If indexed pages fell by roughly the amount this status rose, pages are leaving the index, which is serious. If total known pages grew instead, Google has found new URLs, which points to crawl hygiene rather than a ranking problem.
- Get per-template counts by filtering the Page indexing report by sitemap, with one sitemap per template.
- Check the Crawl stats report around the deployment date. A jump in crawling for discovery suggests new URLs; a jump in JavaScript or JSON requests suggests the new template leans harder on rendering resources.
- Check whether “Soft 404” and the duplicate statuses rose at the same time. Empty or near-identical renders can surface there too, and a rise across several statuses points towards what Google received rather than how it judged the content.
What the examples show Likely meaning Next step Sort, view, quick-view or AJAX fragment URLs that were never linked before The template exposes new crawlable URLs Fix the links or parameters; there is nothing to recover Old URLs from before the redesign If paths changed, these should be redirecting and reported as “Page with redirect” Confirm the old URLs return 301s, not 200 responses with a shell page Previously indexed content pages on unchanged URLs Google reached a different decision on pages it knew Rendering, then internal links One template only, for example products but not categories A template-specific change Diff that template’s old and new HTML Pages that were already weak before the redesign Marginal pages got a fresh decision A quality question, after the technical checks Rendering: a check that can prove a cause quickly
Take ten or so affected URLs and ten still-indexed URLs from the same template and run both groups through URL Inspection’s live test, which fetches and renders each page as it is now. Where “View crawled page” is available, it adds what Google stored from its last crawl: according to Google’s URL Inspection help, it shows the HTML returned, the HTTP headers, JavaScript console output and the page resources loaded, while a screenshot is only available from a live test with a successful result. Search the HTML for a sentence from the main content. If the description, specifications or reviews are missing from affected pages but present on indexed ones, you have a cause.
What to look for after a redesign specifically:
- Main content that only loads after a click, such as tabs, accordions or “read more” controls that fetch content on interaction instead of hiding content already in the HTML.
- Content fetched from API endpoints that robots.txt blocks or bot protection challenges.
- Titles, headings, canonicals or robots meta tags that JavaScript sets differently from the server HTML. Google’s JavaScript SEO documentation notes that a noindex in the original HTML can stop rendering, so removing it with JavaScript may not work, and that pages returning non-200 status codes might not be rendered.
- HTML weight. Googlebot fetches up to 2MB of an HTML file and ignores the bytes beyond that, and a new front-end framework can inline large state objects ahead of the content.
- Boilerplate ratio: new mega menus, footers and promotional blocks that make pages on one template look more alike, with the unique text reduced to a small fraction of the page.
Use the live test to confirm a fix, but do not read a pass as a promise. The same help page states: “Even if you get a valid or warning verdict in the live test, your page must still fulfill other conditions in order to be indexed.”
Internal links: did the redesign change how pages are found and weighted?
Redesigns rebuild navigation. Mega menus shrink, breadcrumbs change, related-product modules disappear, pagination becomes “load more”, and filters become buttons. Google’s link documentation says: “Generally, Google can only crawl your link if it’s an
<a>HTML element (also known as anchor element) with anhrefattribute.” The same page says most links in other formats won’t be parsed and extracted by Google’s crawlers, so links rebuilt as buttons, spans or onclick handlers can drop out of Google’s view even though every user can still click them.A page in this status was crawled, so Google found it somewhere, possibly only through your sitemap. Compare the discovery details in URL Inspection (the sitemaps listed and the referring page, which Google describes as a page it possibly used to discover the URL) for affected and indexed URLs. Then crawl the site twice, with and without JavaScript rendering, and compare inlinks and click depth per template against a pre-redesign crawl. If you have no old crawl, crawl a staging copy of the old template or rebuild one from archived HTML.
Keep the limits of this evidence in view. Google does not document how internal link counts feed indexing decisions, so the chain from weaker internal links to pages dropping out of the index is a practitioner model rather than documented behaviour. Look for a clear before-and-after difference on the affected template, not a small shift in averages.
Quality re-evaluation: the diagnosis you reach by elimination
If rendering and links check out, a quality-type decision becomes the leading explanation, and there are legitimate ways it happens with unchanged text. Pages that were marginal before (thin, near-duplicate, little search demand) were recrawled after the redesign and got a fresh decision. Or the redesign coincided with ranking changes: besides announced core updates, Google’s core updates documentation says it also makes smaller core updates that are not announced because they aren’t widely noticeable.
Two checks separate these explanations. First, pull the affected URLs’ Performance data for the three months before the redesign. If they earned meaningful clicks and impressions and then dropped out with identical text, a pure quality re-evaluation is less plausible than a technical change you have not found yet. If they had almost no impressions before, Google may have been keeping them marginally, and the recrawl produced a decision that was coming anyway. Second, compare the timing with ranking updates listed on Google’s Search Status Dashboard.
Do not rewrite content while diagnosing. Changing copy and template at the same time removes your ability to tell which one mattered.
Which URLs moved into the status?↓New or parameter URLs: fix links and parameters, nothing to recover↓Previously indexed pages: compare rendered HTML for affected and indexed URLs↓Content missing or altered: fix rendering, then wait for recrawl↓Rendering clean: compare internal links and click depth before and after↓Links lost: restore crawlable<a href>links↓Both clean: weigh prior performance and update timing for a quality explanationWhat this looks like in practice
Hypothetical example:
A homewares retailer relaunches its front end. Product descriptions and specifications now sit in a “Details” tab that fetches content from an API when clicked. Customers see the same information as before. Googlebot receives the title, price, delivery boilerplate and an empty tab, so thousands of product pages look nearly identical. Over the following weeks, “Crawled - currently not indexed” climbs on the product template only, while category pages are unaffected. The rendered HTML from URL Inspection’s live test has no description text for affected products.
The fix is to put the tab content in the initial HTML (it can stay visually collapsed), not to rewrite descriptions. Recovery then follows Googlebot’s recrawl of those pages. Google’s definition says there is no need to resubmit them, and URL Inspection works within daily per-property limits, so watch recrawls in the logs rather than inspecting and requesting indexing one URL at a time.
When the order shifts
- The redesign changed URLs. Treat it as a migration: redirect checks come before anything else, and step zero tells you whether old URLs are part of the rise.
- The redesign changed the rendering approach, for example from server-rendered to client-rendered pages or to a new framework. Rendering moves straight to the top after step zero, and it is worth checking every template, not a sample.
- The HTML is structurally identical and only styles changed. Rendering is unlikely to be the cause, so move quickly to links and timing.
Where I would spend the first hour
If this were my site, the first hour would go on the example list and a side-by-side of rendered HTML from URL Inspection for affected and still-indexed pages from the same template. That answers the rendering question with evidence, and it tells you whether the jump involves pages you care about at all. Internal links come next, because the evidence is a before-and-after crawl and the fixes are cheap. Quality is the answer you can give only after both come back clean.
Need help with this on your own site?
Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.
Add a reply Cancel reply
Replies are open to approved members. Log in or apply to join.