Discussion
  • Indexing
  • News SEO

Fifteen years of news archives: prune, noindex, or keep? Has anyone measured the effect on Top Stories or Discover?

Published 1 reply

Discussion starter: a publisher and community SEO question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.

Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.

1 reply

  • admin
    Admin
    Original poster

    Keep your original reporting indexed. For a 15-year archive, the case for noindexing or deleting journalism because of its age is weak: Google’s Search Liaison has called the belief that Google dislikes “old” content “not a thing”, Google’s core update guidance calls deletion a last resort, and older articles can still appear in Discover, but only while they are indexed. The clean-up worth doing is narrower: syndicated wire copy you host, thin auto-generated tag and date pages, and short breaking-news stubs that a fuller story on your own site has replaced.

    On the measurement question, no controlled public test exists that I can point to that isolates the effect of archive pruning on Top Stories or Discover. The mechanisms suggest little upside for Top Stories, which runs on recent coverage, and some downside risk for Discover, which can surface older articles. That makes this a decision for your own crawl and performance data, not for a general belief that a smaller site is a stronger one.

    What Google has said about old content

    When CNET deleted thousands of older articles in 2023, arguing that it would signal freshness to Google, Google’s Search Liaison (Danny Sullivan) responded that deleting content because you believe Google does not like “old” content is “not a thing”. He allowed one narrow benefit: on a massive site, removing pages might mean Google is better able to crawl other content, but that does not mean Google then decides the whole site is “so much better”.

    Google’s core updates documentation puts the same idea in policy form: “Deleting content is a last resort, and only to be considered if you think the content can’t be salvaged.” For a news archive, “salvaged” is the right test. A report that was accurate when published is not broken because it is old. It may need a correction, an update note, a repaired embed or a link to newer coverage. That is maintenance, not deletion.

    Four routes by which an archive could affect Top Stories and Discover

    These are worth separating, because they point to different checks. Only the first has a direct Google statement behind it, and none has public test data isolating its effect on these two surfaces.

    Route What would have to be true How to check
    1. Crawl capacity for new stories Googlebot’s time on archive URLs delays fetching of new articles. Google’s crawl budget guide is written for sites with over a million unique pages changing about weekly, or over 10,000 pages changing daily, so a large news site can qualify. Time from publication to first Googlebot request for new articles (server logs); share of Googlebot requests going to archive URLs (logs and the Crawl stats report)
    2. Site-level quality A large share of the archive is thin or duplicated, and that weighs on how the site as a whole is assessed. Google’s core update guidance tells owners facing broad drops to look at the site overall, not only individual pages. Share of archive URLs that are wire copy, stubs or near-empty tag pages, compared with original reporting
    3. Topical depth Years of coverage on a beat help show expertise. Google’s February 2026 Discover core update (which began with English-language users in the US) aims to show more in-depth, original content from sites with expertise in a topic. Linking that to archive depth is my inference, not something Google has stated. No direct check; watch Discover performance for the topics in any segment you change
    4. Discover eligibility of older articles Old articles earn Discover impressions today. Google’s Discover documentation says content is eligible if it is indexed and meets Discover’s content policies, and that older content may appear if it is helpful and relevant to someone’s interests. A noindexed article is out. Discover performance report, filtered to archive URLs

    Top Stories draws on recent coverage, so an archive can only reach it indirectly, through routes 1 and 2. If new articles are fetched within minutes of publication and the archive is mostly original reporting, pruning gives Top Stories very little to respond to. Discover is different: route 4 is a documented way that noindexing old articles removes them from a surface, and route 3, if the inference holds, points the same way.

    Prune, noindex or keep: decide by segment, not by age

    Archive segment Default action Why Watch for
    Original reporting, investigations, features, obituaries Keep indexed; fix broken media; add correction or update notes where facts changed Unique, able to attract links, and part of the public record If your newsroom has an unpublishing policy, SEO should go through it, not around it
    Evergreen explainers and profiles with current demand Keep, refresh with dated update notes, link from maintained topic pages They can earn search and Discover traffic now Do not change publication dates without a substantive update
    Syndicated wire copy you host Noindex, or remove if your licence requires it The same text sits on many other sites, so your copy adds little Check the agency licence terms first; some limit how long you can host the copy
    Breaking-news stubs replaced by a fuller story Redirect to the fuller story on the same event A true equivalent exists on your site Only redirect to a real equivalent
    Tag, author and date archive pages with a handful of articles Noindex or consolidate into maintained topic pages Thin and auto-generated; on some CMS setups they outnumber articles Keep them crawlable if they are the only internal paths to older stories
    Legally problematic items Legal and standards process Not an SEO decision Keep a record of what was removed and why

    Two implementation traps:

    A noindex only works if Google can crawl the page and see it, so do not also block the same URLs in robots.txt. And mass redirects of unrelated articles to a section front or homepage can be treated as soft 404s, so they preserve little while removing the record readers and other sites link to.

    What to check before changing anything

    1. Discovery speed. From server logs, measure the time from publication to Googlebot’s first request for new articles over a few typical weeks. If new stories are fetched promptly, crawl is not the constraint, and the Search Liaison’s crawl caveat does not apply to you.
    2. Where Googlebot spends its requests. Split requests by URL type (new articles, archive articles, tag and date pages, parameters). A large share on thin tag pages is a stronger case for clean-up than a large share on articles.
    3. What Google already excludes. In the Page indexing report, check how many archive URLs sit under “Crawled - currently not indexed”. Google may already be leaving much of the thin material out on its own.
    4. Current value. Pull 16 months of clicks and impressions per archive URL from Search Console, plus Discover impressions and referring links. URLs with no impressions and no links are candidates for review, not automatic removal.
    5. AI exposure. The Search generative AI performance report can be filtered by page. Archive URLs with AI impressions are appearing in Google’s AI features, which is exposure you would give up by noindexing or removing them, even though the report shows no clicks.

    Hypothetical example:

    A regional publisher has 1.1 million URLs: 450,000 articles (140,000 of them agency copy), 400,000 tag and date pages, and the rest sections, galleries and parameters. Logs show new articles fetched within a few minutes of publication, and about a third of Googlebot requests going to tag pages. The Discover report shows a steady trickle of impressions on older features and obituaries. The sensible plan is to noindex the agency copy (subject to the licence), consolidate thin tag pages into maintained topic pages, and leave the original reporting alone. Top Stories is unlikely to move, because nothing in the data suggests discovery was the constraint, so any Top Stories lift should not be the success measure for this project.

    If you do change something, measure it like this

    There is no clean control. Top Stories and Discover exposure are driven by the whole site’s output and the news cycle, so you cannot split-test them within one site the way you can test a template. What you can do is stage the change by segment and read it as a before-and-after series:

    • One segment at a time. Agency copy first, then thin tag pages. Mark each deployment with Search Console’s custom annotations.
    • A long enough baseline. Google’s Discover documentation describes Discover traffic as “less predictable or dependable” than keyword-driven search traffic, so the pre-change period has to show your normal range of swings before you judge any change against it.
    • Normalised metrics. Discover clicks per new article, the share of new articles with any Discover impressions, and the share of a fixed set of tracked news queries where you appear in Top Stories (tracked externally). Raw totals move more with publishing volume and the news cycle than with any archive change.
    • Known confounders. Core updates (the March 2026 and May 2026 updates both completed this year), the Discover core update, and smaller unannounced core updates, which Google’s core updates documentation says also happen.

    What would change the decision: if discovery latency for new articles falls and Top Stories presence rises on comparable news events after a clean-up, the clean-up was worth extending. If Discover impressions on archive URLs disappear with no gain elsewhere, restore indexing for the segments that were earning them.

    The decision rule

    If this were my archive, I would not ask “prune, noindex or keep?” of the archive as a whole. I would ask it of each URL class, using discovery speed, crawl share, current clicks and Discover impressions, links, and duplication with other sites. Agency copy fails the duplication test by definition, and thin tag pages are likely to fail several of the others. Original reporting passes on uniqueness by definition, even when its traffic is small, and for a news organisation it has value that search metrics do not capture.

    Need help with this on your own site?

    Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.

    See SEO services

Add a reply

Replies are open to approved members. Log in or apply to join.