Discussion
  • SEO Analytics

Holdout groups for SEO changes: have you kept pages unchanged at scale to measure impact, and did leadership accept the lost upside?

Published 1 reply

Discussion starter: an SEO measurement question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.

Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.

1 reply

  • admin
    Admin
    Original poster

    Holding pages back is workable at scale on sites with many pages on a shared template, and it produces a more defensible number than a before-and-after comparison. Two qualifications matter more than the mechanics. First, a page holdout measures the page-level effect of a change. Anything Google assesses across the whole site, and anything that spills from changed pages into unchanged ones, leaks into the holdout and distorts the measured gap: site-level effects shrink it, while pages competing for the same queries can inflate it. Second, leadership acceptance is more likely to turn on price and time limits than on methodology. A holdout’s cost is small, capped and calculable in advance, and it only exists if the change works. If the change turns out neutral, the holdout cost nothing; if it turns out harmful, the holdout protected those pages.

    Public evidence on long-running SEO holdouts is thin, and no public data on how leadership teams responded to them exists that I can point to. What follows is the design and the negotiating logic, with the uncertain parts marked.

    What a page holdout measures, and what leaks into it

    SEO experiments split pages rather than people. SearchPilot’s published methodology, written by Will Critchlow in July 2026, explains why: “each platform’s crawler is a single visitor, and so it can’t be assigned a global control or variant status like a human user.” It builds the counterfactual from the control group’s live performance during the test plus the historical relationship between control and variant pages before it starts, and reports results as credible intervals on the estimated impact. A holdout after rollout is read the same way, just with a smaller control group and a longer life. That article covers time-boxed tests; it does not address holding pages back after a rollout.

    The complication is spillover. Picture a vaccination program that reaches 90% of a town. The unvaccinated 10% also catch the disease less often, because there is less of it circulating. Compare the two groups and the vaccine looks weaker than it is. In experiment terms, the units are not independent: treating some pages changes outcomes for others.

    For SEO, the channels are concrete:

    • Site-level assessment. Google’s core updates documentation says some changes “can take effect in a few days, but it could take several months for our systems to learn and confirm that the site as a whole is now producing helpful, reliable, people-first content in the long term.” Where a change works through that kind of site-wide assessment, holdout pages share in the gain (or the loss).
    • Internal links. A change that alters links on treated pages also alters what reaches the holdout pages they link to, including crawl activity.
    • Query overlap. Treated and holdout pages that compete for the same queries trade positions with each other, which exaggerates the gap.

    Spillover can push the gap either way: site-level effects and shared links shrink it, cannibalisation inflates it. So the gap is most trustworthy for page-local changes on pages that do not compete with each other. For changes with site-wide reach, treat a positive gap as a conservative estimate of the direct effect, and where treated and holdout pages share queries, check for cannibalisation before trusting a large gap.

    Which changes suit a holdout

    Change Holdout design Spillover risk When to read it
    Title and meta description patterns Random pages within the template Low Once treated pages are recrawled and the effect stabilises
    On-page modules (spec tables, comparison blocks, review summaries) Random pages, stratified by traffic Low to medium After recrawl, plus enough time for rankings to stabilise
    Structured data additions Random pages Low After recrawl; confirm eligibility in the relevant rich result report
    Internal linking modules Whole clusters (categories or sections), not individual pages High: links from treated pages reach holdout pages Longer; compare clusters, not pages
    Content refresh or rewrite programs, including AI-assisted ones Random pages, with a site-level check Medium: quality can be assessed site-wide Months; read the gap as conservative
    Speed, navigation, migrations, domain-level changes No page holdout possible; stagger by section or market Site-wide Compare against an untouched section or market
    Accessibility, security, legal and pricing fixes Do not hold out Not applicable Not applicable

    The last row is not a technicality. Holding back an accessibility fix to measure its SEO value is hard to defend, and in the EU the European Accessibility Act has applied to many consumer-facing digital services since 28 June 2025.

    How big and how long

    Holdout size trades forgone upside against precision. For similar, independent pages, precision is limited by the smaller group, so the interval widens quickly as the holdout shrinks:

    Holdout share Share of expected gain forgone Interval width versus a 50/50 split
    50% (a split test, not really a holdout) 50% 1.0 times
    20% 20% About 1.25 times
    10% 10% About 1.7 times
    5% 5% About 2.3 times

    These ratios come from how the standard error of a difference scales with the two group sizes. Model-based methods differ in detail but not in direction. A 10 to 20% holdout is a reasonable opening position. Below 10%, check whether the interval will still be narrow enough to decide anything, using an A/A run on historical data before you ask for the holdout.

    Stratify pages by traffic and randomise within bands; SearchPilot’s bucketing, for instance, accounts for average traffic, variability and seasonality. Then decide openly what happens to your top revenue pages. Include them proportionally and the result speaks for the whole template. Keep them all in the treated group and the holdout is cheaper, but the result no longer speaks for those pages.

    Length follows the change, not a default window. Start the clock when server logs or URL Inspection show the treated pages have been recrawled. A title change can be read once the effect stops moving. A change that might act through site-level assessment needs longer, given Google’s “several months” wording, or an acceptance that the holdout will understate it. Google also documents smaller core updates that “are not announced”; a holdout absorbs those when they hit both groups similarly, which a before-and-after comparison cannot. Agree a maximum length tied to the cycle leadership reports on, and an earlier release if the interval becomes narrow enough to act on.

    During the holdout, freeze the held pages. A CMS flag that blocks other edits, merchandising changes and other tests matters more than statistical refinements, because one content team refreshing half the holdout ends the experiment. If AI Overviews sit on many of the template’s queries, compare AI feature impressions between the groups in Search Console’s Search generative AI performance report, which filters by page (impressions only).

    Putting a price on it for leadership

    Hypothetical example:

    A marketplace has 400,000 listing pages on one template, earning $1.2 million a month in organic-attributed revenue. The team expects a new content module to lift that revenue by about 4%. A 10% holdout for three months forgoes 10% of $48,000 a month for three months: $14,400, and only if the lift is real. Now suppose the module actually cuts revenue by 2% through slower pages or diluted relevance. Shipped everywhere without a holdout, that costs $24,000 a month until someone notices, and the decline will probably be blamed on the next core update.

    The arguments I would bring follow from that example:

    • A price and an end date. “$14,400 of expected upside over one quarter” is a decision leadership can make. “We need a control group” is a request they can refuse.
    • The asymmetry. The cost is capped and exists only if the change works. If it does not, the holdout protected those pages.
    • Attribution. Without a holdout, the next core update gets the credit or the blame for whatever happens after launch.
    • Future budget. A measured result with an interval is what funds the next template rollout; a forecast is not.
    • A decision rule agreed in advance, so the result cannot be relitigated when it arrives.

    The same logic predicts what weakens the case: talking about significance instead of money, holdouts with no end date, and top revenue pages placed in the holdout without anyone saying so.

    Two variations are worth offering. A program-level holdout keeps a random set of pages out of a whole quarter’s SEO changes, which answers the “what did SEO deliver?” question directly; contamination builds over time, so re-randomise it each period. A reverse holdout removes an existing module from a random 10% of pages to test whether it still earns its maintenance cost. That can be an easier sell than withholding something new, because it is framed as a maintenance question.

    Reading the result

    • A clear positive gap: release the holdout, and use the interval, not the point estimate, to size the business case for other templates.
    • A narrow interval around zero: the change is unlikely to be worth its maintenance cost; consider removing it.
    • A negative gap: roll back. The holdout has just paid for itself.
    • Both groups move together after launch: suspect site-level effects, spillover or an outside event, and compare against an untouched section or market before concluding anything.

    The rule I would use

    Hold out pages for any template-level change where the outcome is uncertain, the change is page-local enough for a holdout to see it, and an A/A run shows the interval will be narrow enough to act on. Size it at 10 to 20%, price it in money with an end date, freeze the held pages, and read the gap as conservative for anything with site-wide reach. Do not hold out fixes with a clear mechanism, or anything touching accessibility, security or legal obligations. Verify indexing fixes with URL Inspection and the Page indexing report, ship the rest, and monitor.

    Need help with this on your own site?

    Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.

    See SEO services

Add a reply

Replies are open to approved members. Log in or apply to join.