Discussion
  • SEO Analytics

SEO split testing on sites under 10k pages: is there any method you trust, or is it all before-and-after with confounders?

Published 1 reply

Discussion starter: an SEO measurement question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.

Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.

1 reply

  • admin
    Admin
    Original poster

    Yes, there are methods worth trusting on a site under 10,000 pages, but page count is the wrong thing to look at. Whether a real test is possible depends on how many comparable pages you have on one template, how steady their daily organic clicks are, and how large an effect you need to detect. A 6,000-page store with 2,000 product pages on one template can run a genuine randomised SEO split test. A 400-page B2B site whose leads come from 30 service pages cannot, and no tool changes that.

    The main qualification is statistical power. On small sites a well-run test can usually only detect large effects, so “no significant difference” is a common result. It normally means “inconclusive”, not “no effect”. If you plan for that before launch, a small-site test is still far more trustworthy than a raw before-and-after comparison, which cannot separate your change from core updates, seasonality, SERP layout changes and reporting changes that land in the same window.

    Why SEO tests split pages, not users

    A CRO test splits visitors: half see version A, half see version B. That design does not work for organic search because Googlebot is, in effect, one visitor. It either sees your change or it does not, and deliberately showing Googlebot something different from users is cloaking. Google’s guidance on website testing says: “Don’t show one set of URLs to Googlebot, and a different set to humans.” For user-facing tests it asks for rel="canonical" on variant URLs, 302 rather than 301 redirects, and removal of test elements once the test ends. That guidance covers CRO tests running on your site. It also explains why an SEO test has to work differently.

    An SEO split test changes a random set of pages for everyone, bots included, and leaves a comparable set untouched. The untouched pages are the control, and their job is to absorb everything that hits both groups during the test window.

    A crop trial is a useful way to picture it. You would not judge a new fertiliser by comparing this year’s harvest with last year’s, because the weather changed too. You split one field into plots, treat random plots, and compare them with untreated plots that lived through the same weather. Before-and-after SEO analysis is the year-on-year harvest comparison. In technical terms, the control group gives you a counterfactual: a forecast of what the variant pages would have done without the change, usually built from how the two groups tracked each other during a pre-period.

    The three constraints that matter more than page count

    1. Comparable units. Pages need to share a template and broadly similar intent, so one change applies cleanly to all of them. Eight thousand pages spread across 15 templates are 15 small populations, not one large one.
    2. Traffic per unit and how noisy it is. A page with two clicks a day swings wildly from day to day. Aggregating hundreds of those pages smooths the noise; aggregating 30 does not. Concentration matters as much as volume: if 40 pages carry most of a template’s clicks, the test behaves like a 40-unit test whatever the page count says.
    3. The effect size you care about. The smallest effect a test can reliably detect shrinks as you add pages, traffic and time, and grows with volatility. A small site can often confirm a large swing but will struggle to separate a small improvement from normal variation.

    There is also a timing constraint small sites often ignore. The test clock starts when Google has recrawled and reprocessed the variant pages, not when you deploy. On a site with modest crawl activity, deeper pages can take a while. Check server logs or spot-check variant pages in URL Inspection before you start counting.

    Hypothetical site shape Effective test units Realistic method What it can tell you
    Store, 6,000 URLs, 2,000 product pages on one template, clicks spread fairly evenly About 2,000 Randomised page-group test Moderate to large effects of product template changes
    Same store, but 60 category pages carry most organic revenue About 60 Staggered rollout or difference-in-differences Large effects only; treat as directional
    B2B site, 900 URLs: 500 blog posts and 30 commercial pages Blog template yes, commercial pages no Page-group test on the blog; ship-and-monitor on commercial pages Blog template changes; commercial changes stay judgement calls
    Local service business, 150 URLs Too few Holdout or reversal on a handful of pages Very little causally; decide on reversibility and cost

    Methods ranked by how much I would trust them

    Method What it needs What it controls for Main weakness
    1. Randomised page-group split test Hundreds of same-template pages with steady clicks, several months of pre-period data, stratified random buckets Shocks that hit both buckets similarly: core updates, seasonality, SERP changes, reporting changes Low power on small sites; a few outlier pages can dominate a bucket
    2. Staggered rollout Several page groups, with the change deployed in waves on different dates Any confounder that does not happen to coincide with each wave date Waves may differ from each other; needs discipline to hold later waves back
    3. Difference-in-differences or synthetic control against untouched pages A stable comparison set that did not receive the change Shared shocks, provided the comparison set genuinely behaves like the treated set Selection bias: the pages you chose to change were often already different (declining, neglected, seasonal)
    4. Reversal (on, off, on) A change Google picks up quickly, such as title tags Slow-moving confounders Lag and carry-over; unsuitable for internal links, consolidation or anything with lasting effects
    5. Before-and-after with annotations Nothing Nothing systematic Any concurrent event is indistinguishable from your change

    Staggered rollout is underused on small sites. If uplift appears in wave one pages after wave one’s date, and then in wave two pages after wave two’s date, a coincidental confounder would have to line up with two separate dates and two separate page groups. That is a much harder story to explain away than a single before-and-after step.

    For method 3, Google’s open-source CausalImpact package (Bayesian structural time series) is a common choice. It is a modelling approach, not a guarantee. It assumes your control series were not affected by the change and that the relationship learned in the pre-period still holds during the test.

    Where controls fail:

    A control group only protects you from confounders that hit both groups equally. A core update that treats product pages differently from blog posts will corrupt a test that uses blog posts as the control. That is why controls should come from the same template, and why random assignment beats hand-picked comparison pages.

    How to run a small-site test so the result means something

    1. Write down the hypothesis, primary metric (organic clicks per page from the Search Console API, or organic landing-page sessions), duration and decision rule before launch. Decide now what you will do with an inconclusive result.
    2. Pull several months of daily page-level data. Remove outlier pages whose traffic would swamp a bucket, or test them separately.
    3. Stratify pages by traffic and randomise within each band, so both buckets start with similar volume and volatility.
    4. Run an A/A check on historical data: split the pages, pretend a change happened on a past date, and run the exact analysis you plan to use. If the method “finds” a 10% effect where nothing changed, that is your noise floor. On a small site this is the single most useful step, because it tells you before launch whether the test can answer the question at all.
    5. Deploy to the variant bucket only, then confirm Googlebot has recrawled the variant pages.
    6. Use clicks as the primary outcome, with impressions, CTR and conversions as secondary. Treat impressions and average position with care: Google’s removal of the &num=100 results parameter in September 2025 moved impressions and average position for many sites, while clicks were affected far less, according to Brodie Clark’s analysis. A control absorbs that kind of reporting shift. A before-and-after does not.
    7. Do not stop the moment the chart looks good. Checking repeatedly and stopping early inflates false positives, and small samples swing further.
    8. Log test dates as custom annotations in Search Console, so later analysis does not mistake the test for something else.

    Hypothetical example:

    A store has 1,800 product pages. The top 40 carry about a third of the template’s clicks, so they are held out. The remaining 1,760 are split into ten traffic bands and randomised into two buckets of 880. An A/A analysis over the previous 90 days produces intervals of roughly plus or minus 8% with no change applied. The team then tests a new title template. A measured +3% sits inside that noise, so it is recorded as inconclusive and the pre-agreed rule applies (ship it, because it is cheap and reversible). A +15% result would have been clear enough to act on and to extend to the 40 held-out pages.

    Watch for this:

    The common failure on small sites is reading an underpowered null result as proof that a change “does nothing for SEO”, then repeating it as a rule. Report the interval, not just a verdict. “Somewhere between minus 6% and plus 10%” is honest and useful. “No impact” is neither.

    Timing risks you cannot design away

    Core updates arrive on Google’s schedule, not yours. Google completed announced core updates in March 2026 and May 2026, and also documents smaller core updates that are not announced. A randomised same-template control absorbs most of that, but an update landing mid-test can still raise volatility enough to widen your intervals. If an announced update starts during the test window, extend the test rather than reading the result on the planned date. SERP feature shifts, such as AI Overviews appearing on more of your template’s queries, hit both buckets too, whereas a before-and-after would book that CTR change against your change.

    When you should not test at all

    Many changes on a small site do not need a test. Fixing broken canonicals, indexing blockers or orphaned pages has a clear mechanism, and URL Inspection and the Page indexing report will tell you whether the fix worked. Testing earns its cost on template-level changes where reasonable practitioners disagree and the result could go either way: title patterns, content modules, internal linking modules, above-the-fold layout, structured data changes.

    When those changes cannot be tested properly, I would use published split-test results from testing vendors as priors (they come from other sites, so they inform rather than prove), favour changes that are cheap to reverse, roll out in waves so you at least get a staggered design, and keep a permanent holdout of 10 to 20% of template pages unchanged. The holdout costs a little upside and gives you a standing control. Commercial testing platforms handle bucketing and forecasting, but check their published traffic requirements first: a platform cannot add power your traffic lacks.

    My decision rule: if a template has at least a few hundred pages with steady clicks and your A/A check shows a noise floor smaller than the effect you care about, run a randomised page-group test. If not, use a staggered rollout with an untouched holdout. If neither is possible, stop calling it a test. Ship reversible changes, annotate them, and judge them against a comparison set rather than against last month.

    Need help with this on your own site?

    Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.

    See SEO services

Add a reply

Replies are open to approved members. Log in or apply to join.