Discussion
  • SEO Analytics
  • SEO Testing

How long do you run an SEO test before calling it: fixed duration, a statistical threshold, or until stakeholders lose patience?

Published 1 reply

Discussion starter: an SEO measurement question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.

Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.

1 reply

  • admin
    Admin
    Original poster

    Decide the stopping rule before the test starts, and base it on two things: enough time for Google to recrawl and reassess the changed pages, and enough data for the expected effect to be distinguishable from normal variation. That usually means a minimum duration in whole weeks, a check that most test pages have been recrawled, and an analysis method agreed in advance, whether a split test comparison or a time-series model such as CausalImpact. Stopping when stakeholders lose patience is the most common rule in practice and the worst one, because it stops tests at random points, often right after a noisy week that looks like a result.

    What determines the right duration

    Factor Why it matters
    Recrawl time A change has no effect until Google has fetched the changed pages; on large sites the long tail can take weeks
    Traffic volume Low-traffic pages need longer to show a reliable difference
    Expected effect size Small effects need more data to detect
    Weekly cycles Tests should run in whole weeks to avoid weekday bias
    Seasonality and events Holidays, promotions and news events can swamp a small effect
    Algorithm updates Core updates during a test can move test and control groups differently

    A pre-registered test plan

    1. Hypothesis and metric. What will change, and which metric decides the result: usually clicks, sometimes conversions.
    2. Design. Split test with matched control pages where possible; before-and-after with a time-series model where not.
    3. Minimum duration. For example, four full weeks after most test pages have been recrawled, confirmed in logs or with URL Inspection samples.
    4. Decision rule. The threshold for calling a positive, negative or inconclusive result, and the analysis method.
    5. Stopping exceptions. Conditions for stopping early, such as a clear negative effect on revenue pages, or a core update that makes the test uninterpretable.
    6. What happens next for each outcome, including inconclusive.

    Why peeking causes false results

    Checking results every few days and stopping when they look good greatly increases the chance of declaring a win that is really noise. Search data is noisy week to week, and a test that looks positive after ten days often looks neutral after six weeks. If stakeholders need interim updates, report them as “not yet conclusive” with the planned end date, or use a sequential testing method designed for repeated looks.

    Managing stakeholder patience

    • Agree the duration and decision rule in writing before launch.
    • Explain recrawl time in simple terms: the test has not really started until Google has seen the change.
    • Share interim data with a clear “too early to tell” label, or not at all.
    • Run tests on templates with enough traffic that results arrive in a reasonable time.
    • Keep a log of past tests, including inconclusive ones, so stakeholders see the process working.

    Hypothetical example:

    A retailer tests new title formats on half of 2,000 category pages. After nine days the test group is up noticeably and the merchandising director asks to roll out immediately. Logs show only about half of the test pages have been recrawled. The team holds to its plan: four full weeks after 80% of pages are recrawled. By the end, the difference has shrunk to a small, statistically weak gain on clicks, with no change in revenue. The team rolls out the change to the templates where the effect was strongest and records the result as modest, which prevents an inflated number from becoming the basis for next year’s forecast.

    Watch for this:

    A core update during a test can make results meaningless, because test and control groups may be affected differently. Annotate updates, check whether groups diverged at the update date, and be willing to extend or rerun the test rather than report a confounded result.

    Choosing the analysis method

    Where you can split pages into matched test and control groups, compare the groups over the same period; that controls for seasonality, updates and demand shifts that affect both. Randomise the split, check the groups were similar before the change, and use enough pages that a few outliers cannot drive the result. Where a split is impossible, such as a sitewide template change, a time-series approach like Google’s open-source CausalImpact package can estimate what would have happened without the change, using control series that were not affected. It relies on those control series staying stable, so choose them carefully and treat results as estimates with uncertainty, not exact measurements. Whichever method you use, decide it before launch, not after seeing the data.

    Inconclusive is a result

    Many SEO tests end inconclusive, especially for small changes. That is useful information: the change did not make a detectable difference at this scale, so the decision can rest on other grounds, such as user experience, brand or cost. Treating inconclusive results as failures encourages teams to run tests until something looks significant, which is how false findings get into playbooks.

    Set the duration from recrawl time, traffic and expected effect, run whole weeks, pre-register the decision rule and stopping exceptions, avoid peeking, and accept inconclusive as a legitimate answer.

    Need help with this on your own site?

    Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.

    See SEO services

Add a reply

Replies are open to approved members. Log in or apply to join.