Discussion
  • Enterprise SEO

For SEO business cases, which forecasting method has held up when finance checked back a year later: CTR models, analogues, or staged tests?

Published 1 reply

Discussion starter: an enterprise SEO question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.

Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.

1 reply

  • admin
    Admin
    Original poster

    No public dataset compares CTR models, analogues and staged tests on 12-month forecast accuracy, so nobody can honestly hand you a league table. What you can compare is how each method fails. Staged tests usually hold up best for the part of the forecast they actually measure. Analogues are the best check on size and timing. CTR models are the most likely to miss, because they multiply several assumptions together, and the assumption that has drifted most since 2023 (click-through rate at a given position) sits in the middle of the calculation.

    The more useful point is that a look-back rarely fails on the choice of model alone. It fails on a forecast that never stated its counterfactual, assumed the work would ship on time, or quoted one number where it should have given a range. A forecast built from mediocre methods but with explicit assumptions survives review far better than a sophisticated one that cannot be audited.

    Why CTR models drift the fastest

    A CTR model is a chain: keyword volume, multiplied by the ranking you expect to achieve, multiplied by the CTR at that ranking, multiplied by conversion rate and value. Each link carries its own error, and the errors compound. Being 30% optimistic on three separate inputs produces a forecast roughly 2.2 times too high, even though each assumption looked reasonable on its own.

    Three of those links have moved materially in the last few years:

    • CTR at a fixed position. Ahrefs’ February 2026 analysis found the presence of an AI Overview correlated with a 58% lower average CTR for the top-ranking page. The less discussed finding matters more for forecasting: on informational keywords without an AI Overview, position-one CTR fell from 0.076 in December 2023 to 0.039 in December 2025. A CTR curve built in 2023 would roughly double-count clicks even on queries with no AI feature.
    • The impression baseline. When Google stopped supporting the num=100 parameter in September 2025, many sites saw desktop impressions fall and average position improve, while clicks were affected far less, according to Brodie Clark’s analysis. Any model that sizes the prize from Search Console impressions straddling that date is using two different counting regimes.
    • The ranking assumption. This is usually the largest error and the least discussed. “We will reach position 3” is a hope expressed as an input. The model gives no way to estimate how likely that is.

    CTR models still have a job. They are the right tool for sizing a ceiling (“if we won this query set outright, what is the most it could be worth?”) and for ranking opportunities against each other. They are the wrong tool for a number finance will hold you to.

    Analogues: good at size and timing, weak on selection

    An analogue forecast asks what happened last time something similar was done, rather than building the answer bottom-up. This is reference class forecasting, the approach Kahneman and Lovallo described and Bent Flyvbjerg applied to infrastructure projects. It exists because bottom-up plans are systematically optimistic about both size and speed.

    Analogues capture things CTR models miss: how long indexing and ranking actually took, how often a launch underdelivered, how much of the traffic converted. They fail in two predictable ways.

    • Survivorship bias. External case studies are published because they worked. An analogue set built from conference talks and vendor blogs is a set of winners.
    • Cherry-picking. Choosing the one past launch that most resembles the plan (usually the best one) turns the method back into optimism. The discipline is to include every comparable past project, then use the median and the spread.

    Internal analogues beat external ones, even when you only have a handful. Your own last three page-type launches reflect your domain, your engineering speed and your market. A published result from a different site reflects none of those.

    Staged tests: the most defensible number, within a narrow scope

    A staged test ships the change to part of a template (or a subset of pages or markets), holds a comparable group back, and measures the difference. Of the three methods, it is the only one that measures your change on your pages against a counterfactual. That is why it tends to survive a look-back.

    It also delivers a base rate most business cases ignore. SearchPilot reported that across its platform in 2022 and 2023, “Nearly 75% of tests are inconclusive. 7-8% of tests are negative and statistically significant. About 15% of all tests run are positive and statistically significant.” These are tests run by teams who believed in their hypotheses. If your business case assumes every proposed change is positive, it overstates expected value before any other error enters the picture.

    Staged tests have real limits:

    • They need enough similar pages and traffic to detect an effect. A 40-page site cannot split-test a template.
    • Some of the biggest SEO investments cannot be tested before commitment: migrations, new page types, domain consolidations.
    • A four to eight week result extrapolated to twelve months assumes the effect persists, and assumes the change behaves the same at full rollout as it did on half the template.
    • The result is a range, not a point. Quoting the central estimate and dropping the interval throws away the part finance needs.

    How the three methods compare under a look-back

    Method Estimates well Usually breaks on What finance can audit later Best use
    CTR model Relative size of opportunities; theoretical ceiling Ranking achieved, CTR drift, SERP layout, impression baseline changes Each input separately, if you logged them Prioritising and sizing the upper bound
    Analogues Realistic magnitude and time to value Survivorship bias, cherry-picked comparisons, different site context Whether actuals fell inside the historical range Setting the central estimate and the timeline
    Staged test Causal effect of a specific change on your pages Small sites, untestable projects, extrapolating short tests to a year Test versus control difference with a confidence interval Committing money to rollout; revising the forecast

    What makes a forecast survive the twelve-month review

    Failed look-backs tend to trace back to the same few causes, and most of them have little to do with the model. Build these in from the start:

    1. Agree the counterfactual before launch. Finance will compare actuals against last year, against trend, or against nothing. Pick one with them upfront. Where you can, keep a holdout: untouched pages, a market, or a section with similar seasonality. Where you cannot, a synthetic control approach (Google’s open-source CausalImpact package is the common starting point) is better than a raw year-on-year comparison.
    2. Forecast conditional on shipping. “If deployed by March” is a different forecast from “if deployed by September”. Engineering slippage can explain a large part of a miss, and it is not an SEO error. Put the ship date in the forecast so the look-back separates delivery delay from wrong assumptions.
    3. Give a range with named assumptions. A low, central and high case, each tied to a stated ranking, CTR and conversion assumption, lets finance see which assumption broke rather than concluding “SEO forecasts are unreliable”.
    4. Log confounders as they happen. Core updates (the March and May 2026 updates both landed this year), SERP feature changes and site releases all belong on the timeline. Search Console’s custom annotations make this cheap.
    5. Forecast the business metric. Clicks are an intermediate output. Finance will check leads, revenue or pipeline, and a click forecast that hits while conversion falls is still a miss in their eyes.

    Hypothetical example:

    A retailer proposes 400 new category landing pages. A CTR model, using current rankings of similar pages and today’s CTR by position, puts the ceiling at 60,000 extra monthly clicks. The team’s three previous page-type launches delivered between 25% and 45% of their modelled ceiling after twelve months, with a median of 35%. The business case therefore uses 21,000 clicks as the central estimate, 15,000 to 27,000 as the range, and a nine-month ramp based on how long past launches took to plateau.

    Rollout is staged: 100 pages first, with 100 matched pages held back for eight weeks. If the tested pages beat control by less than half the planned effect, the remaining 200 pages are re-scoped before more engineering time is committed. Twelve months later, finance can check the test result, the ramp assumption and the analogue range separately.

    Choosing the method for a given business case

    Can the change be applied to part of a template or market with a comparable holdout?
    ↓
    Yes: forecast from analogues, then stage the rollout and let the test result replace the forecast before full commitment
    ↓
    No: do you have at least a few comparable past projects of your own?
    ↓
    Yes: use their median and spread as the central case, with the CTR model as a stated ceiling
    ↓
    No: CTR model with an explicit haircut, phased funding gates, and a look-back date agreed with finance

    The deciding factor is reversibility. A testable template change can be funded in stages, so forecast error costs little. A migration or a new page type commits most of the cost upfront, so the forecast has to carry more weight, and the honest response is a wider range and earlier checkpoints, not a more elaborate model.

    A fast way to find out which method works for your organisation

    The best evidence you can get on this question is your own track record, and most teams have never scored it. Pull every SEO business case from the last two or three years. For each, record the method used, the forecast, the ship date against the plan, and the actual result at twelve months. Calculate actual divided by forecast and group the results by method.

    If CTR-model forecasts consistently landed at 30% to 50% of plan, you have both a calibrated haircut and a reason to stop presenting them unadjusted. If analogue forecasts landed within their stated range most of the time, lead with them. If most misses trace back to late shipping rather than wrong assumptions, the fix is in delivery planning, and changing the forecasting method will not help.

    If I were building the next business case, I would set the central estimate from internal analogues, use the CTR model only to show the ceiling, and fund in stages wherever a holdout is possible, so that measured results replace the forecast before most of the money is spent. The method finance ends up trusting is whichever one lets them see, a year later, exactly which assumption was wrong.

    Need help with this on your own site?

    Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.

    See SEO services

Add a reply

Replies are open to approved members. Log in or apply to join.