Discussion starter: an SEO measurement question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.
- SEO Analytics
Causal impact models for SEO changes: which control series do you use when the whole site is affected by the change?
Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.
1 reply
-
adminAdmin
When the whole site is treated, there is no clean control inside the site, so you move the control outside the boundary of the change. The best candidates are series that share the drivers of your organic traffic (seasonality, query demand, algorithm updates) but cannot “see” your change: an untreated sister property or market, paid search demand for the same queries, Google Trends for your head terms, and, for changes that do not touch brand SERPs, your own branded clicks. Competitor visibility and “untouched” sections of the same site are usually the worst choices, even though they are the most common ones.
The qualification is that every one of these is imperfect. Choosing a control is less about finding a pure series and more about picking the least contaminated ones, then proving with placebo tests that they behaved before the change. If nothing survives those tests, the honest output is a forecast deviation with a wide interval, not a causal estimate, and it should be reported that way.
What the model actually needs from a control series
Google’s CausalImpact package (the Bayesian structural time-series approach from Brodersen and colleagues, published in the Annals of Applied Statistics in 2015) states its two assumptions plainly. It relies on control series “themselves not affected by the intervention”, and it assumes “the relationship between covariates and treated time series, as established during the pre-period, remains stable throughout the post-period”.
A useful way to picture it: the control is a thermometer outside a greenhouse. It tells you what the weather did, so you can separate the weather from the heater you just installed. Put the thermometer inside the greenhouse and it rises with the heater, so the model concludes the heater did nothing.
Technically, the model learns in the pre-period how your organic series moves with the controls, then uses the controls in the post-period to predict what your series would have done without the change. The gap between that counterfactual and the observed series is the estimated effect. If a control is itself affected by your change, the counterfactual moves with the treatment. A control that moves in the same direction as your change shrinks the estimated effect towards zero. A control that moves in the opposite direction inflates it.
The selection test:
For each candidate, ask two questions. Does it share the forces that move my organic traffic? Is there any plausible path by which my change reaches it? You want “yes” to the first and “no” to the second. Correlation in the pre-period only answers the first question.
Why competitor visibility is usually the wrong control
Competitor visibility indices look attractive because they capture algorithm volatility and demand in your niche. The problem is that a SERP is close to zero-sum. If your change lifts you from position 6 to position 3 on a few thousand queries, somebody drops. Your control then falls exactly when your treatment rises, which is the opposite-direction contamination described above, and the model overstates your gain. The same spillover applies in reverse after a failed change: competitors gain and your loss looks worse than it was.
“Untouched” sections of the same site have a subtler version of the problem. A sitewide change to navigation, templates, internal links, rendering or a domain move tends to reach every section through crawl allocation, link equity and site-level quality signals, even when the section’s own pages were not edited. Unless you can explain why a section is isolated, treat it as partially treated.
Candidate controls when the whole site changed
Control series What it captures Contamination risk Use it when Untreated sister site, country site or language version Seasonality, algorithm updates, category demand Low if the codebase and domain are separate; higher if they share templates or cross-link heavily You run several properties or markets and can delay rollout to one of them Paid search eligible impressions on matched keywords (impressions divided by search impression share) Query demand for your commercial terms Budget, bidding and match-type changes; impression share is an estimate You run paid search on terms that overlap your organic targets and bidding was stable Google Trends for head terms or categories Relative demand and seasonality Sampled and normalised data; repeat pulls can differ; daily data only for shorter ranges You have no paid data, or want an independent second demand proxy Branded query clicks (Search Console branded filter) Brand demand and site-level seasonality Treated if the change touches brand SERPs: migrations, titles, sitelinks, redirects The change targets non-brand rankings, such as category content or internal linking Organic clicks from Bing or other engines Demand plus non-Google search behaviour Treated by anything every crawler sees: content, internal links, migrations The mechanism is Google-specific, or you are measuring a Google update rather than your own change Non-search channels (email, direct, paid social sessions) Business-level seasonality and promotions Brand halo, tracking changes, campaign timing Only as a supporting covariate for promotions and seasonality, never as the main control Search Console added a branded queries filter in late 2025, which makes the brand and non-brand split far easier to build than it used to be. For the paid search proxy, dividing impressions by search impression share gives an estimate of the auctions you were eligible for, which tracks demand better than raw impressions when budgets moved.
A worked example of choosing and rejecting controls
Hypothetical example:
An Australian homewares retailer with about 25,000 indexable URLs replaces its faceted navigation and internal linking across every template on 1 July. The outcome series is daily non-brand organic clicks from Search Console.
Kept as controls: eligible paid impressions for 400 category keywords that match the organic targets, Google Trends for five head terms, and branded clicks (the change did not touch the homepage, brand titles or sitelinks).
Rejected: Bing organic clicks (Bingbot crawls the same new links, so it is treated), a competitor visibility index (zero-sum spillover), and organic impressions as the outcome (impressions shifted for reasons unrelated to the site, covered below).
The timing also matters. A 1 July launch lands straight after end of financial year sales, so the last weeks of the pre-period contain a promotional spike. The pre-period needs at least one full prior year so the model has seen that pattern before, or the launch date should move away from it.
How I would validate the controls before trusting the output
- Check the pre-period fit. Hold out the last few weeks of the pre-period, fit on the rest, and compare the prediction with what happened. If the model cannot predict a period where nothing changed, it cannot estimate what your change did.
- Run placebo interventions. The CausalImpact documentation recommends testing an imaginary intervention inside the pre-period. Several fake dates should show no effect. If they show “effects” as large as the one you are claiming, the real result is noise.
- Run each control as the outcome. Point the model at each control series with the real intervention date. A control that shows an effect at your launch date is contaminated and should go.
- Drop controls one at a time. If the estimate swings from +12% to +2% when one series leaves the model, the conclusion rests on that series, and you need to defend it specifically.
- Mark known external shocks. Google completed core updates in March and May 2026. The end of the
&num=100parameter in September 2025 cut desktop impressions and improved average position in Search Console, while clicks were affected far less, according to Brodie Clark’s analysis. AI Mode and AI Overview appearances are counted inside the Web search type totals. If your pre-period spans a level shift like thenum=100change, use clicks rather than impressions, or shorten the pre-period so the shift sits outside it. - Compare interval width with the effect you care about. If the credible interval is plus or minus 15% and the change was expected to move traffic 5%, the analysis cannot answer the question, however clean the code is.
- Write the assumptions down. The documentation also recommends stating the assumptions and priors when presenting results. In practice that means listing each control, why it should be unaffected, and what the placebo tests showed.
Watch for this:
Feeding the model dozens of loosely related series and letting it pick the best fit feels rigorous but raises the chance of spurious pre-period correlation. Three to six controls you can defend on mechanism usually beat fifty you cannot explain.
When no control survives
Sometimes every candidate is contaminated or too noisy. A domain migration with a brand change is the usual case: brand SERPs move, every crawler sees the redirects, and there is no sister site. You still have options, but you should label them for what they are.
- A forecast-only model. A structural time-series model with trend and seasonal components but no covariates estimates the deviation from what the past predicted. It cannot separate your change from anything else that happened on the same dates, so a core update in week two gets attributed to you.
- Triangulation. Pair the forecast with evidence that has a mechanism behind it: crawl logs showing Googlebot reaching newly linked pages, a fixed panel of tracked keywords split by affected and unaffected intent, and indexing status by template. Agreement across independent evidence is more persuasive than one tight interval.
- A different design next time. Most “sitewide” changes can be staged: one template, one market or one directory first, or a randomised split of comparable pages where the change is page-level. That turns the rest of the site into your control for a few weeks, which is worth more than any external proxy. Changes that are genuinely atomic (domain moves, platform swaps, HTTPS) are the exceptions.
Practitioners reasonably differ on how much weight a forecast-only estimate deserves. For a reversible change with a clear expected direction, it may be enough to justify keeping the change. For a large investment decision, such as whether to roll a platform out to other brands, I would not treat it as proof.
The decision rule I would use
Before the change ships, write down the effect size that would change a business decision, pick controls by mechanism rather than correlation, and check whether the pre-period fit and interval width can detect that effect. If they cannot, stage the rollout instead of relying on the model. After launch, keep only the controls that pass the placebo and control-as-outcome tests, report the estimate with its interval and the list of external shocks in the window, and state plainly whether the result is causal evidence or a forecast deviation. That last sentence is the one stakeholders will remember, so make it accurate.
Need help with this on your own site?
Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.
Add a reply Cancel reply
Replies are open to approved members. Log in or apply to join.