Discussion starter: an SEO measurement question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.
- SEO Analytics
Which statistical mistakes do you see most in published SEO case studies, and how should practitioners raise the bar on evidence?
Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.
1 reply
-
adminAdmin
The most common mistakes in published SEO case studies are not exotic statistics errors; they are basic design problems. No control group, so seasonality, algorithm updates and demand changes are credited to the intervention. Several changes shipped at once, with the result attributed to the one being promoted. Pages chosen because they were performing badly, which then improve by regression to the mean. Short windows, cherry-picked winners and percentages on tiny bases. And, since September 2025, comparisons that cross Google’s removal of the
&num=100parameter without mentioning that impressions and average position changed for reporting reasons. Raising the bar means reporting the design and uncertainty, not just the result.Common mistakes and what they do
Mistake Effect No control group or comparison period Seasonality, updates and demand shifts credited to the change Multiple changes at once Impossible to know which change mattered Regression to the mean Pages chosen for poor performance improve naturally, making any fix look effective Cherry-picking Only successful pages or sites are reported Small samples Random variation presented as an effect Percentages on tiny bases “300% growth” from 10 to 40 clicks Correlation presented as causation Factors that coexist with rankings presented as causes Ignoring reporting changes Metric shifts from tool or platform changes read as performance Using average position across all queries Query mix changes read as ranking changes Short time windows Early fluctuations reported as lasting effects How to read a case study critically
- Was there a control group or a comparable untreated set of pages?
- What else changed during the period, on the site and in search?
- How were the treated pages chosen?
- How large is the sample, and how large is the effect in absolute terms?
- How long was the observation period, and does the effect persist?
- Are uncertainty ranges or significance tests reported?
- Would the authors have published if the result had been negative?
What a stronger case study includes
- The hypothesis and the change, described precisely.
- How pages or sites were selected, ideally randomly assigned to test and control.
- Baseline data for both groups before the change.
- The time window, with reasons for its length.
- All other changes and external events during the period.
- Results in absolute and relative terms, with uncertainty ranges.
- Limitations stated plainly.
Hypothetical example:
A case study reports that rewriting 20 underperforming product descriptions increased their traffic by 150% in six weeks. A closer look shows the 20 pages were chosen because they had dropped sharply the previous month, the rewrite shipped alongside a site speed improvement, and the six weeks included the start of peak season. A better version would have randomly assigned 200 underperforming pages to rewritten and unchanged groups, shipped nothing else to those templates, and compared both groups over the same period. It might well have found a smaller effect, and that smaller, reliable number would be more useful to readers than the headline figure.
Watch for this:
Publishing only positive results creates a misleading picture of what works across the industry. Null and negative results are informative; practitioners and publications raise the bar when they share them.
Regression to the mean deserves special attention
This mistake is common enough to deserve a closer look. Many SEO projects start by selecting pages that “dropped” or “underperform”. Some of those pages were unusually low for random reasons, such as a temporary ranking fluctuation, a seasonal dip or a data glitch, and would have recovered anyway. When the team improves them and they rise, the recovery looks like proof the work succeeded. The only reliable guard is a control group: select a larger set of underperforming pages, change only half, and compare. If both halves recover, the work did not cause the recovery.
What the industry can do
Practitioners can pre-register tests internally, share method and uncertainty with every result, and publish tests that found nothing. Publications and conference organisers can ask for control groups and method details before featuring case studies. Readers can ask the questions above in comments and discussions. None of this requires advanced statistics; it requires treating SEO claims with the same care as any other business evidence.
Look for control groups, isolated changes, random selection, adequate samples and time windows, absolute numbers, stated uncertainty and awareness of reporting changes. Apply the same standards to your own case studies before you publish them.
Need help with this on your own site?
Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.
Add a reply Cancel reply
Replies are open to approved members. Log in or apply to join.