Discussion starter: an enterprise SEO question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.
- Enterprise SEO
- SEO Testing
Which changes have you seen repeatedly test flat in enterprise SEO split tests, despite being standard best practice?
Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.
1 reply
-
adminAdmin
The changes that come back flat most often are the ones that do not change what Google can match a page to: swapping structured data formats, adding alt text to listing images, heading-hierarchy clean-ups, meta description rewrites, schema that produces no visible result, and generic copy blocks on templated pages. Published results back up part of that list. SearchPilot has written up an ecommerce alt-text test and a microdata-to-JSON-LD test, and both were inconclusive. The mechanism explains the rest.
The important qualification is that “flat” in an SEO split test rarely means “zero effect”. It usually means the effect, if there was one, was smaller than the test could detect. That distinction drives the decision. A flat result on a cheap change with non-SEO benefits is a reason to ship it without a traffic forecast. A flat result on an expensive engineering project is a reason to stop justifying it with SEO. Neither is a reason to rerun the same test at the same power and hope.
What the public record actually shows
Public evidence here is thinner than conference talks suggest. Enterprise teams rarely publish null results, and testing vendors choose which tests to write up. Nobody can give you a reliable base rate of flat outcomes from public data. What does exist is useful because it is specific.
In one SearchPilot test, an ecommerce site added keyword-rich alt text to product images on category pages that previously had none. The write-up states: “This experiment was inconclusive, meaning we were not able to detect any impact to organic traffic with statistical confidence.” The customer deployed it anyway, on the reasoning that it improved accessibility and showed no harm.
In another, a travel site replaced microdata with equivalent JSON-LD for breadcrumb, review and offer markup. SearchPilot reported: “This change had no measurable impact on organic traffic, despite JSON-LD being Google’s preferred way of adding structured data to a page.” The same write-up adds the caveat every team should put at the top of its test log: “An inconclusive result is not proof that the change had no impact, but rather that we were unable to prove any positive or negative impact beyond a 95% confidence level.”
Those are two tests on two sites. They do not prove alt text or JSON-LD never matter. They do show that two of the most commonly recommended changes can fail to produce a detectable traffic effect on real enterprise templates.
Five reasons best practices test flat
- The effect is real but below the minimum detectable effect. Split tests compare variant pages against a forecast built from control pages. Daily organic traffic is noisy, so every test has a band inside which it cannot tell signal from noise. If that band is several per cent wide and the change is worth one per cent, the test will call it flat even though the gain exists and would be worth money at scale.
- Google already compensates for the problem you are fixing. Google parses both microdata and JSON-LD, so swapping formats changes nothing it can read. It frequently writes its own snippet rather than using your meta description. It understands images from surrounding text and product data even without alt attributes. A “fix” for something Google already works around has little room to move traffic.
- The change targets a surface the primary metric barely measures. Alt text mostly affects image search. If a category template earns nearly all its sessions from web results, an image-search gain disappears inside the total. Meta descriptions affect click-through rate, not rank, and only on queries where Google shows your text.
- Google has not seen the change on enough pages. On large sites, long-tail pages may be recrawled every few weeks. A three-week test where only a minority of variant pages were refetched is measuring a partial deployment. Changes injected with JavaScript also need rendering before they count.
- Effects cancel across segments. A change can help tail queries and hurt head queries, or lift click-through rate while nudging position down. The aggregate reads flat while two real effects offset each other.
The usual flat-testers, and when each one is not flat
Change Why it tends to test flat When it can move Ship it anyway? Microdata to JSON-LD (same entities) Google reads both; nothing new becomes eligible The old markup was broken, or the new markup makes a rich result eligible that actually shows for your queries Yes, if it simplifies maintenance; otherwise bundle it with a template rebuild Alt text on listing and category images Web search dominates the traffic; image context already exists Image-led verticals where Google Images drives real visits; measure image search separately Yes. Accessibility is reason enough, and the European Accessibility Act has applied to many EU consumer-facing digital services since June 2025 Heading hierarchy clean-ups (single H1, ordered H2 and H3) The words on the page barely change When the edit also changes the main heading’s wording to match intent better Yes, if cheap Meta description rewrites No ranking role; Google often generates its own snippet Templated pages where descriptions were missing or poor and Google does show yours Usually, but judge it on query-level CTR, not sessions Schema with no visible result No change on the results page When it enables a feature that is still supported and shown, such as merchant listing details Only if another consumer uses it. FAQPage markup, for example, has produced no Google rich result since May 2026 Generic “SEO copy” blocks on category templates Restates what the product grid already signals and pushes products down When the text adds information users need, such as sizing, compatibility or comparisons Test it; do not treat it as a default Core Web Vitals work, measured as a ranking change Field data is a rolling 28-day aggregate and Search Console groups similar URLs, so page-level test buckets blur. Relevance also outweighs page experience Pages failing badly, where conversion and engagement effects are larger Yes for users; measure it on conversion, not rank The contrast group is worth noticing. Changes to title links, the main content, internal links to under-linked pages, and anything that alters what gets indexed change what a page can match or how often Google finds it. Those are more likely to produce detectable movement, in either direction, which is exactly why they deserve most of your test capacity.
How to read a flat result before you file it
Most flat results in enterprise test logs are recorded as a verdict (“no impact”) when they should be recorded as a measurement. Before you close a test, check the following.
- Record the interval, not the verdict. “Somewhere between minus 2% and plus 3%” and “somewhere between minus 8% and plus 9%” are both inconclusive, but only the first tells you the upside is small.
- Confirm Google saw the change. Use log files to check what share of variant pages Googlebot fetched after deployment, and confirm the rendered HTML contains the change. If most variant pages were not recrawled, the test is incomplete, not flat.
- Check for contamination. Internal links between buckets, shared components, and CDN caching that served control HTML to Googlebot can all wash out a real effect.
- Check the window. A core update during the test (Google completed both a March 2026 and a May 2026 core update) hits both buckets, but not necessarily equally.
- Segment before concluding. Split by template, device and query type. Search Console’s branded queries filter helps separate branded demand, which a page-level change rarely affects, from non-branded queries where it might.
- Look at the metric the change was designed to move. Alt text should be judged on image search clicks, meta descriptions on CTR for queries where your snippet shows, and schema on rich result impressions. A flat session count can hide a clear secondary effect.
A decision rule for the test roadmap
Was the test powered to detect the smallest effect you would care about?↓No: call it underpowered, not flat. Decide on cost and non-SEO value, or retest only with more pages or a longer window.↓Yes: is the confidence interval tight around zero?↓Yes: stop using SEO to justify the change. Ship it if it is cheap or helps users, and stop spending test slots on that category.↓No, it leans one way: pool it with similar tests or rerun on a larger template before deciding.Hypothetical example:
An enterprise retailer has 60,000 category and product pages. The SEO team asks for six weeks of engineering to migrate all structured data to JSON-LD and generate alt text for every product image. Split tests on one template return inconclusive results for both, with intervals of roughly minus 3% to plus 3%.
A sensible response: ship alt text through the product data pipeline because it is cheap and improves accessibility, defer the JSON-LD migration until the templates are rebuilt anyway, and give the freed test slots to title link and internal linking tests, where the chance of a detectable effect is higher. Nobody writes “alt text does not work” into the playbook. The note reads “effect on web organic sessions smaller than about 3% on this template”.
If I were planning an enterprise test roadmap, I would treat hygiene changes as deployment decisions and keep split tests for changes that alter matching, discovery or the snippet. For anything that has already tested flat, the question to answer is not “does this work?” but “is the upside large enough to be worth a test slot, given what the interval already rules out?” If it is not, ship it on cost and user value, write down the interval, and move on.
Need help with this on your own site?
Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.
Add a reply Cancel reply
Replies are open to approved members. Log in or apply to join.