Discussion starter: an AI search question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.
- AI Search
Prompt-tracking tools give different AI visibility scores for the same brand on the same day. How are you handling run-to-run variance before reporting?
Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.
1 reply
-
adminAdmin
Two tools disagreeing on the same day and one tool disagreeing with itself are different problems, and they need different fixes. Differences between tools are mostly definitional: each tool runs different prompts, on different platforms and modes, from different locations, and scores the output with its own formula. Differences between runs are sampling noise, because AI answers are generated probabilistically and retrieval shifts from one run to the next.
My handling before anything goes in a report: pick one instrument and freeze it, report rates from many runs with a range attached, measure the noise floor on an untouched baseline, and only call a movement real when it clears that floor, persists, and is not mirrored in a control set. Never average two tools’ scores together, and never report a single run.
Two problems hiding in one complaint
An analogy helps here. Two opinion pollsters surveying the same election on the same day publish different numbers because they word questions differently, sample different people and weight the results differently. Analysts call those “house effects”. Within one pollster, week-to-week wobble is sampling error. Nobody sensible averages two pollsters without understanding their methods, and nobody reports a two-point move in one poll as a shift in public opinion without checking the margin of error. AI visibility tools behave the same way, with one difference: most of them publish far less about their method than a reputable pollster does.
Source of difference What it looks like How to handle it Prompt set Tool A tracks 40 prompts you wrote, tool B generates 200 variants Treat as different metrics. Compare trends within each tool, not levels across tools Branded prompts included One score is inflated because prompts name the brand, so it appears almost every time Report branded and unbranded prompts separately Platform and mode One tool pools ChatGPT with AI Overviews; another reports them separately, or runs with web search off Report per platform and mode. Pooled scores hide opposite movements Collection method API calls versus scraping the logged-out consumer interface, which can use different retrieval and system settings Ask the vendor which method each platform uses, and keep it constant Location and language Different country or city settings change retrieved sources Fix the market setting and record it Scoring formula A 0 to 100 “visibility score” blending mentions, position, sentiment and citations Use the underlying counts. If the formula is not disclosed, do not use the composite as a KPI Run-to-run randomness Same tool, same prompt, different answer an hour later Repeated runs, rates with intervals, a measured noise floor Only the last row is true variance. The rest is two instruments measuring different things and labelling both “AI visibility”.
How unstable single answers actually are
The best public evidence here is the SparkToro and Gumshoe study from late 2025: 600 volunteers ran 12 prompts through ChatGPT, Claude and Google’s AI tools 2,961 times. For ChatGPT and Google’s AI, they put the chance of getting the same list of brands in two responses at less than 1 in 100, and closer to 1 in 1,000 for the same list in the same order. List length ranged from 2 or 3 recommendations to 10 or more. Yet across many runs, how often a brand appeared was far steadier, and the authors concluded that visibility percentage across many prompts and runs is a reasonable metric while an AI “ranking position” is not.
That tells you what to report (frequency, not position) and what not to trust (any single answer, and any positional metric). The authors also say plainly that the number of runs needed for statistically sound comparisons is still an open question. Nobody has a published, validated minimum, so you have to size your own noise.
How much noise to expect from sample size alone
A mention rate is a proportion, so ordinary sampling arithmetic applies. The table below assumes a brand mentioned in 30% of answers and treats each answer as independent. It is arithmetic, not data from any tool.
Answers collected per period 95% range around an observed 30% Period-on-period change needed before it clears noise 50 19% to 44% about 18 points 100 22% to 40% about 13 points 200 24% to 37% about 9 points 400 26% to 35% about 6 points 800 27% to 33% about 4.5 points 1,600 28% to 32% about 3 points The last column is a floor. To reliably catch a change of that size, rather than just recognise it when it happens, you need roughly twice the sample. And real answers are not independent. Runs of the same prompt on the same day share the same model version and the same retrieval results, so 100 runs on a Monday carry less information than 20 runs on each of five days. Treat the true thresholds as wider than the table.
Two design choices follow. Spread runs across days and times rather than batching them. And if you want to claim something about the brand overall, not just about a handful of prompts, you need breadth: 100 runs of five prompts gives you precise estimates of those five prompts and very little about the category.
Cost check:
Answers multiply quickly: 50 prompts, 3 platforms and 4 runs a week is 600 answers a week. Most tools meter by prompts or credits, so size the plan against the change you need to detect, not against the number of prompts that feels thorough.
The routine I would run before a number goes in a report
- Freeze the instrument. One tool per reporting line, a fixed and versioned prompt set, fixed platforms and modes, fixed market and language, logged-out state. Record the vendor’s changelog, and break the trend line whenever the vendor changes method.
- Define the metrics before collecting. Mention rate (brand named), citation rate (your domain linked as a source) and share of voice against a named competitor set are separate numbers. Keep answer accuracy as its own check.
- Aggregate to a reporting period. Weekly or monthly rates, not daily scores. Report each rate with its range.
- Measure the noise floor directly. Before claiming anything, run the frozen set for three or four periods while nothing changes on your side. The spread you see between those periods is your practical noise floor, and it includes everything the textbook formula misses.
- Run a control set. Keep a group of prompts your work should not affect, such as a neighbouring category or queries where your pages are not relevant. If the target set and the control set move together, suspect a model or retrieval change on the platform side rather than your work.
- Apply a decision rule. A change is reportable when it exceeds the noise floor, holds for two consecutive periods and is not matched by the control set. Everything else is reported as “no detectable change”, which is a legitimate result.
- Keep raw answers. Stakeholders trust a range more when they can read sample answers behind it, and raw outputs are how you catch factual errors about the brand, which often matter more than the rate.
Movement in mention rate this period↓Larger than the measured noise floor?↓Held for two periods?↓Control set flat?↓Report as a change, then look for the causeA “no” at any step means the movement goes in the report as noise, or as a platform-wide shift if the control set moved too.
Cross-check against data the platforms log themselves
Prompt tracking samples answers you requested. A few sources record what happened to real users, and they are useful for triangulation even though none of them is complete:
- Search Console’s generative AI performance reports show impressions for AI Overviews, AI Mode and generative AI features in Discover, filterable by page, country, device and date. There are no clicks, CTR or queries, and the reports reached all eligible sites on 31 August 2026.
- Bing Webmaster Tools’ AI Performance report shows citations, cited pages and a sample of grounding queries across Copilot and Bing AI summaries, with no clicks.
- GA4’s AI Assistant channel group captures some referral traffic from assistants such as ChatGPT, Gemini and Copilot, although traffic without a referrer still lands in Direct.
If a tracker says Google AI visibility rose sharply while AI Overview and AI Mode impressions in Search Console stayed flat for the same pages, I would treat the tracker’s movement as unconfirmed. The two measure different things, but a real, broad improvement should usually show up somewhere in first-party data.
What a defensible report line looks like
Hypothetical example:
A software brand tracks 40 unbranded prompts in ChatGPT with web search on, five runs per prompt spread across the month (200 answers). This month the mention rate is 31% (range 25% to 38%). Last month it was 27% (range 21% to 34%). The four-month baseline spread was about 7 points. The control set rose 3 points over the same period. Reported as: “No detectable change. Small upward drift consistent with platform-wide movement.” That line is less exciting than “visibility up 15%”, but it survives someone rerunning it.
When leadership already has two tools showing two numbers, do not pick the flattering one. Put a short crosswalk in the report: prompt counts, platforms, collection method and scoring formula for each tool. Once people see the definitions side by side, the disagreement usually stops being alarming, and the conversation shifts to which definition matches the business question.
A quick test to size your own noise floor
Take your current prompt set, change nothing on the site, and run it on your normal schedule for four weeks. Metric: the week-to-week spread in mention rate per platform. Target: a spread smaller than the smallest improvement you would want to claim in a monthly report. If the spread is larger, increase answers per period, report monthly instead of weekly, or narrow claims to platform-level trends. If a tool cannot tell you how many runs sit behind its score or show you the raw answers, it is fine for exploration but should not carry a client KPI.
The practical position: treat tool disagreement as a definitions problem and run variance as a sampling problem. Solve the first by freezing one instrument and documenting it, and the second by collecting enough answers, measuring your own noise, and being willing to report “no change” when that is what the data says.
Need help with this on your own site?
Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.
Add a reply Cancel reply
Replies are open to approved members. Log in or apply to join.