Discussion
  • Agency

Free AI visibility audits for prospects: fair sales tactic, or selling fear with numbers nobody can reproduce?

Published 1 reply

Discussion starter: an agency question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.

Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.

1 reply

  • admin
    Admin
    Original poster

    It can be either, and the dividing line is reproducibility. A free AI visibility audit is a fair sales tactic when it discloses its method, measures things the prospect could rerun and get similar results, and separates what was measured from what is being sold. It becomes fear-selling when a handful of single-run screenshots or a proprietary “AI visibility score” is presented as a diagnosis, then paired with a fix that implies guaranteed citations.

    The best public evidence on output variability points clearly at the method question. In a study published in January 2026, SparkToro and Gumshoe had 600 volunteers run 12 identical prompts through ChatGPT, Claude and Google’s AI nearly 3,000 times. The chance of getting the same list of brands twice was under 1 in 100, and the same list in the same order closer to 1 in 1,000. But how often a brand appeared across many runs was more stable, with some brands showing up in 60% to 90% of responses for a given intent. A single run is an anecdote. Frequency across many runs is a measurement.

    Why most audit numbers cannot be reproduced

    Each AI answer is a draw from a probability distribution, not a lookup in a fixed index. One run is one roll of a weighted die: it tells you what came up, not how the die is weighted. The audit’s job is to estimate the weights, and that takes many rolls under controlled conditions.

    The conditions are rarely controlled. Answers change with:

    • Sampling randomness, even for the same prompt a minute apart.
    • Model and mode: a model update, or web search switched on or off, can change which sources are retrieved.
    • Personalisation: logged-in history, memory, and opt-in features such as Google’s Personal Intelligence, which can draw on Gmail, Photos and Calendar context.
    • Location and language, which matter a lot for local and ecommerce queries.
    • Prompt wording and follow-ups: small phrasing changes shift the answer, and real users rarely ask one clean question.

    Then there is prompt selection. No AI platform publishes prompt volumes the way keyword tools estimate search demand. Bing’s AI Performance report shows a sample of grounding queries, and Google’s generative AI report in Search Console shows no queries at all. So whoever chooses the prompts controls the result. A seller who picks prompts where the competitor happens to win can produce a scary gap from nothing.

    Small samples make “you are behind” claims meaningless

    Suppose an audit runs each prompt 10 times and the prospect’s brand appears 3 times. That is reported as “30% visibility”. The honest range is much wider.

    Hypothetical example:

    Brand A appears in 3 of 10 runs and a competitor in 5 of 10. Using the Wilson score interval, a standard 95% confidence interval for a proportion, Brand A’s true rate is somewhere between about 11% and 60%, and the competitor’s between about 24% and 76%. The ranges overlap almost entirely, so “your competitor is nearly twice as visible” is not supported. At 100 runs, 30 appearances narrows to roughly 22% to 40%, which starts to be useful for tracking change over time.

    This is basic sampling arithmetic, not a special property of AI tools. It is also why month-on-month “AI visibility” reports can show movement that is pure noise. Before claiming credit for a change, check whether the week-to-week variance on an untouched prompt set is larger than the change you want to report.

    Fear patterns and fair patterns

    Element Fear-selling version Fair version
    Sample One run per prompt, screenshots Many runs per prompt and platform, rates reported with a range
    Prompts Chosen by the seller, not disclosed Drawn from the prospect’s own Search Console queries and sales-call questions; full list shared
    Metric A 0 to 100 score with no formula Separate figures for mentions, linked citations, answer accuracy and referral traffic
    Competitors Cherry-picked prompts the competitor wins The same prompt set run for every brand
    Diagnosis “Invisible because you lack llms.txt or AI schema” Documented, checkable causes such as blocked crawlers or wrong facts in answers
    Business case “AI is taking your traffic” The prospect’s own AI referral and impression data, even when it is small
    Promise “We will get you into ChatGPT” Hypotheses, what will be measured, and an explicit no-guarantee

    The diagnosis row matters because it is where audits most often overreach. Google’s May 2026 guide to generative AI features says you do not need llms.txt, special AI files, special schema or artificial chunking for Google’s AI features, and that optimising for them is still SEO. Other platforms have not published equivalent guidance, but an audit that sells those items as the fix for Google’s AI surfaces is contradicting the only primary documentation available.

    The parts of an audit that are reproducible

    The strongest free audits lead with findings that do not depend on sampling at all. They are more persuasive precisely because the prospect can check them.

    1. Crawler access. Check robots.txt and CDN or firewall rules for AI search crawlers. OpenAI documents separate crawlers: GPTBot for training and OAI-SearchBot for ChatGPT search, so blocking one does not block the other. Cloudflare announced in 2025 that it would block AI crawlers by default for new domains, and in September 2026 added a separate “Disallow AI Training” option for mixed-use crawlers. Sites that blocked AI search crawlers without meaning to are a real, verifiable finding. For Google, also ask whether the site has used Search Console’s AI features opt-out, which rolled out to all websites worldwide on 31 August 2026: opted-out sites get no traffic or impressions from AI Overviews or AI Mode, so a zero there may be a setting, not a visibility problem.
    2. Measured platform data, with read-only access: Search Console’s generative AI performance reports (impressions only, no clicks or queries, available to all eligible sites since 31 August 2026), Bing Webmaster Tools’ AI Performance report (citations and sample grounding queries, no clicks), and GA4’s AI Assistant channel group. Server logs showing AI crawler and user-triggered fetches round this out.
    3. Answer accuracy. What AI answers say about the brand’s prices, services, locations or policies that is wrong, and whether the error recurs across runs. Recurring errors are useful to a prospect whatever their visibility rate, and they often trace back to outdated pages or third-party listings the audit can name.
    4. A sampled visibility study with a method statement, clearly labelled as an estimate.

    Watch for this:

    Analytics undercounts Google’s AI surfaces. Practitioners report that clicks from AI Mode arrive in GA4 as google / organic, and AI referrals without a referrer still land in Direct. An audit that says “you get almost no AI traffic” from GA4 alone may be as wrong as one that says AI is eating everything.

    Keep the stakes in proportion

    For most businesses, AI referral traffic is still small next to search. BrightEdge data from August 2026 put ChatGPT at about 95.1% of tracked AI referral traffic, with AI still a small share of total referrals compared with search, though the methodology is not detailed. A fair audit says so, then explains why exposure without clicks (brand mentions in answers, comparison shortlists) may still matter for that prospect’s buying process. A fear audit skips straight to the exposure story and implies it is already costing revenue.

    A minimum method statement for any audit you send

    • Where the prompts came from, and the full list
    • Platforms and modes tested (web search on or off), with model names where visible
    • Dates, location, language, logged-out state and memory settings
    • Runs per prompt, and how “mention” and “citation” were defined
    • Rates reported as ranges, not single numbers
    • Raw outputs available on request
    • What the audit cannot tell you, including that visibility changes may not be caused by any work done

    If you are on the receiving end, the questions are the same: can I rerun this and get similar numbers, who chose the prompts, how many runs, and what would you measure in 90 days to show your work caused a change rather than a model update?

    A quick check agencies can run on their own offer

    Take a fixed set of 30 prompts for a past prospect and rerun them weekly for four weeks with no changes to the site. If the week-to-week spread in mention rate is larger than the improvement you would typically claim in a monthly report, your reporting cannot show impact and your audit numbers are mostly noise. Either increase runs per prompt, narrow the claims, or lead the audit with the reproducible findings instead.

    My position: a free audit is fair if it would survive the prospect rerunning it. Lead with access problems, measured platform data and factual errors, present sampled visibility as a range with its method attached, and never tie a score to a promised citation. That version still sells, and it produces clients who trust the next report.

    Need help with this on your own site?

    Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.

    See SEO services

Add a reply

Replies are open to approved members. Log in or apply to join.