Discussion
  • AI Search

A client’s AI visibility drops right after a model upgrade. How do you separate the model change from content or competitor changes?

Published 1 reply

Discussion starter: an AI search question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.

Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.

1 reply

  • admin
    Admin
    Original poster

    Start from the assumption that a drop timed with a model upgrade is probably the upgrade, then look for evidence that would prove otherwise. A model change tends to leave a recognisable signature: it hits many brands and prompts at once, on or near a known date, on one platform, and it changes the shape of answers or the types of sources cited, not just your presence. A content or competitor problem looks different: it is specific to your brand or a few topics, it often shows up on several platforms, and it can usually be traced to something that changed on your site, a competitor’s site or the web around you. Comparing your visibility with a competitor set, across platforms and against your own change log, separates the two in most cases.

    The signatures to compare

    Signal Points to a model change Points to your content or competitors
    Timing Aligns with an announced or observed model release Aligns with a site release, content change or competitor launch
    Breadth Many brands in your tracking set move at once Mainly your brand moves
    Platforms Only the upgraded assistant changes Several assistants change
    Answer shape Answers get longer or shorter, list more or fewer brands, cite different types of source Answers look the same, with a competitor or third party in your place
    Crawler activity Unchanged Drop in the provider’s search crawler fetching your pages

    A diagnostic sequence

    1. Look at relative share, not absolute presence. If every brand’s appearance rate fell because answers now name fewer brands, your share of mentions may be unchanged. That is a model effect on the metric, not a loss of standing.
    2. Check other platforms. If ChatGPT dropped but Gemini, Perplexity and AI Mode did not, the cause is probably specific to ChatGPT’s change.
    3. Compare cited sources before and after. If the new model cites more forums and fewer brand sites for everyone, that is a retrieval or ranking change on the platform. If one competitor’s pages replaced yours, look at what they published.
    4. Check your own change log. Site releases, robots.txt edits, CDN or firewall rules affecting AI crawlers, removed or merged pages, and changes to pricing or product pages around the date.
    5. Check crawler logs. A fall in verified requests from the provider’s search crawler after a configuration change is a sign you blocked something, not that the model changed its mind.
    6. Check the wider web. New negative coverage, a competitor’s analyst report, a popular community thread or changes to review sites can shift what assistants say.

    Keep a stable control set

    Tracking is much easier to interpret with a small set of control prompts where your presence has been stable for months, alongside the prompts you care about commercially. If the control prompts fall too, something broad changed. If only a topic cluster fell, look for a topic-specific cause. Annotate model releases, your own site releases and competitor launches on the same chart, so the timeline is visible to everyone who reads the report.

    Hypothetical example:

    A client’s appearance rate in ChatGPT answers for its category falls sharply in the week of a model update. Across the tracked competitor set, every brand’s appearance rate fell, because the new model lists three recommendations instead of five or six. The client’s share of the mentions that remain is about the same as before, and its visibility in AI Mode and Perplexity is unchanged. The agency reports the change as a platform effect, adjusts the metric to share of mentions, and recommends no content changes. Two months later, a genuine drop affects only the client on all platforms after a competitor publishes a widely cited comparison study; that one does need a response.

    Retrieval change or generation change?

    For assistants that search the web, it helps to ask whether the model is finding different sources or using the same sources differently. If your pages are still cited but your brand is no longer named in the answer text, the generation step changed. If your pages are no longer cited at all, retrieval changed: the queries the assistant generates, the search provider’s results or the crawler’s access. Each points to different actions. A generation change is usually outside your control; a retrieval change may be something you can influence through content, access or the search results that feed the assistant.

    Watch for this:

    Reacting to a model change by rewriting content is a common waste of effort. Models are updated and adjusted frequently, sometimes within weeks. Unless the evidence points to your own content or a competitor, wait for results to stabilise before changing anything, and keep measuring.

    How to report it to the client

    Explain the change in terms of what moved across the market, what moved for the client specifically, and what, if anything, they can do about it. Show share of mentions alongside appearance rate, and show platform-by-platform results rather than a single blended score. Clients accept platform effects when the report shows competitors were affected too; they lose confidence when a drop is explained away without evidence.

    Treat the upgrade as the leading suspect, test it against competitors, other platforms and cited sources, rule out your own changes and crawler access, and act only on the part of the drop that belongs to the client.

    Need help with this on your own site?

    Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.

    See SEO services

Add a reply

Replies are open to approved members. Log in or apply to join.