Discussion starter: an AI search question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.
- AI Search
Vector similarity scoring tools claim to predict AI retrieval. Has anyone validated those scores against real citation outcomes?
Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.
1 reply
-
adminAdmin
There is no independent validation showing that vector similarity scores reliably predict which pages AI systems retrieve and cite. The tools calculate how semantically similar a page or passage is to a query using an embedding model, which is a reasonable proxy for relevance. But the systems behind AI answers use their own, undisclosed models and pipelines, and they combine relevance with search rankings, freshness, source authority, access and other factors. A page can be highly similar to a query and never cited because it does not rank or is not trusted; a page with a moderate score can be cited because it is the best source. Treat these scores as a relevance diagnostic at most, and validate them against your own citation data before relying on them.
What the scores measure and what they miss
Factor Captured by similarity scores? Topical relevance of text to a query Yes, approximately The specific embedding model a given AI system uses No Whether the page ranks in the searches the assistant runs No Source authority and trust No Freshness No Crawler access and indexing No Whether the page contains a unique, specific answer Partly How to validate a tool on your own data
- Collect citation data: run a fixed set of prompts across the assistants you care about, repeatedly, and record which pages are cited, including your own and competitors’.
- Build a candidate set: for each prompt, the pages that could reasonably have been cited, including those that were not.
- Score every candidate with the tool.
- Test predictive power: do cited pages have meaningfully higher scores than uncited ones? A simple measure is how well the score ranks cited pages above uncited ones.
- Control for rankings: check whether the score adds anything beyond the page’s search position for the relevant queries. If search ranking predicts citations as well or better, the similarity score is adding little.
Where the scores can still be useful
- Spotting pages whose content does not match the queries they target.
- Checking whether sections of a long page are clearly focused.
- Grouping content by topic and finding overlaps or gaps.
These are editorial uses. They help you find pages worth reviewing; they do not tell you what an AI system will do.
Hypothetical example:
A B2B company pilots a tool that scores pages for AI retrievability. It tests the tool against three months of prompt tracking data across 80 prompts. Cited pages have slightly higher scores on average, but the scores separate cited from uncited pages only a little better than chance, and much less well than simply whether the page ranked in the top five for the matching search query. The company keeps the tool for spotting unfocused pages, drops the plan to rewrite content to raise scores, and focuses on pages that rank and contain specific, citable information.
Watch for this:
Optimising content to raise a similarity score can lead to repetitive, keyword-dense writing that reads worse and may not change retrieval at all. If a tool’s recommendations make content less useful to people, ignore them.
Why rankings are the baseline to beat
For assistants that search the web, the pages they can cite are largely the pages that rank for the searches they run. That makes search ranking a strong, simple predictor of citation, and any new score has to show it adds information beyond it. If a vendor cannot show that its score predicts citations among pages with similar rankings, the score may simply be re-measuring relevance that ranking already captures. Asking vendors for that specific analysis is a quick way to judge whether a tool is worth piloting.
Better uses of the same budget
The money spent on similarity scoring might go further on prompt tracking with good source recording, content that adds specific information to pages that already rank, and work that earns mentions in the independent sources assistants cite. Those have more direct links to what assistants actually do, and their effects can be measured with the same citation data you would use to validate a scoring tool.
What would convince me
Independent validation across many sites and topics, showing that the score predicts citations beyond what search rankings and authority already explain, with the method published. Vendor case studies without controls, or correlations measured only on the vendor’s clients, do not meet that bar. Until such evidence exists, the scores are a hypothesis to test on your own data, not a metric to manage.
Validate any similarity scoring tool against your own citation data and compare it with simpler predictors such as search rankings, use it for editorial diagnosis if it helps, and do not rewrite content to chase a score that has not been shown to predict AI citations.
Need help with this on your own site?
Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.
Add a reply Cancel reply
Replies are open to approved members. Log in or apply to join.