Nobody can sell you an AI ranking
Ask the same model the same question twice and you get a different list of companies. Here is what can be measured instead.
multiple independent measurements · May 5, 2026 · 3 min read
A large part of this industry sells a number: your position in AI answers, tracked over time, on a panel, with a line going up. Here is what the measurements say about how still that number sits.
SparkToro with Gumshoe, November–December 2025. Twelve prompts across ChatGPT, Claude and Google AI Overviews, 2,961 total runs, 600 volunteers. Their finding, verbatim:
“there’s a <1 in 100 chance that ChatGPT or Google’s AI, if asked 100X, will give you the same list of brands in any two responses”
Identical ordering appeared roughly once in a thousand runs.
Kirsten et al., 2026. They checked how many of the pages cited one month were still cited the next: fewer than 2 in 10 for AI Overviews, against nearly half for ordinary Google results. And even with the model’s randomness switched off, asked to be as predictable as it can be, between 1 and 3 answers in 10 changed anyway.
Ahrefs. AI Overviews change the sources they cite, the text they write and the questions they ask themselves roughly every two days.
Put those together. A tool showing you fourth this week and second next week is showing you a number that moves further on its own, with nobody touching anything, than any vendor could move it in a week. Both readings fit perfectly well with absolutely nothing having happened.
This is not a flaw in the tools. It is a property of the systems being measured. A search engine draws from a ranked list that stays put. An answer engine reaches into a bag each time. The first gives you a position. The second gives you a probability. Asking what position you hold in an AI answer is like asking what the next roll of the dice will be, exactly.
So what can be measured? A rate, over repeated asks, held constant.
If you appear in 4 answers out of 40 today, and in 11 out of 40 in three months, using the same questions, the same models and the same method, that is a real difference. It is not chance, because you cancelled the chance out by asking forty times instead of once. That is the entire trick, and it is thoroughly unglamorous: repetition is what turns a coin flip into a measurement.
Three conditions have to hold for the comparison to mean anything, and any one of them breaking invalidates it:
The questions stay the same. If they change between rounds, what you are measuring is the questions, not yourself.
The models are the same. If a new version comes out halfway through, there is no way to tell what changed because of you and what changed because of it. Say so when it happens, rather than pretending it didn’t.
Every question is asked more than once, every round. Asking once cancels out nothing. We ask twice, which is the minimum that means anything, and we publish both answers, including the ones that contradict each other.
How to test a vendor on this. Ask them a single question: how many times do you ask each prompt, and do you show me the variation? If the answer is “once”, their panel is a random number generator in a nice typeface. If they cannot answer at all, that tells you something too.
And the honest limit on our own side. Even done properly, this measures visibility, not money in the till. Whether being named more often produces more customers is a separate question, and the survey literature is blunt about it: no reviewed technique shows a stable, cross-platform causal effect on downstream behavior. What we can show you is whether the models name you more than they used to. The rest is your judgment about what that is worth.
This is part of the work we do here: AI Visibility Consultant
