Nobody can sell you an AI ranking
Ask the same model the same question twice and you get a different list of companies. Here is what can be measured instead.
multiple independent measurements · May 5, 2026 · 3 min read
A large part of this industry sells a number: your AI visibility rank, tracked over time, on a dashboard, with a line going up. Here is what the measurements say about that number’s stability.
SparkToro with Gumshoe, November–December 2025. Twelve prompts across ChatGPT, Claude and Google AI Overviews, 2,961 total runs, 600 volunteers. Their finding, verbatim:
“there’s a <1 in 100 chance that ChatGPT or Google’s AI, if asked 100X, will give you the same list of brands in any two responses”
Identical ordering appeared roughly once in a thousand runs.
Kirsten et al., 2026. Page overlap across two months: 18% for AI Overviews against 45% for organic Google. And on the surfaces where temperature could be held at zero — that is, with the model’s randomness turned off — repeated runs still changed 9 to 28% of decisions.
Ahrefs. AI Overviews change their citations, their content and their fan-out queries roughly every two days.
Put those together. A tool showing you at position four this week and position two next week is showing you a number whose expected week-to-week movement is larger than almost any effect a vendor could produce in that time. Both readings are equally consistent with nothing having happened.
This is not a flaw in the tools. It is a property of the systems being measured. A search engine returns a ranked list from an index; an answer engine samples from a distribution. The first has a position. The second has a probability. Asking for your position in an AI answer is a category error, in the same way that asking for the exact value of a dice roll is.
So what can be measured? A rate, over repeated asks, held constant.
If you appear in 4 answers out of 40 today, and in 11 out of 40 in three months, using the same questions, the same models and the same method, that is a real difference. It is not noise, because you have averaged the noise out by asking forty times instead of once. That is the entire trick, and it is unglamorous: repetition is what turns a coin flip into a measurement.
Three conditions have to hold for the comparison to mean anything, and any one of them breaking invalidates it:
The question set is frozen. If the questions change between runs, you are measuring the questions.
The models are the same. A model version update mid-period is a confound you cannot separate out. Note it when it happens rather than pretending it didn’t.
Every question is asked more than once, per run. A single ask per question does not average anything. We use two, which is the minimum that means anything, and we publish both — including the cases where they disagree.
How to test a vendor on this. Ask them a single question: how many times do you ask each prompt, and do you show me the variation? If the answer is once, their dashboard is a random number generator with good typography. If they cannot answer at all, that tells you something too.
And the honest limit on our own side. Even done properly, this measures visibility, not revenue. Whether being named more often produces more customers is a separate question, and the survey literature is blunt about it: no reviewed technique shows a stable, cross-platform causal effect on downstream behaviour. What we can show you is whether the machine names you more than it used to. The rest is your judgement about what that is worth.
