Elevate Talk – I ran the same prompt 50+ times and never got the same sources twice

what-ai-is-saying-about-your-business

I needed a demo for my Elevate talk so I picked a query most Canadians have typed at some point, “best travel credit card canada,” and ran it into ChatGPT, Gemini, Google AI Overviews and Perplexity. All temporary chats or incognito, because my own account knows too much about me and I wanted to see what a stranger sees.

Fifty-odd runs later I had fifty-odd different sets of citations. Zero repeats. Which meant deleting a chunk of my talk, because I’d written it assuming something else.

What stayed the same and what didn’t

The recommendations were pretty stable. Same two or three cards every time, Amex Cobalt and Scotiabank Gold trading places at the top depending on the run. If you only tracked which brands got named you’d walk away thinking this whole thing is basically solved.

The sources cited underneath those recommendations were all over the place. One run leaned on three of Scotiabank’s own pages. The next used Ratehub. After that I got Forbes, Rewards Canada, a site called Frugal Flyer I’d never heard of, MoneySense, Milesopedia, rates.ca. Fifteen-plus domains, one question, one afternoon.

I think we’ve been treating those as the same thing and they’re not.

Ranking isn’t a gate

I went in assuming the SERP would roughly predict who got cited. It doesn’t. NerdWallet’s Canadian site doesn’t rank top ten for that query and it was one of the most-cited sources across every engine I tried. CIBC sits at position five organically and I didn’t see it cited once in fifty runs.

Ranking still matters, to be clear. Ratehub is position two and the most-cited domain in the category, so there’s obviously a relationship. It’s just not the gate I assumed it was, and a lot of what gets pulled in lives well outside page one.

The model isn’t searching for what you typed

Watching the retrieval step was the bit that actually changed how I think about this. The tools weren’t running my query, they were writing their own and running those. I asked for “best travel credit card canada” and one of them went off and searched “Scotiabank Passport Visa Infinite annual fee no foreign transaction fee 2026.” I never said Scotiabank. It had already picked a candidate and then went looking for evidence about it.

Which explains the whole pattern, really. The candidate list is stable because it’s coming from the model. The citations move because every sub-query is a fresh retrieval.

The bit I’d actually action

Ratehub got 66 citations off 22 pages. TD had more distinct pages picked up than almost anyone in the category, 32 of them, and got cited 12 times. So roughly three citations per page versus 0.4.

TD has a page for every single card. What they don’t have is a page that answers “which one should I get,” and I suspect that gap is worth more than another twenty product pages would be.

Go do this to your own category

Twenty years of Google has trained us to expect the same page one every time we look, and that expectation just doesn’t hold here. You can read about it or you can watch it happen to your own business, and the second one is faster.

You don’t need fifty runs. Five to ten will show you the recommendations settling and the sources refusing to. Spread them across different tools instead of hammering one, because the engines disagreed with each other way more than any single engine disagreed with itself.

I made a one-page worksheet for this, which is what we used in the room at Elevate. You write three decision-point prompts, guess what’s going to come back before you run anything, then run them and mark where you were wrong. There’s a technical checklist on there too for crawler access and whether your facts are actually in your HTML or hiding behind JavaScript.

Download the worksheet

Here’s the deck

Obvious caveat: 50 runs of one query in one category isn’t a study.
I had five theories going in and four of them broke on my own data, so take this as a reason to go look at yours rather than a rule about how any of it works.

Facebook
Twitter
LinkedIn
Pinterest
Email