How to Prove AI Engines Actually Cite You, Not Luck

A marketer runs a prompt in ChatGPT, sees their brand cited, takes a screenshot, and reports a win. Two problems: they were logged in, and they only ran it once. Either one alone can produce a result that has nothing to do with whether the content is actually influencing the model's output.

This isn't a nitpick. It's the difference between a testing methodology and a highlight reel. AI answer engines introduce two distinct sources of noise that traditional rank tracking never had to deal with — personalization (the same account sees a different answer than a stranger would) and volatility (the same account, same prompt, same moment sees a different answer on the next run). Controlling for both is table stakes for any claim about AI visibility that needs to survive contact with a skeptical stakeholder.

Confound 1: personalization means you might be testing your own echo chamber

Direct answer: Modern AI assistants increasingly answer with context about you, not just the prompt. OpenAI's 2025 memory expansion made this explicit: as of the April 2025 update, ChatGPT began referencing "all past conversations to deliver responses that feel more relevant and tailored to the user," combining explicitly saved memories with inferred "chat history" insights — and this reached free-tier accounts by June 2025 (OpenAI, OpenAI Help Center). Two people signed into two different accounts, asking the identical question, can get answers shaped by entirely different stored context, prior chats, and custom instructions.

Google is just as explicit about this on the search side. Google's own support documentation states plainly that results differ between people based on "location, search history, sign-in status, device type, language settings, timing, and the intent behind the search," and that this extends to AI Overviews and other AI-powered features — not just classic blue links (Google Search Help: Why your Google Search results differ from others, Personalization & Google Search results). Location matters even for AI-generated answers specifically — a query run from one city can surface geographically different sources than the identical query run from another (SERP Wizard).

The practical trap: if you test from your own logged-in, geographically fixed, history-laden account, you're not measuring what a prospective customer sees. You're measuring what the model thinks you specifically want to hear — which, if you've spent any time researching your own company, may well include your own company.

Confound 2: volatility means the same prompt is not the same prompt twice

Even with personalization fully controlled — logged out, no history, clean session — the same exact prompt can return materially different answers on consecutive runs. For a long time the assumption was that "temperature 0" or greedy decoding meant deterministic output. Research from Thinking Machines Lab published in September 2025 overturned that assumption in a widely-cited technical breakdown: sampling 1,000 completions from a large open model at temperature 0 produced 80 distinct outputs, diverging as early as the 103rd token — not from floating-point rounding per se, but because production inference servers dynamically batch concurrent requests, and the batch size a request lands in (which depends on unrelated traffic load) changes the numerical path through normalization, matrix multiplication, and attention kernels (Thinking Machines Lab; summarized by Simon Willison). In other words: nondeterminism isn't a bug you can prompt-engineer around. It's a property of how these systems are served at scale, and it holds for CPU and TPU deployments too, not just GPUs.

Independent empirical work backs up how large the practical effect is. A 2025 ACL Anthology study tested five API-based LLMs configured for "deterministic" settings across eight tasks over ten runs each and found accuracy swings of up to 15% between runs on the same task, with as much as a 70-point gap between the best-performing and worst-performing run — concluding that no model in the study reliably reproduced its own output (ACL Anthology, eval4nlp 2025). Microsoft's Phi-4-reasoning technical report found the same pattern at the reasoning-model level: across 50 independent runs of the same benchmark, one model's accuracy ranged from 30% to 70% depending purely on which run you happened to look at — leading the authors to warn that any comparison based on a single run "can easily produce misleading conclusions" (arXiv 2504.21318). A separate analysis of 1,000 repeated completions from a production model found the divergence isn't usually catastrophic — about 95% of decision points in the output are single-path — but the remaining 5% are branch points, and because autoregressive generation compounds, one early branch can cascade into a completely different final answer, including whether a citation appears at all (arXiv 2511.02620).

For visibility testing this means: a single run where your brand is cited is not evidence of a citation rate. It's one draw from a distribution you haven't characterized yet.

A protocol that controls for both

The fix isn't complicated, but it has to be deliberate. Four controls, applied together:

1. Logged out / incognito by default. Every test session should start from a clean, unauthenticated state with no saved memory, no custom instructions, and cookies cleared between runs. This is the only way to approximate what a prospect with zero relationship to your brand would see.
2. Multiple accounts and personas where login is unavoidable. Some engines require authentication for full functionality. When that's the case, run the same prompt set across several accounts with genuinely different histories — not just one "clean" account you keep reusing, since that account accumulates its own history the moment you start testing with it.
3. Multiple locations. Route a subset of runs through different geographic exit points, since both classic and AI-generated search results incorporate location signals (Google Search Help). A brand that's cited from a US IP and invisible from a UK one isn't "sometimes visible" — it's regionally absent, which is a different finding with a different fix.
4. N repeated runs per condition, not one. Every cell in your test matrix (prompt × engine × location × logged-in-state) needs multiple independent runs before you draw a conclusion, for the volatility reasons above.

That last row matters more than it looks. The academic literature on this specifically flags it: in one large-scale study of AI search visibility, 57.8% of ChatGPT repetitions for a given query set didn't invoke web search or return citations at all — meaning if you only look at runs that do have citations and calculate "share of citation" among those, you've built in a selection bias that inflates apparent visibility (Schulte, Bleeker & Kaufmann, "Don't Measure Once: Measuring Visibility in AI Search"). The denominator has to be every run you attempted, not just the ones that gave you a citation to count.

How many runs is "enough"?

This is where testing methodology turns into statistics, and it's worth being precise instead of hand-wavy. The same 2026 paper that documented the search-activation selection bias also ran a bootstrap convergence analysis on repeated same-day query runs: the standard error of the estimated per-brand detection rate dropped below 0.10 (95% CI of roughly ±0.158) at 7 runs, and below 0.08 (95% CI ≈ ±0.121) at 8 runs. Source-level coverage (which domains get cited, not just whether the brand is mentioned) converged more slowly, needing 8 runs to hit the same 0.10 threshold, reflecting that which sources get pulled into an answer is noisier than whether a brand name appears at all (arXiv 2604.07585). Seven to eight independent runs per condition is a defensible floor, not an arbitrary round number — below that, your confidence interval is wide enough to make a real trend indistinguishable from noise.

The opposite failure mode — concluding "we have zero AI visibility" — needs its own statistical care. If a brand shows up in 0 of 20 test runs, that doesn't mean the true citation rate is zero; a standard approximation for the upper bound after observing zero events in n trials is roughly 3/n, so 0-for-20 is still statistically consistent with a true citation rate as high as 15% (MaxAEO, "AI Visibility Sample Size"). Reporting "no visibility" as a categorical fact based on a handful of runs is exactly the same error as reporting "we're cited!" based on one — just in the opposite direction. Precision scales with the square root of your sample size, not linearly, so halving your margin of error requires roughly four times the runs, not twice.

Putting it together

Direct answer: A defensible AI-visibility claim, in practice, needs: a fixed prompt set, tested logged-out by default with rotated personas where login is unavoidable, spread across the locations that matter for the business, run at least 7-10 times per condition before any single condition is summarized, with every run logged — hits, misses, and no-search responses alike — before a rate is calculated. That rate should be reported with a range, not a point estimate, and re-checked periodically, since model updates and routing changes shift the underlying distribution without warning.

None of this requires exotic tooling — a spreadsheet with disciplined logging and a consistent run cadence gets most of the way there. What it does require is treating "we got cited" the way you'd treat any other probabilistic claim: as a rate with an error bar, established over enough independent trials to mean something, not a screenshot from one good run.

Sources:

• OpenAI: Memory and new controls for ChatGPT
• OpenAI Help Center: Memory FAQ
• Google Search Help: Why your Google Search results differ from others
• Google Search Help: Personalization & Google Search results
• SERP Wizard: How Location Affects Google's AI Overviews
• Thinking Machines Lab: Defeating Nondeterminism in LLM Inference
• Simon Willison: Defeating Nondeterminism in LLM Inference (summary)
• ACL Anthology: Non-Determinism of "Deterministic" LLM System Settings in Hosted Environments (Eval4NLP 2025)
• Phi-4-reasoning Technical Report (arXiv 2504.21318)
• Verifying LLM Inference to Detect Model Weight Exfiltration (arXiv 2511.02620)
• Schulte, Bleeker & Kaufmann: "Don't Measure Once: Measuring Visibility in AI Search (GEO)" (arXiv 2604.07585)
• MaxAEO: AI Visibility Sample Size — How Many Prompts and Runs to Trust