AI Visibility Tracking: Is Your Number Signal or Noise?

AI visibility tracking means asking ChatGPT, Perplexity, Gemini, Claude, Google AI Overviews and AI Mode your buyers' questions on a fixed schedule, many times each, and reporting how often each engine names you, describes you correctly and cites your pages.

The hard part is telling a real change from a reshuffle. In SparkToro's study, run in November and December 2025, the tools tested gave the same list of brands or recommendations twice fewer than 1 in 100 times. So a dip in your number may mean nothing, and you can spend real money fixing a problem that was only a reshuffle.

Key points

• One answer is a single sample. Report a rate across repeated runs, with the counts beside the percentage.
• A quick range check tells you whether a change is bigger than the noise.
• Spread your runs across several wordings of each question, and log every page each answer cites.

What does AI visibility tracking measure?

Each run fills four log columns: whether your business was named, whether the details were right, whether one of your pages was cited, and which pages the engine drew on. Those four come from Searchable, a company that sells a tracker.

Record description accuracy in its own column, only for runs that name you. List the facts a buyer needs, such as your services and service area, and mark each one right, wrong or missing. A mention rate alone won't show those errors.

Split your questions too. A branded prompt names your company, and an unbranded one describes a need without naming any company. In Searchable's data (brands it tracks, over the 30 days to September 18, 2026), the median brand was named in 93% of answers to branded prompts and in only 7% of answers to unbranded ones. That's one vendor's customers, not a market average. Still, a buyer who doesn't know your name can only ask the unbranded kind, so a score that blends the two can flatter you.

Why does your AI visibility number move when nothing changed?

In SparkToro's test, 600 volunteers ran 12 different prompts through ChatGPT, Claude and Google's AI Overviews, 2,961 times in all. Presence held up better than position. For headphones, Bose, Sony, Sennheiser and Apple showed up 55% to 77% of the time.

Rand Fishkin, who wrote it up, calls visibility percentage across dozens to hundreds of prompts run multiple times a reasonable metric. The post discloses that Fishkin's research partner, Patrick O'Donnell, joined Gumshoe.ai, a startup in the AI tracking sector. Perplexity and Gemini weren't among the tools tested.

How do you tell a real change from noise?

Treat each rate as a range. NIST's statistics handbook gives the confidence expression most often used for a proportion. With a round 2 as the multiplier, it becomes:

range = rate, plus or minus 2 x the square root of (rate x (1 - rate) / runs)

Enter the rate as a decimal, so 30% is 0.30. A multiplier of 2 gives roughly 95% confidence.

Hypothetical example: ChatGPT names you 30% of the time, then 50% after a page change. Only one row clears this first screen.

1. Hold the conditions fixed. Same engine, model, search setting, location and wording, in a fresh chat each time.
2. Write the counts beside every percentage.
3. Work out both ranges. If they overlap, the question is still open, so re-run before you spend anything.
4. If they don't overlap, look again. Treat a change as real only if it holds on the next check and across several questions. Even then, it doesn't prove your page edit caused it, because the engine's model or the pages it reads may have moved too.

This is a cautious screen and not a formal test. It treats runs as independent, which may not hold for your setup, and NIST warns that limits built this way may not be accurate enough when the sample or the number of failures is very small. So don't use it when you're named on none, very few, nearly all or all of your runs.

Hypothetical example: 0 of 40 runs gives 0% to 0%, as if no doubt were left. NIST's Wilson formula, with the same multiplier, gives about 0% to 9%. For those counts, use Wilson or NIST's exact binomial method.

Should you add more repeats or more wordings?

Try more wordings first. Dmitrij Żatuchin's July 2026 paper split the noise in 12,933 LLM responses about 20 Central and Eastern European brands, across eight languages and three models. It found that a single response says almost nothing that separates one brand from another, and that "repeats beyond five are the least efficient place to put a query." Adding paraphrases did better, even at fewer total queries.

It measured the sentiment of each response, not whether a brand was recommended. GPT-5.2 and Gemini 3 Flash ran without retrieval and Perplexity ran with it, so it isn't a test of ChatGPT search.

Hypothetical example: rather than asking one wording 100 times, ask 20 wordings 5 times each, with the same wordings, the same split of runs and the same weighting before and after. The pooled percentage describes those 20 wordings, not all your buyers, and 100 total runs alone doesn't make the runs independent, so the range check is only a rough guide here. More repeats narrow the range on one exact wording. More wordings widen what the number covers.

Which pages should your AI visibility tracking log?

Log every page the answer cites, on every run. Benjamin Tannenbaum's September 2026 paper analyzed 34,960 unbranded prompt-engine observations from 75 Aiso projects, on GPT and Gemini, June to September 2026. With neither the brand nor its own domain in the observable retrieval path, GPT mention rates were 2.8%. With the brand's own domain in the stored source list, and no search query naming the brand, they were 49.0% (58.4% on Gemini). The engines differ, so report each one separately.

That doesn't mean a citation earns you a mention. The paper calls its model "predictive and observational, not a causal description," and warns that "citation and recommendation can diverge dramatically." The author, founder and CEO of Aiso Boost Ltd., which develops commercial AI-search measurement software, flags "a financial conflict of interest."

In a smaller benchmark run through ChatGPT with web search, "four of the sixteen prompts produced zero citations across all ten runs." Label runs like those "no visible citations." The paper can't exclude other paths: "Search could be unobserved, a tool may return uncited information, or the model may rely on latent knowledge." So those runs don't show that no sources were used, and they don't clear your page. Keep displayed citations in one column and any retrieval record you can see, such as a stored source list, in another.

Treat each row below as a lead to test:

Hypothetical example: one log entry for a bookkeeping firm.

• Buyer question: "Who handles payroll for small restaurants near me?"
• Saved answer: Names the firm and says it only does tax returns.
• Cited URL: A local directory's page, `/listings/your-firm/`, not the firm's own site.
• Discrepancy: The firm's services page lists payroll. The directory page doesn't.
• Next check: Ask the directory to correct the page, then re-run the same question on the next scheduled check.

Our AEO marketing test walks through the run-by-run logging.

Is AI visibility tracking worth paying for?

Start by hand if you can keep the routine going. Searchable says doing it yourself "works for testing the idea on ten prompts, but it gets expensive in staff time at a hundred."

Before you pay for a tool, ask:

• How many times does each prompt run, and how many wordings of each question does it test?
• Can you see the raw answer behind any number?
• Are branded and unbranded prompts reported separately, by engine?

Walk away from a pitch built on a rank position or promised citations.

If you'd rather start from stored answers than build the log first, request Vector Index's AI visibility audit. It asks 25 questions selected from your business, four times each, on ChatGPT, Gemini, Claude, Perplexity and Google AI Overviews, and stores every answer. You get three change specifications in writing, so you know what to change and have a baseline for your later AI visibility tracking. It doesn't make the changes or guarantee citations.

FAQ

How many times should I run each prompt?

There's no agreed number. Searchable says nobody can hand you a universal minimum, but any vendor should be able to state their own. Żatuchin's paper suggests repeats past five add little, though it measured sentiment, so test that on your own questions.

Can Google Search Console track ChatGPT visibility?

No. Its generative AI performance report counts how many times links to your site were shown in AI Overviews and AI Mode on Google Search.

Can a tool tell me my rank in ChatGPT?

Not in a way that holds up. SparkToro's study took about 1 in 1,000 runs to see two lists in the same order. Ask for an appearance rate across repeated runs, by engine, because that's the number AI visibility tracking should give you.