LLM Visibility Teardown: Did ChatGPT Search or Remember?

LLM visibility is how often ChatGPT, Gemini, Claude, Perplexity, Google AI Overviews and AI Mode name your business when your buyers ask, with factual accuracy and citation frequency measured separately.

Each answer may draw on what the engine already knows, a live search, or both, and the evidence of which one happened tells you what to check first. In Graphite's test of logged-out ChatGPT, 58% of responses to entity-comparison prompts used search in early August 2026. If the gap sits in what the model already believes about you, paying for content may not fix it.

Key points

• Log whether each ChatGPT run shows a search signal, then report your rate both ways. A blended score hides the difference.
• A citation alone doesn't prove a search. ChatGPT can insert links without showing any search signal.
• Leave the search setting alone. In an August 2026 test, forcing it moved visibility by 20 points on average for prompts that never searched unforced.

What is LLM visibility?

LLM visibility is the share of answers to your buyers' questions in which an engine such as ChatGPT or Perplexity names your business. How often it gets your facts right and how often it cites your pages are separate rates. Measure each per engine, over a fixed set of prompts run repeatedly.

How do you tell if ChatGPT searched before it answered?

Look for a search signal, because links alone won't settle it. Graphite classed a response as searching when it had "at least one fan-out query or search result," and it identifies the search result as the Sources panel on the right of the answer. It didn't count citations by themselves, since "as of May 2026, ChatGPT can insert inline links without exposing either of these search signals."

Record observed queries and search results in separate columns. Either one counts as an observed search signal. If neither appears, log "no observed search signal." If your view of ChatGPT doesn't expose that evidence, log "unavailable." Don't call the no-signal group memory. A September 2026 paper by Benjamin Tannenbaum says, "We do not equate citation-free output with pure pretraining memory," because a search can go unobserved.

Mark this on every run, because Graphite found that "no prompts in our dataset always use search when unforced."

Why shouldn't you force web search when you measure?

A forced run can show you an answer your buyer wouldn't get. Graphite collected 100 responses to each of 100 entity-comparison prompts, with and without forced search, from July 31 to August 5, 2026. For prompts that never triggered search on their own, visibility differed by 20 points on average. Graphite estimates sampling alone would yield differences of 3.4 to 4.1 points.

Its advice is to let the engine decide and report "the grounding (search) rate for each prompt." These were general entity-comparison prompts, so treat the figures as a reason to run your own test.

What should you check for each result?

Split your ChatGPT named rate into runs with an observed search signal and runs with none, then read the pair. Each row is a lead to confirm before you spend money.

OpenAI's crawler documentation says "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers," so rule that out first. The five-check teardown walks through the access check.

Can better pages make up for an engine that doesn't know you?

Sometimes, and the evidence is mixed, so test before you commit a budget.

Graphite's October 2026 paper compared logged-out ChatGPT answers with and without search across 3,115 entity-question pairs from 57 questions, collected in August 2026. When an entity's visibility without search sat in the lowest band, 0 to 25%, Graphite reports that "Even entities mentioned on at least 15% of cited pages reach only 11% median visibility with search."

Tannenbaum measured whether a brand's own domain appeared in the retrieval path. In unbranded GPT runs from June to September 2026, mention rates were 2.8% with neither the brand nor its domain in the observable path, and 49.0% with an own-domain citation and no branded search query. The author runs a company that sells AI-search measurement software and calls the model "predictive and observational, not a causal description."

Neither result proves a new page will get you named. Together they're a reason to look at both routes before you fund one.

Hypothetical example: a plumbing firm finds ChatGPT naming a competitor and citing that competitor's service page. Open the page and compare it with your own on service coverage (does it describe the exact service the buyer asked about), location relevance (does it name the buyer's area) and the evidence behind the answer (such as a license number or a stated warranty). That won't prove why the page was used, but it gives you specific gaps to test.

How do you test your own LLM visibility?

This procedure is for ChatGPT. You finish with an overall named rate, a rate on each side of the split, and a short list of pages to read.

1. Write the questions your buyers ask. Include your service plus your town, and freeze the wording between checks.
2. Run each one several times in a fresh chat. Leave the search setting alone. The signal-or-noise test covers how many runs you need.
3. Log five things per run. Were you named, were the facts right, was your page cited, did you see a search query, and did a Sources panel appear? Use the two labels above when neither shows.
4. Calculate the rates. Overall named rate is runs naming you divided by all runs. Each split rate is runs naming you divided by the runs in that group, leaving out "unavailable" runs. Also report each prompt's observed search rate, runs with a signal divided by runs where the evidence was visible.
5. Compare only matched prompts, the same question on both sides, because some questions search more than others. Where a group is empty for a prompt, mark that rate "unavailable."
6. Read the table above and open the cited pages before you change anything.

Hypothetical example: you run 20 questions 5 times each in ChatGPT, for 100 runs. An observed search signal appears on 60 and not on 40. You're named in 6 of the 60 (10%) and in 20 of the 40 (50%). Your overall named rate is 26 of 100 (26%). If the same questions appear on both sides, the gap is a reason to check the cited pages first. It doesn't show that the engine remembers you or that those pages cost you the mention. So you'd open the pages cited on the 54 observed-search runs that left you out before you pay for new content.

How long does each route take to change?

On the search route, OpenAI says "it can take ~24 hours from a site's robots.txt update for our systems to adjust." That covers access only, and doesn't mean you'll be cited.

Expect the memory route to be slower. For work aimed at what models learn in training, Graphite warns against expecting impact "in two weeks when the knowledge cutoff gap is five months." It recommends "targeting many mentions across many distinct pages, while avoiding near-duplicate content," and checking again when new models are released.

FAQ

Will blocking GPTBot hurt my LLM visibility?

It won't take you out of ChatGPT search answers, because OpenAI says "Each setting is independent of the others" and OAI-SearchBot governs search. It can matter for the memory route, since disallowing GPTBot indicates your content "should not be used in training generative AI foundation models."

Do these numbers apply to Gemini, Claude and Perplexity?

Don't carry them over. The figures above come from ChatGPT and GPT runs, and the search signals are documented for ChatGPT. Track each engine's overall named rate separately, Google AI Overviews and AI Mode included, and mark its search split "unavailable" unless you have a verified way to see that engine's searches.

Should I pay for an LLM visibility tracker?

Ask two questions first: does it force web search, and does it report how often each prompt searched on its own? If a vendor can't answer both, its LLM visibility number can't show you which route to check.

What should you do next with your LLM visibility?

You can run this split by hand for the cost of your time. If you'd rather start from stored answers when you decide which changes deserve your budget, request Vector Index's AI visibility audit. It asks 25 questions selected from your business, four times each, on ChatGPT, Gemini, Claude, Perplexity and Google AI Overviews, and stores every answer. You get three change specifications in writing and a baseline for your LLM visibility. It doesn't make the changes or guarantee citations.