![]()

Key Takeaways
- A test asking ChatGPT the same question twice on the same day found that none of 83 repeated pairs returned identical lists, though the first-named company reappeared somewhere in the second answer in 60 of those 83 pairs
- The study spanned 123 cities and towns in Ohio and Pennsylvania across window and door replacement, epoxy garage floors, foundation repair, and funeral homes, and separate larger studies found the same pattern of shifting answers
- Wording, web pages pulled during search, location, memory, and model version can all change which businesses ChatGPT names, which is why counting how often a business appears matters more than chasing a single ranking
- Tracking real customer questions monthly and measuring appearance rate instead of position gives a clearer read on ChatGPT visibility
- Common mistakes owners make when checking their ChatGPT visibility include testing only once and fixing only their website
Ask ChatGPT the same question twice in one day, and the odds of seeing the exact same list of businesses are close to zero. That is not a glitch. It is simply how the tool works, and understanding it changes how a local business owner should think about getting recommended.
0 of 83 Pairs Matched Exactly
A direct test makes the point hard to ignore. When the same “who should I hire” question was asked twice on the same day, in the same city, with the exact same wording, the two lists matched exactly zero times out of 83 pairs. Not once did ChatGPT repeat itself word for word.
That might sound alarming at first, but it points to something more useful than bad news: a single screenshot of ChatGPT naming a business, or failing to, proves very little on its own. Profit Acuity has examined this pattern closely in Does ChatGPT Give Everyone the Same Answer?, and the takeaway is that owners need a different way to measure whether they show up, one based on repetition rather than a one-time check.
The sections below walk through what the repeated testing found, why the answers move around so much, and what a business owner can actually do about it.
What 83 Repeated Questions Showed
The numbers behind that headline figure tell a more complete story. Out of the 83 matched pairs, the business named first in the original answer stayed first in the repeat answer only 18 times, about 22 percent of the time. Far more often, the top name shuffled down the list or got replaced outright.
Still, there was a thread of consistency worth noting. The company listed first in the initial answer turned up somewhere in the second answer, even if not in the top spot, in 60 of the 83 pairs, which works out to 72 percent. The two answers shared an average of 4.3 business names between them, and in two of the pairs, the lists had no companies in common at all. Businesses were not vanishing randomly. They were shuffling position, dropping in and out depending on small shifts in timing and wording.
How the 123-City Test Worked
The underlying test covered 123 cities and towns across Ohio and Pennsylvania, spanning four trades: window and door replacement, epoxy garage floors, foundation repair, and funeral homes. Each city received about eight questions, ranging from broad prompts like “Who’s the best window replacement company in [city]?” to narrower ones like “Who should I hire to replace a patio door?”
For 11 of the window-replacement markets, every question was asked twice on the same day, a few minutes apart, using identical wording. That setup produced the 83 matched pairs. One important detail: the questions were run through OpenAI’s API with web search turned on, rather than typed directly into the ChatGPT app. The app layers in memory, location, and chat history on top of search, which means a real customer’s results can vary even more than what the test captured.
Bigger Studies Found the Same Pattern
This is not an isolated quirk. Larger independent studies have landed on the same conclusion using much bigger sample sizes. One study ran nearly 2,961 tests with hundreds of volunteers, asking for brand recommendations 60 to 100 times per platform. The same list in the same order came up less than one-tenth of one percent of the time.
A separate study out of the UK focused on local plumbers, asking ChatGPT 30 identical questions across three towns. Across all those answers, 51 different businesses were named. In two of the three towns, no single plumber showed up in every answer. The pattern holds whether the search covers window installers in Ohio or plumbers in England: repetition reveals instability that a single check would never show.
Why the Same Question Gets Different Answers
Three forces are constantly in motion behind the scenes, and together they explain why two runs of the identical question can land on different results.
Web Search Pulls Different Pages Each Time
When search is turned on, ChatGPT reads a handful of web pages before writing its answer. It does not always pull from the same set of pages twice in a row. In one Columbus-area check on window replacement, the answer drew from eight websites, and seven of them were review sites, directories, and blogs rather than a company’s own site. Swap even one or two of those eight pages for different ones, and the list of recommended businesses shifts along with it.
Small Wording Changes Shift the List
The exact phrasing of a question steers the search in a different direction. In one city, four separate garage-floor-related questions, covering the best epoxy company, who to hire, best reviews, and who installs polyaspartic coatings, together named 20 different companies, with only one appearing across all four. Across eight questions per city, an average of roughly 34 different companies got named. That means a business doing well on “best window company” might be completely absent from “who fixes a foggy window,” simply because the wording sent the search down a different path.
Location, Memory, and Model Add More Variation
Beyond wording and search results, a handful of other factors shape what a real customer sees. Location settings can change what “near me” actually means for two different people in the same town. Saved memory or earlier chat history can nudge the answer toward preferences a customer mentioned days earlier. Free and paid plans can run different underlying models, each of which searches and writes somewhat differently. Whether web search runs at all also matters, since an answer generated purely from the model’s training can look very different from one built on fresh web pages.
Appearance Rate Beats a Single Screenshot
Given all that movement, the sharper question to ask is not whether ChatGPT named a business once, but how often that business shows up across many tries.
Some businesses do show up again and again, even while the exact list around them keeps changing. In one check across 25 towns near Columbus, asking “Who installs bay or bow windows in [town]?”, one company appeared in 19 of the 25 towns, far more often than other well-reviewed competitors in the same area. That is not because its reviews were dramatically better. It simply had more presence in the places ChatGPT pulls information from.
The larger brand study mentioned earlier backs this up at a bigger scale. In the headphones category, brands like Bose, Sony, Sennheiser, and Apple each showed up in somewhere between 55 and 77 percent of 994 responses, even though the exact ranking rarely repeated. Rank is noisy. Appearance rate is steady, and steady is what a business owner can actually track and improve.
Common Mistakes Owners Make Checking ChatGPT
A few habits show up again and again when business owners try to check their own ChatGPT visibility, and each one leads to a false sense of security or a missed opportunity.
Testing Once and Relaxing
An owner asks the question once, sees the business name pop up, and assumes the job is done. A week later, a customer asks the same thing and gets a completely different list. One answer is a sample of one, and with roughly a one-in-five chance of even the top name repeating, that single check tells an owner almost nothing reliable.
Asking Only by Company Name
Typing “Tell me about [my company]” into ChatGPT usually produces a flattering, accurate description, since the tool often knows the business and describes it well. That is not the question a real customer asks, though. Customers ask who to hire, not what a specific company is about, and plenty of businesses pass the first question easily while missing the second one entirely.
Fixing Only the Website
Spending time and money polishing a homepage feels productive, and it does help. But in the Columbus-area check, seven of the eight sources ChatGPT relied on were outside sources entirely, things like review platforms, directories, and local articles. A business that only exists on its own website is holding very few tickets in a drawing that pulls heavily from everywhere else.
A Monthly Process for Tracking Appearance
A workable system does not require guesswork. It follows a repeatable rhythm:
- Pick 10 real questions customers actually ask before hiring, using their words rather than marketing language.
- Ask each one at least three times, keeping the wording and city identical across runs.
- Count how many of those runs name the business, rather than focusing on where it lands in the list.
- Note which competitors get named instead, and where those competitors tend to show up online.
- Review which web sources ChatGPT cites, since those sources hint at who the tool trusts for that question in that town.
- Prioritize the questions where the business got named zero times, publishing clear, specific answers in more than one place.
- Repeat the whole count every month, tracking the appearance rate over time rather than any single screenshot.
Running this kind of check does not require expensive tools to get started. A handful of questions asked a few times over, with the results written down and compared month to month, is enough to see whether visibility is trending up or down.
Tickets Beat Ranking in This Raffle
The clearest way to think about all this is as a raffle rather than a leaderboard. Every time a customer asks a question, ChatGPT draws a handful of names from everything it can find. Nobody controls that draw directly. What a business can control is how many tickets it holds going into it.
More tickets come from being mentioned in more places that answer real customer questions, not just from a polished homepage. A wide research effort covering 75,000 brands found that how often a brand gets mentioned across the web lines up with AI visibility roughly three times more strongly than backlinks do. For a local business, that points toward three habits worth building: publishing a clear answer for each specific question customers ask before buying, getting those answers echoed across review sites, local news, and other outside sources, and keeping the business name, phone number, and address consistent everywhere it appears.
None of this happens overnight, and a freshly published page typically needs some time before it gets picked up and folded into AI-generated answers. For a business owner ready to stop guessing and start measuring, tracking a handful of real customer questions over time is a more reliable starting point than hoping one lucky screenshot tells the whole story.
Profit Acuity
+1-610-936-6839
2138 W Union Blvd
#1047
Bethlehem
PA
18018
United States
