ChatGPT Citations Aren’t Stable: What SMEs Should Track (And How To Win In AI Search Anyway)
A new research preprint suggests you can’t reliably reproduce ChatGPT’s citations using Google results or OpenAI’s API—and even identical prompts can yield different sources. Here’s what changed, why it matters for SMEs and agencies, and how to build AI search visibility with monitoring plus approved execution.
AI Search is forcing businesses to relearn a lesson we already learned in traditional SEO: if you can’t measure something consistently, you can’t manage it. But with ChatGPT citations, the gap is bigger than most teams realize.
A recent research preprint covered by Search Engine Journal found that when researchers tried to reproduce ChatGPT’s citations using two “stand-ins”—Google organic results and OpenAI’s Responses API—they could match the sites behind citations far more often than the exact pages. They also observed that ChatGPT rewrote queries, and that sources changed between identical runs.
That sounds academic. It isn’t. If you’re an SME (or an agency serving SMEs), it changes how you should:
- Report “AI visibility” without fooling yourself (or your client).
- Prioritize content and technical work for AI answers.
- Build Monitoring that reflects how AI systems actually behave.
- Operationalize fast, safe website changes—because volatility punishes slow teams.
This editorial is my practical take (Marius Dosinescu, AYSA.ai) on what changed, why it matters, and what to do next—without chasing vanity metrics or building a tracking Rube Goldberg machine.
Concise summary (what you need to know)
- Don’t trust a single source of truth for “ChatGPT citations.” Google rankings or API results may not reproduce what the ChatGPT app cites at the page level.
- Track both site-level and page-level signals. Site mentions can be more stable, but page-level accuracy is where revenue often lives.
- Expect volatility—even for identical prompts. Run prompts multiple times and log the model/version used.
- Assume query rewriting. Your customer’s prompt is not the query the model runs, especially across languages.
- Execution matters now. The teams that win will monitor, prepare changes, get approvals fast, and implement cleanly. That’s the gap AYSA is designed to close.
Table of contents
- What changed: citations are harder to reproduce than we thought
- The core finding: you can match the website, but miss the exact page
- Why “citation tracking” is harder than rank tracking
- Query rewriting: the invisible step that breaks naive reporting
- Volatility across runs: why one screenshot is not evidence
- Practical implications for SMEs (and the hidden risks)
- What agencies should rethink: from rank reports to AI visibility systems
- What to monitor now: a realistic AI visibility measurement stack
- How to structure content so AI systems can use it (without gimmicks)
- A concrete SME scenario: an ecommerce brand that “lost” AI visibility overnight
- Where AYSA fits: monitoring + recommendations + approved execution
- What to do next (action list)
- Sources and further reading
What changed: citations are harder to reproduce than we thought
For two decades, SEO measurement mostly revolved around a stable idea: a query goes into Google, a set of ranked URLs comes out, and you track where you sit. The system isn’t perfectly stable, but the measurement model is clear.
AI answers break that simplicity. When a chatbot produces an answer with citations, it’s effectively doing a multi-step workflow that can include:
- Rewriting the user’s prompt into one or more “search-like” queries
- Searching via one or more providers
- Retrieving and reading pages
- Selecting sources (sometimes grouped or deduplicated)
- Generating a response
- Choosing which sources to show as citations
The SEJ-covered preprint put real numbers behind a suspicion many of us already had: if you try to mimic ChatGPT’s citation behavior using “stand-ins” (Google results + OpenAI API), you’ll often find the right domain but not the right URL. And even when you run the same prompt repeatedly, citations can change meaningfully.
That is not just a measurement problem. It’s a strategy problem. If your strategy is “rank for Keyword X,” you’ll miss the mechanics of how AI systems source answers and how they decide which pages are quote-worthy, comparable, or trustworthy.
The core finding: You can match the website, but miss the exact page
The headline takeaway from the SEJ summary: the stand-ins matched cited sites more often than the exact cited pages.
In plain business terms:
- Site-level match answers: “Did the AI system cite our company at all?”
- Page-level match answers: “Did it cite the specific product page / clinic service page / pricing page we want customers to land on?”
Those are radically different outcomes. Site-level visibility might feel like brand presence. Page-level visibility is where your conversion funnel starts.
Why do stand-ins match sites more often than pages?
- Multiple URLs can represent the same “idea.” AI systems may treat a domain as a topic authority and pull from whichever page is easiest to parse or most relevant at retrieval time.
- Canonicalization and duplication in the wild. The “best” page for a concept may exist in multiple forms (print versions, parameterized URLs, localized variants), and different systems pick different ones.
- Different retrieval stacks. The ChatGPT app, Google organic, and the API may use different query expansions, timing, and filtering even when they appear to use “web search.”
- UI decisions vs underlying sources. What’s shown as a citation in the UI may be a subset of what the model looked at.
The strategic implication: don’t build your entire AI search program on the assumption you can recreate citations perfectly via an API call or a Google SERP scrape. Use those tools, but treat them as directional, not definitive.
Why “citation tracking” is harder than rank tracking
Businesses love dashboards. The temptation is to demand a single metric: “How many times did we get cited by ChatGPT this week?” Then the team goes shopping for a tool that claims it can track that number.
Here’s why that approach breaks.
1) AI answers aren’t single-output systems
Rank tracking assumes a stable query → stable list model. AI answers are query → rewrite → retrieval → selection → generation → UI citations. Each step adds variance.
2) “The model” can change without you noticing
The SEJ-covered paper notes that monitoring strategies should be re-evaluated when the underlying model changes. That’s not a footnote—it’s the rule. If you don’t log which model produced an output, you can mistake a product update for a content problem.
If you run AI visibility reporting for clients, you must store metadata: date/time, model/version, language, prompt variant, location (if applicable), and the “stand-in” used (Chat app vs API vs search provider output).
3) Page-level precision is inherently fragile
Even in classic SEO, page-level rank changes happen. In AI citations, the system might decide that the domain is credible but swap the exact page due to:
- Freshness (newer page)
- Readability (cleaner structure)
- Availability (blocked, slow, paywalled)
- Language (English page chosen for a non-English prompt)
- Format (FAQ page preferred over a category page)
4) Citations are not the same as traffic
Even if you get cited, users may not click. The business outcome is still: qualified visitors and conversions. AI visibility must connect to web analytics, lead tracking, and revenue attribution—otherwise it becomes a new flavor of vanity reporting.
This is why our approach at AYSA starts with monitoring and execution that ties back to business goals, not citation count as an isolated trophy. Learn more about how we frame AI visibility: AI Search Visibility.
Query rewriting: the invisible step that breaks naive reporting
One of the most important details in the SEJ summary is that ChatGPT rewrote the user’s question into different search terms—and in observed cases, didn’t use the exact wording of the original question.
If you’re a business owner, here’s what that means:
- Your customer types: “What’s the best running shoe for flat feet under $120?”
- The model might search for: “best stability running shoes flat feet budget”
- Or even: “overpronation stability shoe guide”
If your product page is written purely around your internal terminology (“Model X200 stability foam system”), you might rank for some niche queries—but you may not be the page an AI system pulls into an answer when it rewrites the prompt into shopper language.
This aligns with a broader trend you can see across AI search: models optimize retrieval by moving toward standardized, high-signal query patterns. Businesses need to respond by:
- Using customer language in headings and summaries
- Publishing comparison-friendly information (dimensions, compatibility, constraints)
- Answering “why/when” questions, not just “what” descriptions
Query rewriting also has a reporting implication: if your “monitoring prompt set” doesn’t reflect how the model rewrites questions, you’ll systematically measure the wrong thing.
At AYSA, our monitoring approach is designed to capture visibility signals across a prompt set and then translate them into website changes you can approve and ship. Monitoring is here: AYSA Monitoring.
Volatility across runs: why one screenshot is not evidence
The paper also observed that sources changed between runs, and that a meaningful share of distinct pages appeared only once across multiple runs. Whether you focus on the exact percentage or not, the direction is clear: single-run testing is unreliable.
This matters because most AI “SEO” today looks like this:
- Someone asks ChatGPT a question once.
- They see (or don’t see) their brand in citations.
- They declare victory or panic.
That’s not measurement; it’s superstition.
If you want a defensible operational process, borrow the mindset from experimentation:
- Run prompts multiple times over multiple days.
- Store the outputs (answer text + citations + metadata).
- Measure distributions (how often you appear), not a single outcome.
- Track variance: if your appearance rate swings wildly, treat it as instability and prioritize robustness work (content clarity, technical access, entity consistency).
For agencies, this is the difference between “we think we improved AI visibility” and “we can show repeatable uplift across a monitored prompt set.”
Practical implications for SMEs (and the hidden risks)
SMEs don’t have infinite time. So let’s translate the research into practical risks you can actually control.
Risk #1: You optimize the wrong page because your tool matches the domain
If a tool says “ChatGPT cited your site,” it might be technically true—but if it’s citing a blog post instead of a product or service page, your revenue impact may be close to zero.
What to do: create a “money page” list (top products, top services, booking pages, pricing pages). Your AI visibility program should explicitly track whether those pages—not just your root domain—appear in relevant answers.
Risk #2: You chase citation wins that don’t convert
Some pages are citation magnets (glossaries, definitions, generic explainers). Those can build authority, but they often sit at the top of the funnel.
What to do: pair top-of-funnel pages with strong internal linking and clear next steps. Make it easy for both humans and AI systems to map the journey from “explanation” to “purchase/booking.”
Risk #3: You misread a model change as an SEO failure
If the underlying system changes, citations can change. That doesn’t necessarily mean your content got worse.
What to do: annotate your monitoring timeline with known changes (when you can detect them) and compare performance before/after. If you can’t verify why something changed, label it as “system variance” until you have evidence.
Risk #4: Multilingual businesses get silently rerouted into English
The SEJ summary notes that some Chinese prompts triggered English-only searches in many runs. Even if you’re not operating in Chinese, the general principle stands: AI systems can pivot languages during retrieval.
What to do: if you operate in multiple languages, ensure your localized pages are not thin translations. Build localized FAQs, policies, and comparisons, and maintain consistent entity info (name, address, services) across languages.
Risk #5: You underestimate governance requirements
In healthcare, legal, finance, or even regulated ecommerce categories, you can’t just let an AI tool rewrite pages.
What to do: build an approval workflow that is fast but safe. This is exactly why AYSA focuses on approved execution: we monitor, prepare changes, ask for approval, and execute only what you accept. More here: AI SEO Tools.
What agencies should rethink: from rank reports to AI visibility systems
Agencies are under pressure to productize “AI SEO” quickly. The risk is creating a report that looks sophisticated but rests on shaky assumptions—especially if it uses stand-ins that don’t match page-level citations.
Here’s what I believe agencies need to change now.
1) Stop selling “we track ChatGPT citations” as a single metric
Instead, sell a system:
- Prompt set design (based on real customer questions)
- Repeated sampling (multiple runs, multiple days)
- Outcome tiers: site mention, page citation, click/visit, conversion assist
- Change velocity: how fast you can ship improvements safely
2) Build reports that separate “visibility” from “reproducibility”
Two clients can have the same visibility rate but very different stability. Stability is an asset. If you’re consistently cited across runs, you’re harder to displace.
3) Adjust incentives: reward execution, not just insights
AI search punishes slow implementation. If the agency model is “audit → recommendations → client disappears for 3 months,” you will lose.
This is where an execution system matters. AYSA is built to help teams monitor and implement approved changes continuously, not once per quarter. Pricing and packaging context is here: AYSA Pricing.
What to monitor now: a realistic AI visibility measurement stack
Given the limitations described in the SEJ-covered preprint, what should businesses track today without falling into measurement theater?
Think in layers—each layer answers a different business question.
Layer 1: Business outcomes (non-negotiable)
- Organic traffic to money pages (products/services/pricing/booking)
- Lead submissions, calls, bookings, purchases
- Assisted conversions from informational pages
If AI citations increase but outcomes do not, your “visibility” isn’t commercial.
Layer 2: AI visibility sampling (directional but actionable)
- Prompt set coverage (do you monitor what customers actually ask?)
- Appearance rate across repeated runs (how often are you included?)
- Citation type: site mention vs page citation
- Competitor set: who appears when you don’t?
Layer 3: Website readiness signals (you can fix these)
- Content clarity: concise summaries, scannable headings, FAQs
- Entity consistency: business name, product names, locations, policies
- Technical access: crawlability, load speed, blocked resources
- Internal linking: can AI and users navigate from info → money pages?
Layer 4: Change log + model/version notes (your sanity layer)
- What changed on your site (and when)
- What changed in your monitoring method (and when)
- What model/version generated the outputs (when known)
AYSA’s approach is to connect monitoring to an execution queue—so insights turn into approved, shipped improvements. Start here: AYSA Monitoring.
How to structure content so AI systems can use it (without gimmicks)
Even though the SEJ-covered research focuses on matching citations, it indirectly reinforces something practical: AI systems can only cite and use what they can reliably parse and understand.
So what should SMEs do on their pages?
1) Add “answer-first” summaries where intent is informational
If a page is meant to answer a question, give the answer quickly, then elaborate. This helps both humans and AI retrieval systems.
- Use a short summary near the top (“In short…”)
- Define constraints (price range, location, compatibility)
- Use plain language, then add technical detail
2) Use headings that reflect customer language
Remember query rewriting: your internal jargon won’t be the rewritten query. Use headings like:
- “Which plan is best for a 10-person team?”
- “What insurance do you accept?”
- “Is this compatible with iPhone/Android?”
- “Shipping times and return policy”
3) Create comparison-friendly blocks
AI answers often compare. Help them.
- Tables that compare variants (size, materials, warranty)
- Pros/cons sections grounded in factual attributes
- Clear “who it’s for” guidance
4) Strengthen internal linking around “money moments”
If your glossary page gets cited, can the user (and the AI system) easily find your service page or product category page? Add contextual links that make sense.
5) Avoid fragile tactics
Don’t over-index on “being cited” by forcing unnatural phrases or creating doorway-like AI bait pages. The goal is durable usefulness and clarity, not gaming a brittle output.
If you need a system to manage these improvements across many pages, that’s the point of an execution engine. AYSA’s tooling overview: AI SEO Tools.
A concrete SME scenario: an ecommerce brand that “lost” AI visibility overnight
Let’s make this real with a scenario many SMEs will recognize. (This is a generalized example—no fabricated client metrics.)
Business: a small ecommerce brand selling specialty home coffee equipment.
Situation: the founder notices that ChatGPT used to cite their “Burr Grinder Buying Guide” when asked “best burr grinder under $150,” but now the citations show other sites. Sales feel flat, and the team assumes they “got penalized.”
What’s actually likely happening (based on the research patterns):
- The model rewrote the query into something like “best burr grinder budget consistent grind size” and began retrieving different pages.
- Even if the domain is still considered relevant, the exact page cited may change due to freshness, readability, or retrieval variance.
- A competitor published a more scannable comparison table with clear “under $150” constraints and updated timestamps.
- The founder tested once and assumed the output was stable.
A practical 7-day response plan:
- Stop guessing. Run the prompt set 10–20 times across a few days and log citations and outputs.
- Separate site vs page signals. Are you still cited at the domain level for related prompts, but with different URLs?
- Audit the cited competitor pages for structure: do they have clearer constraints, tables, and summaries?
- Update your guide with a top summary, an “under $150” table, and internal links to your best-selling grinders.
- Improve product pages so they can be cited directly: specs, comparisons, “who it’s for,” FAQs.
- Monitor outcomes. Track not only citations but organic landings and conversion assists.
This is the kind of workflow AYSA is built to support: monitor changes, recommend specific page updates, get approval, and execute the accepted changes quickly. Start from the AYSA blog for related frameworks and updates: AYSA Blog.
Where AYSA fits: monitoring + recommendations + approved execution
AI search creates a new kind of operational pressure: you need to respond to volatility with disciplined iteration. That’s hard for SMEs because the bottleneck isn’t ideas—it’s execution.
Here’s the execution reality inside most small businesses:
- The marketer identifies what to change.
- They need buy-in from the founder, brand, legal, or a clinician.
- They need a developer or CMS access.
- Weeks pass.
In a world where AI outputs can change between runs, weeks is too slow.
AYSA’s model is intentionally built for that gap:
- Monitors AI search visibility signals and website performance inputs (Monitoring)
- Prepares clear, specific recommendations: what to change, where, and why
- Asks for approval so humans stay in control (brand, compliance, accuracy)
- Executes accepted changes on the website—turning strategy into shipped improvements
That last step is the difference between “we know what to do” and “we did it.” In AI search, that difference compounds over time.
If you want to explore how this fits your team (SME or agency), start here:
What to do next (action list)
If you want to act on the research without overcomplicating your life, do this in order.
1) Decide what you’re optimizing for
- Leads/bookings?
- Ecommerce revenue?
- Brand authority in a niche?
2) Build a prompt set that reflects real demand
- 10–30 customer questions that map to your products/services
- Include comparison and “best for” prompts
- Include pricing, policy, and trust prompts (returns, warranty, credentials)
3) Sample repeatedly
- Run prompts multiple times across days
- Store outputs + citations + metadata
- Track both site-level and page-level appearance
4) Identify your “citation-ready” pages
- Top category/service pages
- Pricing and policy pages
- FAQ hubs
- Comparison/buyer guides
5) Improve structure, clarity, and internal paths
- Add answer-first summaries
- Add tables and constraints
- Improve headings and FAQs with customer language
- Strengthen internal links from informational pages to money pages
6) Implement with a governance-friendly workflow
- Create an approval queue for website changes
- Ship improvements weekly, not quarterly
- Maintain a change log to correlate visibility shifts with updates
7) Use AYSA to operationalize it
- Start with monitoring: AYSA Monitoring
- Then connect to execution: AI SEO Tools
Sources and further reading
- Search Engine Journal: OpenAI’s API Only Partly Matched ChatGPT’s Citations, Paper Finds
- Search Engine Journal: AI Search coverage
- Search Engine Journal: SEO coverage
- AYSA.ai: AI Search Visibility
- AYSA.ai: Monitoring
- AYSA.ai: AI SEO Tools
- AYSA.ai: Pricing
- AYSA.ai: Blog
Note on evidence: This editorial relies on the supplied SEJ coverage of an arXiv preprint (not peer-reviewed, per SEJ’s summary). Where exact mechanisms or causes can’t be verified from the provided research context, I’ve framed them as practical hypotheses and operational guidance rather than hard facts.
Continue the AI search topic inside AYSA.
Use these pages to connect the article with AI SEO tools, AI visibility monitoring, AI Overviews and approved website execution.
Turn this topic into a website action plan.
Use these AYSA hubs to move from reading to technical fixes, AI visibility monitoring, research, glossary context and approval-first SEO execution.
