LLMs.txt Won’t Get You Discovered: What Google’s Critique Really Means (and What to Do Instead)

Many publishers and marketers want a simple lever for “AI discovery.” A file. A spec. Something you can ship this week and feel ahead of the curve.

That’s why llms.txt caught on so quickly. The pitch is intuitive: create a machine-friendly inventory or guide for large language models (LLMs), and you’ll show up more often in AI answers.

But Google’s public critique highlights a fundamental problem: the core assumption driving adoption conflicts with the original intent. In plain English, LLMs.txt was not designed to make you discoverable—and treating it like a discovery/Ranking lever is wishful thinking.

This editorial explains what changed, why it matters to SMEs and agencies, what can go wrong if you chase the wrong deliverable, and what to do instead. I’ll also show where AYSA’s AI search visibility workflow fits: monitor what’s happening, prepare fixes, ask for approval, and execute the changes that actually move the needle.

Concise summary

• Discovery still starts on the web layer (HTML pages, links, Crawl paths). A standalone text file doesn’t replace that.
• Google’s point isn’t “never publish helpful machine-readable docs.” It’s that llms.txt is being marketed/used as something it isn’t: an AI discovery mechanism.
• LLMs can’t “trust” self-described inventories for ranking decisions because every site can claim it’s the best source.
• The durable path to AI visibility looks a lot like modern SEO: crawlable architecture, clear entities, structured information, strong Internal linking, and content that actually resolves user tasks.
• Execution matters. Strategy without shipping site changes is theater. AYSA is built to turn Monitoring into Approved Execution.

Table of contents

1. The short version: what changed and why it matters
2. Discovery vs. ranking: the part most “AI optimization” skips
3. What Google says LLMs.txt was originally for
4. The trust problem: why self-claims don’t work for ranking
5. Why LLMs.txt is tempting (and why it keeps spreading)
6. Agentic UX is real—but it’s a different conversation
7. Why HTML still wins: the “boring” foundations that power AI visibility
8. How to structure content so AI systems can use it (without gimmicks)
9. A concrete SME scenario: ecommerce brand chasing AI traffic the wrong way
10. What agencies should change: from deliverables to outcomes
11. A practical 30/60/90-day plan
12. Where AYSA fits: monitoring + approved execution
13. What to do next
14. Sources and further reading

The short version: what changed and why it matters

The most important “update” here isn’t a new Google Algorithm release. It’s clarification of a misconception spreading through the market.

In a discussion highlighted by Search Engine Journal, Google’s John Mueller explains that llms.txt was not conceived as a way for LLM systems (or search engines) to discover all of a site’s content. Instead, the idea was closer to: if a system already knows your site and wants help finding what else is there, a file could potentially help navigate your own content.

That distinction sounds subtle until you translate it into business impact:

• If you publish llms.txt expecting it to get your pages “picked up” by AI answers, you may be investing in a lever that doesn’t exist.
• If you treat it as a navigation aid for agents already on your site, it becomes a different type of asset—more like developer documentation or a lightweight directory.

Mueller’s second point is even more practical: a system can’t rely on self-reported claims for comparing or ranking websites. If every site says “we’re the best” and “these are the pages you must cite,” a competitive retrieval system has to down-weight that type of input.

My take: LLMs.txt is becoming the latest “meta keywords” moment—well-intended, easy to ship, easy to spam, and therefore unlikely to become a reliable ranking signal. You don’t want your growth plan tied to something that fails under adversarial pressure.

Discovery vs. ranking: the part most “AI optimization” skips

To understand why Google’s critique matters, you need a simple model of how web information retrieval works. Search engines (and many AI retrieval systems built on top of web content) need to:

• Discover that a URL exists
• Crawl and fetch it
• Index it into a retrievable structure
• Rank it when a user asks something relevant
• Serve the result (link, snippet, AI answer, citation, etc.)

Discovery is the gate. If discovery fails, everything downstream fails.

The practical reason LLMs.txt doesn’t solve discovery is simple: the web’s discovery layer is still overwhelmingly driven by links, sitemaps, crawlable HTML, and the normal mechanisms of the open web. A separate file that claims your important pages exist does not automatically put those pages into the discovery pipeline of every AI system.

Even if some systems fetch it, you’re still left with the “trust and abuse” problem (we’ll cover that next). In competitive environments, the incentives to manipulate a self-descriptive file are too high.

In other words: you can’t duct-tape your way around the discovery layer.

What Google says LLMs.txt was originally for

According to the Search Engine Journal coverage, John Mueller relayed that one of the proposal’s creators described the original concept as not about discovery. It was closer to a scenario like:

• An LLM or agent already knows your site (maybe a user is on it, or the system has it in context).
• The system wants a compact way to understand what else is available.
• The file helps navigate within that known context.

This matters because most of the market conversation turned llms.txt into an “AI visibility” deliverable—something you add and then expect citations, inclusion, or ranking improvements.

That’s a category error. It mixes up “help an agent navigate once it’s here” with “help the world find you.”

If you’re an SME, your first question shouldn’t be “Where do I put the file?” It should be:

• Are my pages easily discoverable and crawlable?
• Do my pages clearly answer the questions customers ask?
• Do I have a strong, linked information architecture that exposes important pages?
• Is my brand/entity information consistent enough to be recognized and cited?

Those are the levers that persist across classic search, AI Overviews, AI Mode-style experiences, and whatever comes next.

The trust problem: why self-claims don’t work for ranking

Google’s critique also highlights a game theory problem: self-reported files are easy to abuse.

If a ranking system allowed llms.txt to heavily influence which pages to surface, every site owner would be incentivized to:

• List only money pages
• Describe themselves as the best source
• Omit competitors’ context
• Shape the file to manipulate downstream answers

That’s not a moral judgment—it’s simply what happens when a system rewards self-description. The web has already seen this movie (multiple times). When a signal becomes easy to publish and hard to verify, it becomes noisy and loses value.

So even if you assume more AI systems will support a standard like this, you still have to ask:

• Will the system trust it?
• How will it validate it against the HTML?
• What happens when it conflicts with what the page actually contains?

My editorial stance: If a tactic depends on AI systems trusting something you can write about yourself, it’s not a strategy—it’s a temporary exploit at best. SMEs should build assets that remain valuable after platforms harden against manipulation.

Why LLMs.txt is tempting (and why it keeps spreading)

LLMs.txt is spreading because it fits the incentives of three groups:

1) Busy business owners want certainty

If you’re running a clinic, a small ecommerce store, or a local services company, you don’t have time to decode “AI search.” A single-file solution feels manageable.

2) Agencies need a sellable deliverable

It’s easier to sell “we’ll implement LLMs.txt” than “we’ll rebuild your information architecture and rewrite your category pages to satisfy both users and retrieval systems.” One is a line item; the other is real work.

3) The market is anxious about AI-driven zero-click answers

When platforms change distribution, everyone looks for a new knob to turn. LLMs.txt looks like a knob.

But there’s a cost to chasing the wrong knob: you lose cycles that should be spent on fundamentals—crawl paths, indexability, content clarity, entity consistency, and reputation signals.

If you’re deciding where to invest, ask: Is this asset likely to remain useful if AI systems ignore it? If the answer is “no,” it’s probably not your first priority.

Agentic UX is real—but it’s a different conversation

The Search Engine Journal piece also touches on something I agree with strongly: the future isn’t only “AI answers,” it’s AI agents performing tasks on behalf of users.

That means workflows like:

• “Find me a birthday bouquet under $70, deliver tomorrow, include a note, and check out.”
• “Book a hotel near the convention center with free cancellation and late check-in.”
• “Compare these two SaaS tools, then start a trial on the better one.”

In that world, a machine-readable interface—something more actionable than a plain list of URLs—can be valuable. Mueller’s comments (as relayed in the SEJ coverage) frame this as an “agent on your website” problem, not a “discover my site” problem.

Important boundary: Agent enablement is not the same thing as search discovery. You can win at one and lose at the other.

If your site is not easily discoverable and understood via normal web mechanisms, the agent may never arrive. If it does arrive, you still want to make task completion smooth—product finding, filtering, booking, quoting, paying, contacting, etc.

So the strategic sequence is:

1. Be discoverable and credible (web fundamentals)
2. Be usable for humans (UX fundamentals)
3. Be operable for agents (machine-actionable workflows where it makes business sense)

That third layer is where standards may evolve. But you don’t want to skip layers one and two.

Why HTML still wins: the “boring” foundations that power AI visibility

Google’s point that discovery and ranking are still tied to HTML is not a step backward. It’s a reminder that the web’s interoperability layer is still the strongest distribution advantage most SMEs have.

If you want to be cited in AI answers, recommended by AI shopping assistants, or surfaced in AI-powered local experiences, you need:

1) Clean, crawlable information architecture

• Logical navigation
• Strong internal linking to money pages and informational hubs
• Consistent canonicalization
• Non-broken pagination and faceted navigation controls

2) Indexability hygiene

• No accidental noindex
• Correct robots directives
• Accessible server responses (no endless redirects, no soft 404 traps)

3) Content that is structured for retrieval and extraction

• Clear headings and sections
• Definitions, comparisons, specs, and constraints expressed plainly
• Tables where tables help
• Scannable “decision support” content

4) Entity clarity

• Consistent brand, product, and location information
• Transparent authorship and business details where relevant

None of this is glamorous. It’s also the work that compounds.

If you want a north star: build pages that are unambiguous, navigable, and useful even if the reader is a machine. Not because machines are your customer—but because machine interpretation increasingly mediates customer discovery.

How to structure content so AI systems can use it (without gimmicks)

Let’s get concrete. If you want your content to be “AI-usable,” think less about files in your root directory and more about whether your pages can be:

• Found (discoverable URLs, linked from relevant hubs)
• Parsed (clean HTML, stable rendering)
• Understood (clear intent, clear entities, clear relationships)
• Cited (specific, verifiable statements; stable source pages)

Write for resolution, not impressions

Many AI answers collapse the top-of-funnel. Users ask for decisions, not reading lists. Your pages should help a system (and a human) complete a decision:

• What is it?
• Who is it for?
• What does it cost (or what affects cost)?
• What are the constraints (delivery, eligibility, geography, timing)?
• How do I do the thing (book, buy, apply, schedule)?

Use predictable page patterns

SMEs often have inconsistent templates: one service page has FAQs, another doesn’t; one product has specs, another hides them in a PDF.

Standardize. Systems learn patterns. Humans prefer patterns. Your conversion rates usually improve too.

Turn “hidden knowledge” into on-page answers

In many businesses, the real differentiators live in staff heads or email replies:

• “We can deliver same-day if ordered before 1 pm.”
• “We accept HSA/FSA.”
• “This treatment isn’t recommended if you’re on X medication.”

If it’s not on the page, it’s harder to retrieve, cite, and trust.

Make citations easy

AI systems and human writers cite sources that are:

• Specific (not vague marketing)
• Stable (not ephemeral)
• Clearly titled (so the citation makes sense)

Create pages that deserve to be cited: pricing explainers, policy pages, comparison guides, and “how it works” pages that reduce ambiguity.

Don’t confuse “format” with “authority”

Markdown files, text files, and JSON endpoints can help with tooling. But they don’t create authority. Authority is earned by usefulness, consistency, and external validation.

A concrete SME scenario: ecommerce brand chasing AI traffic the wrong way

Here’s a realistic scenario I’ve seen in different forms across the market.

The business

A small ecommerce brand selling specialty home goods (think: candles, skincare, kitchen tools). They rely on organic traffic and returning customers. They’re hearing that “AI answers are taking clicks,” so they want to “be in the AI box.”

What they do first (the wrong order)

• They generate an llms.txt file listing their top categories and products.
• They add a short “AI-friendly” summary to each product page.
• They call it done.

What goes wrong

• Discovery doesn’t improve. Their deep products still aren’t well linked and many are effectively orphaned.
• Indexing remains messy. Parameter URLs (sorting, filtering) create duplicates, while canonical tags are inconsistent.
• AI answers still cite bigger sites. Not because the big sites have a file, but because they have strong internal structure, strong external signals, and clearer “decision” content.

What actually moves the needle

Instead of betting on a file, they invest in:

• Category page upgrades: clear descriptions, buyer guidance, and internal links to bestsellers and subcategories.
• Product template consistency: structured specs, shipping/returns surfaced, FAQs standardized, clear availability signals.
• Internal linking: “best for” collections, editorial guides linking into categories and products.
• Technical controls: consistent canonicals, faceted navigation rules, tidy sitemap strategy.

Even if AI systems never read a single extra file, these improvements make the site more discoverable, more indexable, more understandable, and more cite-worthy.

That’s the real path to AI visibility: build pages that win retrieval.

What agencies should change: from deliverables to outcomes

If you’re an agency, llms.txt is a tempting add-on service because it’s cleanly billable. But it also risks becoming a credibility trap when clients don’t see results.

Here’s the reset I’d recommend.

Stop selling “AI SEO files.” Start selling “AI-ready architecture.”

Clients don’t ultimately care if you shipped a standard. They care if:

• Leads increase
• Calls increase
• Revenue increases
• The brand becomes the default recommendation in its niche

The only durable way to do that is to improve the underlying assets (pages) and distribution (crawl + index + reputation).

Move from “one-and-done” to monitoring-driven iteration

AI search surfaces are changing quickly. If your service is a one-time implementation, you’ll always be behind.

Instead, operationalize:

• Continuous monitoring of visibility and site health
• A prioritized backlog
• Fast, safe execution of changes

This is where systems like AYSA Monitoring are designed to help: identify what’s drifting, what’s breaking, and what’s missing—then convert that into approved work.

Be honest about what you can’t guarantee

No agency should promise “AI citations” as a guaranteed outcome. You can increase the probability by improving structure, clarity, and credibility—but you can’t control every surface.

What you can guarantee is the work: better pages, cleaner architecture, clearer entities, stronger internal linking, better conversion paths, and more useful content assets.

A practical 30/60/90-day plan (SME-friendly)

If you’re a business owner or marketing lead, here’s a plan that doesn’t require betting on speculative standards.

Days 1–30: Get “discovery-ready”

• Audit indexability basics: check accidental noindex, robots rules, canonical consistency.
• Map your money pages: top services/products/locations—are they within 2–3 clicks from the homepage?
• Fix orphan pages: ensure every important page is linked from a relevant hub.
• Clean up duplicates: parameter URLs, thin variants, or near-duplicates that waste crawl and confuse retrieval.

If you need an execution layer (not just a report), explore AYSA’s AI SEO tools and monitoring so you can turn findings into shipped fixes.

Days 31–60: Build “AI-usable” pages

• Standardize templates (service/product/location pages): consistent sections, FAQs, policies, specs.
• Add decision support content: comparisons, “how it works,” “pricing factors,” “best for” guides.
• Improve internal linking: from guides to categories, from categories to bestsellers, from FAQs to service pages.
• Clarify entities: consistent business info, clear definitions, avoid ambiguous naming.

Days 61–90: Earn citations and referrals (the part files can’t do)

• Create cite-worthy assets: policy pages, research summaries (if you have them), transparent pricing explainers.
• Strengthen reputation signals: expert authorship where relevant, partner mentions, community references.
• Measure and iterate: track which pages drive qualified leads and where drop-offs happen.

AI visibility is increasingly correlated with being a high-trust, low-ambiguity source. That’s not a single tactic—it’s a system.

Where AYSA fits: monitoring + approved execution (no risky autopilot)

The biggest gap I see in the market is not “ideas.” Everyone has ideas. The gap is execution without chaos.

Most SMEs and many agencies fall into one of two traps:

• Analysis paralysis: audits, spreadsheets, and “recommendations” that never get implemented.
• Risky autopilot: tools that change things without safeguards, creating brand, compliance, or revenue risk.

AYSA is designed for a third path: monitor → prepare → approve → execute.

• Monitor what matters for search and AI visibility over time (not just a one-off snapshot). See AYSA Monitoring.
• Prepare specific, prioritized changes: internal linking updates, content structure improvements, technical fixes.
• Ask for approval so humans stay in control—especially important for regulated industries and brand-sensitive pages.
• Execute accepted changes so improvements actually ship.

If you’re trying to win in AI search, this workflow matters more than any single file format. Because the work that wins is iterative: pages evolve, templates evolve, products change, services change, and the AI surfaces keep shifting.

If you want to explore how this would look for your site, start with AI search visibility, then review pricing to understand how teams typically roll it out.

For ongoing education and playbooks, the AYSA blog is where we publish practical guidance as the landscape changes.

What to do next

Use this as your next-step checklist:

1. Reframe LLMs.txt: if you’ve implemented it, treat it as optional navigation help—not a discovery strategy.
2. Run a discovery audit: are your important pages linked, crawlable, and indexable?
3. Fix the architecture first: internal linking, hub pages, consistent canonicals, clean sitemaps.
4. Upgrade page structure: templates that surface specs, policies, constraints, and FAQs consistently.
5. Create cite-worthy assets: pages that are specific, stable, and verifiable.
6. Operationalize execution: choose a workflow where monitoring turns into approved, shipped changes (this is where AYSA is built to help).

Sources and further reading

• Search Engine Journal: Google Exposes The Fundamental Flaw Of LLMs.txt (primary source for the reported discussion and quotes)
• Search Engine Journal: Latest news (context for ongoing search/AI updates)
• Search Engine Journal: SEO section (broader coverage and analysis)
• AYSA: AI Search Visibility
• AYSA: AI SEO Tools
• AYSA: Monitoring
• AYSA: Pricing
• AYSA Blog

Note on standards and official documentation: The source discusses proposals and emerging ideas around agentic interfaces. Because the provided research context does not include official specification links for those proposals, I’m intentionally not citing any specific “standard doc” beyond what’s referenced in the source coverage. If you have official links you trust, we can add them in a revision.

Related AI SEO resources

Continue the AI search topic inside AYSA.

Use these pages to connect the article with AI SEO tools, AI visibility monitoring, AI Overviews and approved website execution.

AI SEO tools The commercial hub for AI SEO, AI search optimization and approved execution. AI search monitoring Track AI visibility, answer-engine mentions and the next actions worth approving. AI Overviews SEO Prepare clearer, more structured pages without promising guaranteed inclusion. SEO, AEO and GEO visibility A plain-English explanation of the search visibility layer AYSA prepares for.

Execution hubs

Turn this topic into a website action plan.

Use these AYSA hubs to move from reading to technical fixes, AI visibility monitoring, research, glossary context and approval-first SEO execution.

Technical SEO Open the related AYSA hub for deeper context and next-step workflow. AI search visibility Open the related AYSA hub for deeper context and next-step workflow. Monitoring Open the related AYSA hub for deeper context and next-step workflow. SEO research Open the related AYSA hub for deeper context and next-step workflow. Business owners Open the related AYSA hub for deeper context and next-step workflow. SEO glossary Open the related AYSA hub for deeper context and next-step workflow.

Written by

Marius Dosinescu

Marius Dosinescu is the founder of AYSA.ai, an entrepreneur focused on SEO automation, ecommerce growth, authority building and approved website execution for businesses that want organic growth without specialist overhead.

Author profile Dosinescu.ro

Share inShare on LinkedIn cpCopy article link fShare on Facebook @Share by email