LLMS.txt Isn’t Your AI SEO Strategy: What Ahrefs’ “97% Zero Requests” Finding Really Means (And What To Do Instead)
Ahrefs found 97% of llms.txt files received zero requests. That’s not a reason to panic—it’s a reality check: most “AI discovery” wins won’t come from a new file format. Here’s what changed, why it matters, and a practical action plan for SMEs and agencies to earn visibility in AI answers with measurable execution.
In the last year, llms.txt became one of those ideas that spreads faster than it gets validated. It sounds familiar (like Robots.txt), it feels actionable (“just add a file”), and it promises a shortcut to a hard problem: getting your business discovered and cited in AI answers.
Then Ahrefs dropped a reality check: across a large dataset of domains, 97% of llms.txt files received zero requests, and AI Retrieval bots were only a small share of the total activity observed. That finding was covered by Search Engine Journal (source).
This isn’t a “gotcha” moment. It’s a useful signal that many businesses are spending time on artifacts that don’t (yet) influence the systems that actually drive demand: Google Search, Google’s AI surfaces, and the growing set of AI assistants people use to decide what to buy, where to go, and who to trust.
I’m writing this from the perspective of building AYSA.ai: a system that monitors your site’s search and AI visibility, prepares the exact improvements, asks for approval, and then executes accepted changes. The point of that loop is simple: in SEO—and especially AI Search—strategy without disciplined execution is just storytelling.
Concise summary
- Ahrefs’ data suggests llms.txt currently has little real-world demand across most websites (most files weren’t requested at all).
- Even when llms.txt is requested, it’s often not the bots you care about (auditors, scanners, generic crawlers, and tool ecosystems can dominate activity).
- AI visibility is earned more through being the best, most quotable source—clear content, strong entities, verifiable claims, and technical cleanliness—than through a new file format.
- SMEs should treat llms.txt as “optional hygiene,” not a growth strategy. Prioritize actions that measurably improve discoverability and conversion.
- AYSA’s advantage is operational: continuous Monitoring + prioritized recommendations + Approved Execution, so changes actually ship and outcomes can be measured.
Table of contents
- What changed: the llms.txt hype met request-log reality
- What Ahrefs actually measured (and what it didn’t)
- Why 97% got zero requests: the four most likely explanations
- Robots.txt vs llms.txt: why the analogy breaks
- Who fetched llms.txt (and why that matters)
- The industry studying itself: when tooling outpaces adoption
- What this means for SMEs: focus on outcomes, not artifacts
- What content tends to work well in LLM-driven discovery (practically)
- Technical foundations that still matter more than llms.txt
- A concrete SME scenario: local clinic vs. “just add llms.txt”
- A practical 30-day action plan (SMEs + agencies)
- Where AYSA fits: monitoring + preparation + approved execution
- What to do next
- Sources and further reading
What changed: the llms.txt hype met request-log reality
llms.txt is usually pitched as a way to “help LLMs understand your site” or “tell AI what to read.” The pitch often borrows the mental model of robots.txt: a small, standardized file that bots reliably fetch and follow.
Ahrefs’ analysis (as reported by Search Engine Journal) challenges the assumption that bots actually fetch llms.txt at meaningful scale. Their headline number is what should reset expectations: 97% of llms.txt files got no requests in the measured period. If nobody requests the file, it can’t influence anything—no matter how well you write it.
That doesn’t mean llms.txt is worthless. It means it’s not yet a proven lever for the outcomes most businesses want:
- showing up in AI answers when someone asks for “best [service] near me,”
- earning citations/mentions that drive Referral traffic,
- getting qualified leads from discovery surfaces,
- reducing “unknown unknowns” in how your brand is represented.
If you’re a business owner, the right question isn’t “Should I add llms.txt?” The right question is: What actions are most likely to change my visibility and revenue in the next 30–90 days?
What Ahrefs actually measured (and what it didn’t)
Before we weaponize the “97%” number, we should be precise about what it represents.
Based on the Search Engine Journal summary of the Ahrefs report, Ahrefs analyzed request logs across a large set of domains and classified user agents fetching llms.txt. They observed:
- A meaningful portion of the domains they studied published llms.txt files.
- Only a small subset of those files received any requests.
- Among requested files, the majority of requests came from bots, and many were not “AI retrieval” bots in the way marketers typically mean (live answer engines that cite sources).
What this kind of dataset does well:
- It measures reality: whether the file is actually being fetched.
- It reduces guessing: no need to assume adoption; you can see behavior in logs.
What it doesn’t fully answer:
- Whether fetching leads to compliance: a bot can request a file and ignore it.
- Whether the file affects training, retrieval, or citations downstream.
- Long-term trend: this is a snapshot in time, not a multi-year adoption curve.
So the correct interpretation is not “llms.txt is dead.” It’s: llms.txt is not currently a reliable channel to influence AI discovery at scale, and you should allocate your time accordingly.
Why 97% got zero requests: the four most likely explanations
When a file gets zero requests, there are only a few plausible reasons. Understanding them helps you avoid the wrong conclusions.
1) Most bots don’t know (or care) that llms.txt exists
Robots.txt works because it became a de facto web standard over decades. llms.txt is new, fragmented, and not universally recognized. If a crawler or assistant doesn’t have llms.txt in its “preflight checklist,” it will never request it.
2) AI discovery often doesn’t happen by “fetching your homepage”
A lot of modern discovery is indirect:
- through structured databases and knowledge graphs,
- through third-party aggregators (local listings, review platforms, marketplaces),
- through search indexes that are already curated and filtered,
- through citations in other sources that are easier to trust than self-published claims.
In those flows, llms.txt may never be consulted.
3) Retrieval bots may focus on page-level signals, not site-level instructions
For live answers, systems often need to pull a specific page that answers a question. That implies page-level evaluation (content quality, relevance, freshness, trust) more than site-level “here’s what to read.” A site-wide file can’t substitute for a page that’s actually the best answer.
4) The ecosystem is early: tools are auditing the file before users demand it
SEJ’s summary notes that a notable share of llms.txt requests came from auditors/scanners/validators—tools measuring the existence of the file rather than using it for user-facing retrieval. That’s a classic early-adoption pattern: measurement tooling emerges before mainstream utility.
Robots.txt vs llms.txt: why the analogy breaks
People love the “new robots.txt” framing because it’s simple. But it’s also misleading.
Robots.txt evolved into a widely supported convention for crawl control. It fits the crawler model: bots request a domain, check robots.txt, then crawl pages according to allowed/disallowed rules.
LLM-driven systems don’t operate as one unified crawler. “AI” in practice includes:
- traditional crawlers (search engines),
- training crawlers (collecting data for model training),
- coding agents (fetching docs to complete tasks),
- assistant retrieval bots (pulling pages to answer questions in real time),
- auditing tools (scanning your configuration for “AI readiness”).
They have different incentives and risk profiles. A universal file only works if the majority of relevant agents agree to consult it—and keep consulting it.
That’s why, in the SEJ summary, Google’s John Mueller characterization matters: he has publicly downplayed llms.txt as something “not done for search” and framed it more like a token-saving helper for AI coding tools. SEJ reports that Lily Ray pushed on the discrepancy between Google Search’s position and Chrome Lighthouse auditing behavior. (I’m referencing SEJ’s reporting here; for the underlying primary statements, readers should look for Google’s own documentation or recorded commentary where available.)
Who fetched llms.txt (and why that matters)
The most important part of the Ahrefs finding isn’t just “zero requests.” It’s who makes requests when they happen.
Per the Search Engine Journal coverage of the Ahrefs data:
- Requests came heavily from SEO audit tools, unidentified bots, and web crawlers.
- AI bots were present but the mix leaned toward categories like coding agents and training crawlers, with a smaller share from assistants/retrieval bots commonly associated with AI citations.
- There was a meaningful “meta layer” of bots scanning or validating llms.txt rather than using it to deliver user-facing results.
Here’s why this matters for business outcomes:
- If your goal is citations and discovery, you care about systems that produce answers people read (and click), not validators that score your setup.
- If the primary requesters are coding agents, llms.txt may benefit developer documentation sites more than local service businesses or ecommerce stores.
- If the primary requesters are training crawlers, you may be dealing with dataset ingestion and licensing questions—not lead gen.
So “Should we implement llms.txt?” depends on your business model and the bots that actually touch your site.
The industry studying itself: when tooling outpaces adoption
One of the most revealing notes in the SEJ summary is that a chunk of requests came from tools that audit/scan/study llms.txt files.
This is not unusual in SEO. We’ve seen similar cycles:
- Core Web Vitals tooling exploded before many sites had practical workflows to improve CWV.
- Schema validators became common even though most sites never implemented schema beyond breadcrumbs.
- “AI content detectors” became a product category despite shaky reliability and unclear outcomes.
The pattern is: a new concept appears → tools emerge to score it → marketers optimize the score → outcomes lag.
From an operator’s perspective, scoring without outcomes is dangerous because it creates false progress. It feels like work. It produces a checklist. It generates a report. But it doesn’t change demand.
If you run an agency, this is also where incentives get distorted. Selling “llms.txt setup” is easy. It’s discrete. It’s deliverable. But it can become the new “submit your site to 1,000 search engines” if we aren’t careful.
What this means for SMEs: focus on outcomes, not artifacts
Most small and mid-sized businesses don’t have the time to chase every new technical idea. You need a decision framework.
A simple decision framework: will it change one of these?
When evaluating any AI/SEO tactic (including llms.txt), ask whether it will realistically impact at least one of these within a reasonable window:
- Visibility: Do you appear more often in relevant searches or AI answers?
- Click/share of attention: Do you earn citations, mentions, or clicks?
- Conversion: Do more visitors become leads or customers?
- Cost/time: Does it reduce manual work or prevent costly mistakes?
If the best evidence says “the file isn’t even being requested,” it’s hard to argue for meaningful visibility impact today for most SMEs.
The new SEO question: “Where do people discover us?”
Traditional SEO asked: “How do we rank?”
AI search asks: “Where do people discover us, and what does the system quote when it mentions us?” That includes:
- Google Search and its evolving AI experiences
- Local results (Maps/pack behavior)
- Review sites and directories
- Community platforms and forums
- Third-party comparison lists
Most of those channels don’t care whether you have llms.txt.
What content tends to work well in LLM-driven discovery (practically)
We should separate two concepts that often get lumped together:
- Crawl directives (what bots are allowed to access)
- Answer suitability (whether your content is the best thing to cite)
llms.txt is framed like a directive. Most businesses need to improve suitability.
In practice, content that earns citations or becomes “the source” tends to have these traits:
Clear definitions and direct answers
If someone asks, “How much does teeth whitening cost?” or “What’s the difference between gravel and asphalt for a driveway?” your page should answer in the first 5–10 seconds of reading.
Specificity beats fluff
Vague marketing copy is hard to cite. Specific claims are easy to cite—especially when bounded (“in most cases,” “typical range,” “in our service area”).
Structured sections that map to questions
Use headings that mirror real queries:
- What it is
- Who it’s for / who it’s not for
- Step-by-step process
- Pricing factors
- Timeline / expectations
- Risks and alternatives
- FAQ
Evidence and provenance
When you reference standards, safety, compliance, or research, cite reputable sources. If you can’t cite a primary source, phrase claims carefully. This is both good editorial practice and good “AI citation” hygiene.
Entity clarity (who you are, where you operate, what you do)
SMEs often lose AI visibility because the web has conflicting signals about basics: brand name, address, service area, specialties, hours, policies. This isn’t glamorous, but it’s foundational to being recommended.
AYSA’s approach is to operationalize these improvements: identify the high-impact pages, propose changes, and then execute the accepted updates—continuously. (More on that later.)
Technical foundations that still matter more than llms.txt
I’m going to be blunt: if your technical SEO is shaky, adding llms.txt is like putting a new mailbox on a house with no street address.
Here are technical foundations that tend to matter across both search engines and AI-driven retrieval:
1) Indexing and crawl accessibility
- Make sure important pages are indexable (no accidental noindex).
- Don’t block key resources needed to render content.
- Maintain clean sitemaps and internal linking so important pages are discoverable.
2) Canonicalization and duplication control
If your site produces many near-duplicate URLs (parameters, faceted navigation, printer pages), you create confusion. Confusion reduces the chance that the “best” version is the one that gets cited or shown.
3) Structured data where it’s appropriate
Structured data (schema markup) is not a magic wand, but it’s a clear way to express entities, products, reviews, organizations, locations, and FAQs. That clarity can help systems understand your content.
For a general reference point on structured data, Google’s documentation is the best place to start (primary source): Google Search: Understand structured data.
4) Renderability and performance basics
Not every bot executes JavaScript well. If critical content only appears after heavy client-side rendering, you can reduce accessibility for some systems. Keep key content in the HTML where possible.
5) Trust signals that are visible on-page
- Clear authorship or editorial ownership (where relevant)
- Accurate contact information
- Policies (returns, cancellations, shipping)
- About pages that state who you are and why you’re credible
These aren’t “AI tricks.” They’re the basics of being a dependable source.
A concrete SME scenario: local clinic vs. “just add llms.txt”
Let’s make this real.
Scenario: A multi-location physical therapy clinic wants to be recommended when people ask AI assistants and Google questions like:
- “Best physical therapist for runner’s knee near me”
- “How long does ACL rehab take?”
- “Do I need a referral for physical therapy in [state]?”
What the clinic hears online: “Add llms.txt so ChatGPT knows your site.”
What actually moves the needle:
- Create (or upgrade) one definitive page per condition with: symptoms, when to see a pro, what treatment looks like, typical timelines, pricing/insurance notes, and FAQ.
- Make location pages truly useful: services per location, provider bios, accepted insurance, parking, appointment flow, reviews, and clear contact paths.
- Implement appropriate schema (Organization/LocalBusiness where applicable; avoid spammy markup).
- Fix duplication and thin pages that dilute topical authority.
- Monitor: which pages get impressions, which get clicks, which queries trigger visibility, and whether AI surfaces mention the brand in a way that’s accurate.
Where llms.txt fits here: maybe as a low-effort supplement once the fundamentals are done, especially if the clinic has a knowledge base that could be useful to assistants. But it’s not step one, and it’s not the strategy.
This is exactly why we built AYSA as a loop, not a report: monitoring tells you what’s happening, preparation turns that into a prioritized plan, and approved execution ensures changes go live without chaos.
A practical 30-day action plan (SMEs + agencies)
If you’re reading this as an operator, you want the “do this next” list. Here’s a practical plan that doesn’t require betting on llms.txt adoption.
Days 1–3: Establish a baseline you can measure
- Inventory your money pages: the top service pages, product categories, location pages, pricing pages.
- Pick 10–20 target questions customers ask before they buy.
- Set up monitoring for search visibility and changes over time (AYSA focuses on this operationally: AI search visibility + monitoring).
Days 4–10: Fix technical issues that block discovery
- Resolve obvious indexing mistakes (noindex, canonical mishaps, broken internal links).
- Clean up duplicate and thin pages that compete with your best content.
- Ensure key content is accessible without heavy client-side rendering.
Days 11–20: Upgrade 3–5 pages into “best answer” assets
Pick the pages most likely to be cited or to convert when discovered. Then:
- Rewrite intros to answer the main query immediately.
- Add a structured “how it works” section and clear next steps.
- Include constraints and specifics (service area, timelines, pricing factors).
- Add FAQs that match real buyer objections.
- Add citations to reputable sources where appropriate (standards, safety guidance, definitions).
This is where an “AI SEO tools” approach should help you ship changes, not just suggest them. See: AYSA AI SEO tools.
Days 21–27: Add structured clarity (schema) and internal linking
- Add relevant schema where it accurately reflects the page (avoid marking everything as FAQ if it’s not).
- Link from high-authority pages (homepage, primary categories) to the upgraded assets.
- Ensure each important topic has a “hub” page and supporting pages.
Days 28–30: Decide whether llms.txt is worth adding (now)
Only after the basics are in place, you can decide whether to publish llms.txt as an experiment. If you do:
- Keep it simple and avoid sensitive instructions.
- Don’t assume it changes AI retrieval or citations.
- Watch your logs (or your platform’s monitoring) to see whether anything actually requests it.
Most importantly: treat it like an optional test, not a core growth lever.
Where AYSA fits: monitoring + preparation + approved execution
At AYSA.ai, we care less about “being right on Twitter” and more about building an operating system for search and AI visibility that busy teams can actually run.
That operating system has four steps:
1) Monitor what’s happening
Visibility is moving across channels. You need monitoring that surfaces what changed and where. Start here: AYSA Monitoring.
2) Prepare the specific changes that matter
Not a generic audit. Not a 60-page PDF. A prioritized set of changes tied to pages that drive pipeline and revenue.
3) Ask for approval (because websites are risk)
Most SMEs have been burned by changes that broke layouts, tracking, or conversion flows. “Just publish it” doesn’t work in the real world. AYSA is built around approval as a feature, not a bottleneck.
4) Execute accepted changes and track impact
This is where most SEO programs fail: recommendations don’t get implemented. Approved execution closes that gap.
If you want to explore how AYSA is positioned for this new era, start here:
What to do next
If you only take one thing from Ahrefs’ “97% zero requests” finding, let it be this: optimize for what systems actually use, not what feels like the next standard.
- Don’t treat llms.txt as your AI strategy. Treat it as an optional experiment once fundamentals are strong.
- Invest in 3–5 “best answer” pages that are structured, specific, and easy to cite.
- Fix technical blockers (indexing, duplication, renderability) that reduce discoverability.
- Strengthen entity clarity (who/what/where) across your site and key pages.
- Measure changes continuously, not quarterly.
- Build an execution loop: monitor → prepare → approve → execute. That’s what AYSA is built to do.
Sources and further reading
- Search Engine Journal coverage of the Ahrefs findings: 97% Of LLMS.txt Files Got No Requests, Ahrefs Data Shows
- Google Search documentation on structured data (primary reference): Understand structured data
- AYSA.ai: AI SEO tools: AI SEO Tools
- AYSA.ai: AI search visibility: AI Search Visibility
- AYSA.ai: Monitoring: AYSA Monitoring
- AYSA.ai: Pricing: AYSA Pricing
- AYSA.ai: Blog: AYSA Blog
Continue the AI search topic inside AYSA.
Use these pages to connect the article with AI SEO tools, AI visibility monitoring, AI Overviews and approved website execution.
Turn this topic into a website action plan.
Use these AYSA hubs to move from reading to technical fixes, AI visibility monitoring, research, glossary context and approval-first SEO execution.
