llms.txt: what the file does, and who actually reads it

Ahrefs checked 137,210 domains with traffic in May 2026 and found that 97 % of the llms.txt files published on them were fetched by nothing at all that month. Not a search engine, not an assistant, not a person. The file takes about twenty minutes to write, so the useful question is not how to write it. It is what those twenty minutes buy when almost nothing is reading.

What is llms.txt, and does anything read it?

llms.txt is a proposed plain-text file at a site's root that lists its main pages in Markdown for language models. Google documents that no such file is needed. On a site under 30 pages it changes nothing, on a few hundred it works as a curated index, and on documentation it is the case it was built for.

Quick answer

• It is a proposal, not a standard. Jeremy Howard published version 1 on 3 September 2024 and version 2 on 10 August 2026. No search engine has adopted it.
• Google names the file by name and calls it unnecessary. Its guide to optimizing for generative AI features, updated on 10 July 2026, tells readers they can ignore "creating unnecessary AI text files (like llms.txt)". Its page on AI features in Search says the same thing without the parenthesis.
• The measured read rate is close to zero. In the Ahrefs study, 97 % of published files received no requests during the month measured.
• The one family that does fetch it writes code. In that same data, Claude-Code, the Anthropic coding agent, out-fetched every AI retrieval bot, assistant and training crawler.

The analogy everyone brings, and why it misleads

Almost every guide opens by calling this file the robots.txt of AI. The comparison is comfortable and it is wrong in the way that matters.

robots.txt is a permissions file. It grants or denies crawling, a crawler that ignores it can be caught doing so, and a decade of tooling has been built on the assumption that it gets read. llms.txt grants nothing and denies nothing. It is a claim you make about your own site, in Markdown, in the hope that something on the other side finds it worth loading.

That difference decides everything downstream. A permissions file has a consumer with a reason to fetch it. A recommendation file only has a consumer if somebody chooses to build one, and that choice has not been made by anyone whose citations a business would want. Nothing about publishing the file obliges a single system to load it, and no version of the format changes that, because a convention with no enforcement is a convention with no leverage.

The second half of the misleading analogy is subtler. Adoption of robots.txt is measured by who obeys it. Adoption of llms.txt gets measured, in almost every article on the subject, by who publishes it. Those are opposite ends of the same transaction, and only one of them costs the publisher anything.

What server logs say about who fetches it

The method first, because the conclusion depends on it.

Ahrefs published the study on 15 June 2026, written by Louise Linehan. It covers 137,210 domains in Ahrefs Web Analytics that received traffic during May 2026. Each domain root was checked for an llms.txt returning HTTP 200, and every request to those paths was then classified by response code and user agent. The authors attach their own limit to it, which is worth repeating because most write-ups drop it: their customers skew more technical and more SEO-aware than the web at large, so the adoption figure should be read as an upper bound.

Adoption, upper bound: 28 %. That is 38,360 domains serving a valid file.

Reads: 97 % of those files received zero requests in the month. Inside the 3 % that got anything, 96 % of the requests came from bots, and only 19.5 % of the fetches came from named AI tools. Retrieval bots, the category that actually feeds AI search answers, accounted for 1.1 % of requests. OAI-SearchBot led that category with 0.74 %. GPTBot, a training crawler, was the single biggest fetcher at 4.51 %.

That split matters more than the headline, because those two names do different jobs and get conflated constantly. A training crawler collects material for a future model; a retrieval bot goes and fetches pages while an answer is being written, which is the step a citation comes out of. Which crawler belongs to which job, and what blocking each one costs, is set out in how to get cited by ChatGPT.

And the finding that settles the question is a negative one. No AI bot ever requested an llms.txt that did not exist. The AI share of the 404s was zero. Nothing out there is probing domains for the convention, which is what adoption would look like from the server side.

Set that against how the work is normally sold. The layer that does have measured effects on generated answers is editorial rather than infrastructural, and the evidence for it is laid out in generative engine optimization.

Google says no, Chrome audits for it anyway

Two parts of the same company give two different answers, and both are correct.

Google's documentation on AI features in Search, opened again on 5 September 2026, is explicit: "You don't need to create new machine readable files, AI text files, or markup to appear in these features." It adds that there is no special schema.org structured data to add either, and states the requirement plainly: a page has to be indexed and eligible to be shown with a snippet. Nothing else. Indexing being the gate in front of all of this is the reason the diagnosis order in why is my page not indexed comes before any of it.

Then, in May 2026, Chrome added an Agentic Browsing category to Lighthouse, and one of its discoverability audits checks for llms.txt at the domain root. Search Engine Land reported it on 20 May 2026, quoting John Mueller on the same page: the short answer is that it is not done for search.

The two statements only look contradictory if agent readiness and search ranking are treated as one thing. Lighthouse is scoring how a site behaves toward software that drives it; Search is scoring whether a page deserves to be shown. A failing agentic audit is a compatibility note, not a penalty, and reading it as a ranking signal is the same error as reading a green Lighthouse performance score as a promise of position. What does move a generated answer sits a layer above any file at the root, in the passage a system can lift out and attribute, which is where answer engine optimization vs SEO begins.

None of it is checkable from the outside. A verified property reports which pages Google stored, which queries surface them and at what position, and that report is where an argument about any of this stops being an opinion. The afternoon it takes is in how to set up Search Console.

Writing one: the shape, and what version 2 moved

The format is short enough to hold in your head. An H1 with the site name, which is the only required part. A blockquote under it with a summary. Then H2 sections holding lists of links in the form - [Name](url): one line about it. By convention a section called Optional collects links an agent can skip when it needs a shorter context.

# Riverside Dental

> Family dental practice in Yonkers, New York. Cleanings, implants and emergency
> visits, in English and Spanish.

## Main pages

- [Services](https://example.com/services): what we treat and what it costs
- [Our dentists](https://example.com/team): who works here and their credentials
- [Emergency visits](https://example.com/emergency): same-day appointments and hours

## Optional

- [Legal notice](https://example.com/legal)

Version 2, published on 10 August 2026, is the first revision since the format launched, and it moved two things worth knowing. It adds formal ways for an agent to find a Markdown copy of a page, either at a companion .md address or through rel="alternate" type="text/markdown" and rel="describedby", delivered as HTML link elements or as an HTTP Link response header. And it keeps Optional as a heading convention while dropping the mechanical weight version 1 gave it. If you wrote a file in 2024, the update is a small edit rather than a rebuild.

Filling in fields beats writing Markdown by hand, and the llms.txt generator on this site assembles the file in the browser from a site name, a summary and one line per link, with no sign-up and nothing leaving the machine it is typed on. The llms.txt validator checks the other half: one title on the first line with content, a summary line, sections that are not empty, links in the correct bracket form, absolute URLs, and duplicates. What it does not check is whether those URLs resolve, because that would need a server and the page deliberately has none. That gap is the one part only a human eye closes, and saying so is better than hiding it.

The last check is the one that quietly fails most often. A file that returns the theme's HTML with a 200 status is not a file, and curl -sI https://example.com/llms.txt settles it in a second: a 200 and a plain-text content type, or the thing does not exist where you think it does. Serving checks of this shape are collected in the website launch checklist.

When NOT to publish one

When the pages are not indexed yet. A curated index of content that no search engine has stored describes a site that, as far as any retrieval system is concerned, is not there.

When nobody is going to maintain it. This file is written by hand and does not regenerate itself, so at six months it points at three articles that moved and one that was deleted. An index that leads to 404s teaches a reader, human or otherwise, that the site is unreliable, which is a worse outcome than silence.

When the demand is local. A business that lives on searches with a place name in them gets almost nothing here, and the same hours put into reviews and the profile return more, as the ordering in local SEO for small business shows.

What the twenty minutes actually buy

Here is the argument that survives the measurement, and it has nothing to do with machines.

Filling the file forces a decision most sites have never made: which twenty pages represent what this business knows. That list almost never matches the twenty pages with the most traffic, and where the two diverge you have found a real architecture problem rather than a missing file. The same discipline applied to the whole site, in the order that keeps the cheap fixes first, is the basic SEO checklist.

This site runs its own version of that. The llms.txt at alexgonzalezgo.com is generated by the publishing script alongside the sitemap, from published articles only, so it cannot describe a site that no longer exists. Checked on 5 September 2026 it served a 200 with a plain-text content type, carried 94 links across 4 sections, 85 of them articles, and passed the validator published on this same domain with zero errors and zero warnings. Publishing a generator without having the file would have been the same hole as selling audits without passing your own.

It also demonstrates the failure mode, which is the honest part. That file lists every published article, which is the dump-the-sitemap pattern under a different name. The validator on this site warns above a hundred links because an index carrying everything filters nothing, and the queue publishes three articles a day, so it crosses that line on its own within days of the count above. The fix is not raising the threshold; it is deciding what belongs in the list, which is the same decision the file existed to force in the first place. Anyone being sold this file as an AI search deliverable is being sold twenty minutes of typing at a monthly retainer rate, and separating a vendor who sells work from one who sells a filename is what the questions in how to choose an SEO agency are for.

Mistakes that repeat

• Reading publication as adoption. OpenAI, Anthropic and Perplexity each serve one for their developer docs, which says they publish, not that they consume.
• Dumping the whole sitemap into it. A file that carries everything filters nothing, and filtering was the entire proposition.
• Filling it with marketing adjectives. They occupy the space where an entity a model could use was supposed to go.
• Expecting traffic from it. At best it changes how something describes you, and it has no path to a visit.
• Publishing it while blocking the crawlers. The two files then tell opposite stories about the same site, and only one of them is enforced.
• Repeating an uplift percentage. Figures in the 30 to 45 % range circulate for this file with no published method behind any of them, and none names a study.

Data and transparency

The read-rate figures come from the Ahrefs study published on 15 June 2026 by Louise Linehan, opened on 5 September 2026: 137,210 domains in Ahrefs Web Analytics with traffic during May 2026; 28 % of them, or 38,360 domains, serving a valid file, with the authors' own note that their customer base skews technical and that this figure is an upper bound; 97 % of those files receiving zero requests in the month; 96 % of the requests that arrived coming from bots; 19.5 % of fetches from named AI tools; 1.1 % from retrieval bots; 0.74 % from OAI-SearchBot; 4.51 % from GPTBot; and zero AI requests for files that did not exist.

That no machine readable file, AI text file or markup is needed to appear in AI features, and that the requirement is being indexed and eligible to be shown with a snippet, is quoted from Google's own documentation on AI features in Search, opened on 5 September 2026. The version dates and the format description come from the specification at llmstxt.org, opened the same day, which is where version 2 of 10 August 2026 is published. That Chrome added an Agentic Browsing audit checking for the file, and the accompanying remark that it is not done for search, was reported by Search Engine Land on 20 May 2026 and the audit itself is documented on developer.chrome.com.

The search demand for this query, 400 monthly searches in the United States with a keyword difficulty of 31 and a cost per click of 0.80 USD, and the absence of an AI summary on its results page, come from this blog's own keyword measurement for this article and are not a third-party publication. The uplift percentages named in the mistakes above are named in order to be dismissed: none of them carries a method or a study, which is the reason they appear here without being used anywhere else. The figures about this site's own file were measured against the served domain on 5 September 2026. The operational judgment behind the article comes from a portfolio recording more than 300 million impressions a year in Search Console. Verified as of September 2026.

Primary sources, opened on 5 September 2026: the Ahrefs llms.txt study; Google's documentation on AI features in Search; Google's guide to optimizing for generative AI features, which names the file, last updated 10 July 2026 and opened on 6 September 2026; the llms.txt specification; the Lighthouse agentic browsing docs.

What this changes

The tempting conclusion from a 97 % non-read rate is that the file is a waste of an afternoon. That is the wrong lesson, and the right one is more uncomfortable.

Almost nothing on a normal business site is written to be lifted out and used by something that did not arrive through the front door. The file is only the first place that becomes obvious, because it forces you to name twenty pages and then look at the list. When the list is thin, the missing thing was never the file at the root. It was that a site can rank, convert and pay for itself while containing nothing a stranger would repeat, and no format published at a domain root fixes that.

Frequently asked questions