llms.txt Has Adoption but No Evidence Behind It

llms.txt has exactly one outcome test behind it, and that test found the file made a citation-prediction model less accurate rather than more.

Shipping one is a courtesy to future crawlers, not a lever on current ones.

Every llms.txt AI citation claim now in circulation traces back to a single vendor study, and that study is the one that found its model improved when the file was taken out as a variable. SE Ranking scanned nearly 300,000 domains to get there.

I run a hand-built file on both sites I own and have never billed an hour for it. Work through the four access constraints below and you will know whether an engine can reach your pages at all, which is the one question this file cannot answer.

Adoption Is Real, a Citation Effect Is Not

Two claims travel together in llms.txt coverage, and only one of them survives contact with data. Adoption is measurable and modest. A citation benefit has never been measured at all.

An llms.txt file is a plain-text markdown document at a site’s root listing the pages an owner wants an AI system to treat as canonical, each with a one-line description. The specification sets the format: an H1 with the site name, a blockquote summary, then H2 sections of linked pages. It is a curation layer, and it grants or denies a crawler nothing.

That is the answer in one paragraph, and the rest is mechanism.

This is the second claim I have put through the Verdict Model set out in What the AEO Evidence Actually Shows Right Now: state the claim as it circulates, find the test, land a verdict, then say plainly what the test cannot tell you.

Schema markup went through the same four steps and produced a similar shape of answer. A real test existed, and it did not say what practitioners were repeating.

Here is llms.txt run through the same four steps.

• Claim: Adding llms.txt improves a site’s odds of being cited by AI systems.
• Test: SE Ranking’s scan of nearly 300,000 domains, paired with an XGBoost regression model predicting citation frequency.
• Verdict: Adoption is real and flat across site sizes. The file’s contribution to the model was negative.
• Limit: One vendor, one model, one run, no peer review, and no time series behind it.

The adoption half is worth looking at first, because it is the half that gets quoted.

Ten Percent of the Web, and Nobody in Particular

SE Ranking’s llms.txt study, published November 7, 2025, is the only large scan I can trace to a stated sample and a stated method. It found the file on 10.13 percent of nearly 300,000 domains.

What makes that number interesting is how little it moves by traffic band.

Site traffic band	Share running llms.txt	Source
0 to 100 monthly visits	9.88%	SE Ranking, published November 7, 2025
1,001 to 5,000 monthly visits	10.54%	SE Ranking, published November 7, 2025
100,001+ monthly visits	8.27%	SE Ranking, published November 7, 2025
All domains scanned	10.13%	SE Ranking, published November 7, 2025

The largest sites in the sample are slightly less likely to run the file than mid-sized ones. Under three percentage points separate the top band from the bottom.

That flatness is the finding, not the headline number.

If llms.txt were quietly working, the sites with the most to gain and the most budget to test would be over-represented. They are not, and SE Ranking reads its own spread the same way: a scattered experiment rather than an emerging best practice.

• Adoption sits at roughly one site in ten, which is a minority behavior and not a norm.
• The gap between the heaviest and lightest traffic bands is 2.66 percentage points.
• This is one dated snapshot, not a tracked series, so no direction of travel can be read off it.

None of that says whether the file works. That is a separate measurement, and a harder one.

Inside the Only llms.txt AI Citation Test on the Record

How the Model Was Built

SE Ranking paired its domain scan with citation data and trained an XGBoost regression model to predict how often a domain gets cited by large language models. Measurable site factors went in as candidate variables, and llms.txt presence was one of them.

XGBoost is a gradient-boosted tree model. It is built to separate the inputs that carry signal from the ones that do not. A variable that makes such a model worse is telling you something.

What Removing the File Did to It

Taking llms.txt out of the model improved its accuracy.

SE Ranking’s own summary is blunt. Both its statistical analysis and its machine learning showed no effect of llms.txt on how often a domain gets cited, and the variable looked more like noise than value.

That is not a finding that the file is harmful. It is a finding that the file carried nothing the model could use.

An independent check arrived from a different direction in August 2026, when Mark Williams-Cook of Candour published cats.txt, a deliberately absurd file documenting his office cats, their job titles and their affection scores.

He then tested it against the four proofs usually offered for llms.txt. The joke file passed all four.

Bots crawled it. Google indexed it. Models repeated its invented details as fact, and ChatGPT endorsed the format as good for rankings.

• Measured: Citation frequency per domain, against a set of observable site factors.
• Result: llms.txt presence reduced the model’s accuracy, and removing it improved the model.
• Cross-check: A file about fictional cats cleared every proof commonly cited for llms.txt.

Crawling is not consumption, and an endorsement from a language model is a reflection of its training data. A single vendor study still deserves to be read at its actual weight, and that weight is where I want to be careful.

Reading a Single In-House Study at Its Real Weight

The SE Ranking work is in-house research. It was run once, it is not peer reviewed, and the model behind it is proprietary.

A weak variable degrading a tree model is a routine statistical outcome. It means the variable did not help this model do this particular job.

It does not mean that shipping llms.txt costs a site citations, and I would not want anyone quoting me as if it did.

The scan also cannot separate a hand-built file from a plugin-generated one. A curated map of forty pages and an auto-dumped list of every URL on a site both register as present.

So the honest reading is narrow.

One credible attempt to connect the file to an outcome found nothing, and no attempt has found the opposite. That is the whole evidentiary position, and anything stronger in either direction is somebody’s preference wearing a study as a jacket.

• One vendor, one model, one run, no replication.
• No control for how well each file was built.
• No causal design, so even a positive result would not have proved cause.
• No published counter-test showing a citation gain.

Which leaves the practical question of what actually decides whether an engine reaches a page.

Crawler Access Is the Real Constraint, and llms.txt Sits Downstream

Four things gate whether an AI system can use a page, and llms.txt touches none of them. Each has a fix that is duller and more reliable than adding a file.

Gotcha One: robots.txt Decides Who Gets In

Google publishes the full list of crawlers it operates, with separate user agents for search, for extended AI training, and for other products. OpenAI documents GPTBot the same way, and Anthropic documents its crawler and how to block it.

None of those pages lists llms.txt as an input. All of them describe a user agent that your robots.txt either admits or refuses.

Open the file. Read the disallow lines against that list of agents. It is a five-minute job, and the only one here that can silently zero out everything else you do.

Gotcha Two: Pages That Need JavaScript May Never Render

Google documents Googlebot’s rendering behavior in detail, down to the 2MB crawl limit per file and the separate fetch of every referenced CSS and JavaScript resource. Several AI crawlers do not execute JavaScript at all.

Aleyda Solis tested that gap directly and found ChatGPT leaning on Google’s SERP snippets rather than fetching and rendering the page itself.

HubSpot’s guide to getting indexed by ChatGPT states the consequence plainly: OpenAI’s crawlers do not render JavaScript, so a sentence that only appears after a client-side fetch may never be seen at all.

Server-render anything you want quoted. A sentence that only exists after a client-side fetch is not made reachable by a curated link pointing at the page it lives on.

Gotcha Three: Nothing Documented Reads the File

John Mueller was asked about llms.txt at Google Search Central Live in Madrid on April 9, 2025. Aleyda Solis’s write-up of that session records his answer: the file only makes sense if a system does not already know your site, and he was not aware of any AI system using it.

Sixteen months on, no provider has published anything that contradicts him.

There is no fix for this one, only a scope correction. Treat the file as optional until a provider documents reading it, and hold every claim about it to that standard.

Gotcha Four: A Generated File Repeats Your Sitemap

A plugin that dumps every URL into llms.txt has built a second sitemap in a format nothing consumes. The curation is the only part of the idea that was ever doing work.

Structure on the page itself is where the same effort pays. Seer Interactive changed the wording in its own site footer and saw ChatGPT reflecting the new phrasing within thirty-six hours, which is a useful reminder of what these systems demonstrably read.

If you ship a file, hand-write it, keep it to the pages you would defend, and skip it altogether if you will not maintain it.

• Permission: Check robots.txt against the published user-agent lists before anything else.
• Rendering: Serve the sentences you want quoted in the initial HTML response.
• Readership: Assume nothing reads llms.txt until a provider says otherwise.
• Redundancy: A generated file duplicates the sitemap you already have.

Against that list, the decision about whether to ship the file at all turns out to be much smaller than the argument around it.

Founder Insight: Free and Harmless Is a Real Category Not every technical task needs a measured return to justify an hour, and llms.txt is the clearest current example of one that does not. The line I hold is that an unmeasured hedge never gets described to a client as a lever.

Ship the File, Sell Nothing With It

Both sites I own run a hand-built llms.txt today. The TDM Insights file is 85 lines and roughly 17 KB; The Gourmet Host’s is 92 lines and roughly 10 KB.

Both are written by hand, sectioned by topic, with a one-line description under every link. Neither was generated.

I built them because a curated list of the pages I want representing each site is useful to me regardless of who reads it. The justification stops there, and it is enough for an hour of work.

What I do not have is a before-and-after test isolating their effect.

Given SE Ranking’s result I would not expect one to show much, and I am not going to run a single-site test on a property I own and present it as evidence.

• Ship it if the hour is genuinely spare and you will keep it current.
• Curate it by hand, because a generated dump is a sitemap with extra steps.
• Never present it as an AI visibility tactic or bill for it as one.
• Fix crawler permission and rendering first, in that order, every time.

For the how-to side of the file itself, llms.txt Explained: Should Your Site Have One? walks through building one, and How AI Answer Engines Choose Their Sources sets out the retrieval path a curated file was meant to serve.

Neither of those is the argument here, which is about where a spare hour earns the most.

Where the Hour Actually Goes

Given one hour and a site that has never been checked for access, spend it on robots.txt and rendering rather than on llms.txt. The technical checklist I work from, ranked by severity, puts crawl and index faults above everything cosmetic for that reason.

The order is not a preference. A blocked crawler makes every downstream improvement unreachable, and no file at your site root overrides it.

Crawl Budget and Indexation for Small Sites covers the reachability half of that, and How to Structure a Page to Rank and Get Cited covers the layout half.

If a widely adopted free fix turns out to have no measured effect, the headline statistics behind the rest of the checklist deserve the same scrutiny, which is the subject of Every AI Overview Statistic Contradicts the Next.

Adoption told us how many people shipped a file. It never told us whether anything on the other end was listening, and until something is, llms.txt stays a hedge rather than a plan.

Frequently Asked Questions

What is an llms.txt file and where does it live?

An llms.txt file is a plain-text markdown document at a site’s root that lists the pages the owner wants AI systems to treat as canonical, each with a short description. The published specification defines the format: a site-name heading, a summary blockquote, then sections of linked pages.

How many websites actually have an llms.txt file?

SE Ranking’s scan of nearly 300,000 domains, published November 7, 2025, found the file on 10.13 percent of them. Adoption was almost flat across traffic bands, running 9.88 percent at the low end and 8.27 percent among sites above 100,001 monthly visits.

Does having an llms.txt file measurably help a site get cited by AI?

No published test has shown that it does. The one study that used llms.txt as a predictive factor found its citation model became more accurate once the factor was removed. Adoption and effectiveness are separate claims, and only the first has been measured.

Should I still add an llms.txt file if there is no proven citation benefit?

I do, on both sites I own, because a hand-curated file costs about an hour and no test has shown a downside. I do not sell it as an AI visibility tactic and I do not bill for it, because an unmeasured hedge is not a lever.

Do robots.txt and crawler permissions matter more than llms.txt?

Yes. Google, OpenAI and Anthropic all publish the user agents they operate, and your robots.txt decides which of them can fetch a page at all. A blocked crawler or a JavaScript-only page defeats every other optimization, and llms.txt cannot override either.

What should I prioritize instead of llms.txt for AI visibility?

Prioritize crawler permission, server-rendered HTML, and clean on-page structure, in that order. Those three decide whether an engine can reach and parse your content. llms.txt sits downstream of all of them and cannot repair a single one on its own.

Continue Reading

More From This Series

• What the AEO Evidence Actually Shows Right Now
• Schema Markup and AI Citation: What a Test Found
• Every AI Overview Statistic Contradicts the Next
• Brand Mentions Beat Backlinks in Ahrefs’ Own Data
• Being Cited by AI Is Not the Same as Being Chosen

More from TDM Insights

• llms.txt Explained: Should Your Site Have One?
• How AI Answer Engines Choose Their Sources
• Technical SEO Checklist for Founders, by Severity
• Crawl Budget and Indexation for Small Sites
• How to Structure a Page to Rank and Get Cited

Explore TDM Insights Topics

• SEO
• AEO
• Content
• Case Notes