llms.txt Explained: Should Your Site Have One?
llms.txt tells AI which pages matter. Do engines read it yet, is it worth adding, and what actually controls AI crawler reach.
By David Jubé · May 30, 2026 · 13 min read
llms.txt is a proposed plain-text file that the major AI answer engines mostly do not read yet, so the honest answer to “should my site have one” is this: it is cheap to add and it is not where your citations are won.
Add it if you want the insurance. Do not expect it to move a single AI answer in your favor today.
The work that actually decides whether an AI engine can use your content is older and less glamorous than a new file in your root directory. Can its crawler reach your pages? Can it parse them once it gets there?
That is the Retrieval step from the model for how AI engines find, evaluate, and cite a source, and llms.txt sits to the side of it, not at the center.
This article gives you the real verdict, then spends the bulk of its length on the access fundamentals that govern reach. The file is the easy part. The crawler is the part that pays.
Key takeaways
- llms.txt is a proposed plain-text file at your site root that points AI systems to your most important pages, but the major engines mostly do not read it yet, so it is cheap insurance, not where citations are won.
- As of mid-2026 no major provider (OpenAI, Google, Anthropic, Perplexity) has confirmed using llms.txt, and tracked logs show close to zero crawler visits to the file while the same crawlers hammer normal pages.
- robots.txt and sitemap.xml are widely honored standards that do real work; llms.txt is a proposal with no enforcement behind it, so robots.txt is where AI access decisions actually get made.
- Retrieval has three failure points that have nothing to do with llms.txt: crawler access, client-side JavaScript rendering that hides your content, and weak page structure that buries the answer.
- The engines run several crawlers per provider, so control access per user-agent by name in robots.txt; blocking the training crawler like GPTBot does not block the search crawler like OAI-SearchBot that earns citations.
The One-Line Verdict
Adding llms.txt is low cost, low risk, and currently low return. Treat it as a five-minute hedge against a future where engines start honoring it, then put your real effort somewhere that pays off now.
That is the whole decision. Everything below is why, and what “somewhere that pays off now” means in practice.
What llms.txt Actually Is
llms.txt is a plain-text file you place at the root of your domain, at yoursite.com/llms.txt, that points AI systems toward the pages and summaries you consider most important. It was proposed in 2024 as a curated map for large language models, a way to say “if you are going to read me, read these pages first, and here is a one-line description of each.”
The format is deliberately simple. It is Markdown, it leads with your site name and a short description, and it lists your priority links under headings. A minimal one looks like this:
# TDM Insights
> SEO and answer-engine optimization, explained founder to founder.
## Core guides
- [What AEO actually is](https://tdminsights.com/insights/answer-engine-optimization-what-aeo-actually-is/): the definition and the model
- [How AI engines choose sources](https://tdminsights.com/insights/how-ai-answer-engines-choose-their-sources/): retrieval, evaluation, citation
## About
- [Who we are](https://tdminsights.com/about/): the team and the proving groundThe pitch is reasonable on its face. A model has a limited context window, the web is noisy, and a hand-curated list of your best pages in clean Markdown is friendlier to a language model than asking it to crawl and rank your whole site.
The idea is sound. The adoption is the problem.
Do the Engines Actually Read It Yet?
Mostly not, as of mid-2026, and this is the part the breathless “add this file now” posts skip.
No major AI provider has confirmed using llms.txt to source or cite content. Not OpenAI, not Google, not Anthropic, not Perplexity.
When practitioners went looking for evidence the file was being fetched at all, they came up close to empty: tracked crawler logs show close to zero visits to the file from GPTBot, ClaudeBot, PerplexityBot, or Google-Extended over a multi-month window, while the same logs show those crawlers hammering normal pages.
The platforms that actually meter how AI crawlers reach a site point control back at robots-level access, not at a curated file the engines do not request. A practitioner explainer from Ahrefs on what llms.txt is lands in the same place, and trade reporting that llms.txt adoption has stalled documents the major platforms declining to commit.
So here is what llms.txt can do today:
- Sit in your root directory at no cost.
- Position you to benefit instantly if an engine flips it on tomorrow.
- Serve as a clean, human-readable index of your best content, which is mildly useful for your own reference.
And here is what it cannot do today:
- It cannot get you cited.
- It cannot override what your robots.txt says.
- It cannot make an unreachable page reachable, or a JavaScript-buried answer parseable.
It is a suggestion to systems that are not yet listening, which is why the cost-benefit math is simple: spend five minutes, expect nothing, move on.
One caveat matters here. Adoption is moving. Write the date on any claim you read about llms.txt, including this one, and re-check it, because a standard that is ignored in June can be honored by some engine in October.
That possibility is exactly why the file is worth the five minutes. It is also why it is not worth fifty.
llms.txt vs robots.txt vs sitemap.xml: Who Does What
These three files get blurred together because they all live in your root and all talk to crawlers. They do completely different jobs, and only two of them are honored.
The practical reading of that table: robots.txt is your access control and AI crawlers actually read it, so it is where access decisions get made. Google’s introduction to robots.txt documents the format the AI crawlers also follow.
sitemap.xml is your discovery surface and it works. llms.txt is the proposal, and it is the one with no enforcement behind it.
If you are going to spend an afternoon on a root-directory file, robots.txt earns it. The other two are quicker wins many sites already have in place.
The Real Lever: Can the Crawler Reach and Parse You
Here is where the effort belongs. An AI engine cannot cite a page it never retrieved, and retrieval has three failure points that have nothing to do with llms.txt.
The first is access. AI crawlers obey robots.txt, so a blanket disallow, an overzealous firewall rule, or a bot-management product set to “block everything that looks automated” can quietly lock the engines out. If GPTBot or PerplexityBot gets a 403 at your door, no curated link list inside the building matters.
The second is rendering. If your important content only appears after client-side JavaScript runs, a crawler that does not execute that JavaScript sees an empty shell.
Google’s own guidance on rendering on the web walks through why server-rendered or pre-rendered HTML is the safe default: the content has to be in the markup the crawler receives, not assembled in a browser the crawler may not run. Google’s JavaScript SEO basics spells out the same trap for search. This is the single most common reason a page that “should” be cited is not even in the candidate set.
The third is structure. Even reachable, rendered content can be hard to parse if the answer is buried, the headings are decorative rather than semantic, or the page is a wall of undifferentiated text.
Clean HTML with real heading hierarchy is what lets a parser locate the passage worth lifting. If your site has deeper indexation trouble, the crawlability and indexation fixes in the SEO technical cluster go further than this page can.
Access, rendering, structure. Get those right and the engine can use your content. Get them wrong and llms.txt is a note pinned to a locked door.
Book a free diagnosis
If you have added llms.txt and your pages still never show up in AI answers, the file was never the problem, and you probably suspect that already. The likely culprit is upstream: a crawler blocked at the edge, an answer that only renders in JavaScript, or a page no engine can parse cleanly. We will run your priority pages through the Retrieval checks, founder to founder, and tell you the one access fix that clears the path for the rest. No deck, no retainer pitch, just a clear read.
AI User-Agents: What to Allow, What to Block
The engines do not run one crawler each. They run several, with different jobs, and the difference matters because blocking the wrong one costs you citations while blocking the right one protects training data you did not want scraped.
The major engines each publish which agents they run and what each one does:
- OpenAI’s documentation on its crawlers names three: GPTBot, which gathers data that may train models; OAI-SearchBot, which builds the index behind ChatGPT search; and ChatGPT-User, which fetches a page live when a user asks about it.
- Perplexity’s crawler documentation draws the same line between PerplexityBot, the indexing agent, and Perplexity-User, the live fetch.
- Anthropic’s documentation on its web crawler covers ClaudeBot the same way.
The trap is treating all of these as one decision. If you disallow GPTBot to keep your content out of model training, you have not touched OAI-SearchBot or ChatGPT-User, the agents that actually surface you inside an answer.
Block the training crawler and keep the search crawler, and you protect your content while staying citable. Block both by accident and you have removed yourself from the room.
This is a per-agent choice, made in robots.txt, by name. It is the access decision llms.txt cannot make for you, which is exactly why robots.txt is where the real control sits.
A 15-Minute Access Check
Run this before you spend a minute on llms.txt. It is the Retrieval-step work that decides whether anything downstream matters.
- Read your robots.txt for AI user-agents. Confirm you are not blanket-disallowing GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, or Google-Extended unless you mean to. Decide each one deliberately.
- Check the live fetch. Load a priority page with JavaScript disabled. If the core answer disappears, your content is rendering client-side and crawlers may not see it. Move it into server-rendered HTML.
- Check the edge. Ask whether a firewall or bot-management rule is returning errors to known AI user-agents. A 403 to GPTBot is invisible in your analytics but fatal to your citations.
- Check the structure. Confirm your important answers sit in real headings and clean paragraphs, not buried in decorative markup or images of text.
- Then, optionally, add llms.txt. A clean Markdown index of your best pages, in your root. Five minutes, zero expectation, free upside if an engine ever honors it.
The order is the point. Items one through four decide whether you get cited. Item five is the hedge you add after the real work is done, not instead of it.
Once the crawlers can reach you, the next job moves inside the page: making the content itself easy to lift, starting with the most extractable block you own. That is the FAQ block, rebuilt for answer engines, the next step in the full liftable-page checklist. For how one engine’s crawler reaches and ranks you specifically, the Perplexity playbook goes deep on PerplexityBot.
We refined this access work on a site we operate before bringing it to clients, so the order here is grounded in practice rather than vendor theory. The file is the footnote. The crawler is the story.
Frequently Asked Questions
What is llms.txt?
llms.txt is a proposed plain-text file placed in your site’s root directory that points AI crawlers to the pages and summaries you consider most important. It is meant to act as a curated map for large language models, similar in spirit to a sitemap. It is a proposal, not a standard the major engines have adopted.
Do AI engines actually read llms.txt yet?
Mostly not, as of 2026. No major AI provider, including OpenAI, Google, Anthropic, and Perplexity, has confirmed using llms.txt to source or cite content. A tracked experiment recorded zero visits to the file from GPTBot, ClaudeBot, PerplexityBot, or Google-Extended over a multi-month window. Treat any benefit as unproven.
Is llms.txt the same as robots.txt or sitemap.xml?
No. robots.txt tells crawlers what they may and may not access. sitemap.xml lists your URLs so search engines can discover them. llms.txt is a newer proposal meant to highlight your most important content for AI models. The first two are widely supported standards; llms.txt is not yet honored by major engines.
Should my site have an llms.txt file?
Adding one is low cost and low risk, so it is reasonable as cheap insurance if a future engine starts honoring it. But it will not get you cited today, so do not prioritize it. Spend your real effort on crawlability, clean HTML, and AI user-agent access, which actually decide whether engines can use your content.
What actually controls whether AI crawlers reach my content?
Your robots.txt rules for AI user-agents, your server response and access settings, and whether your important content renders in clean, crawlable HTML rather than only via heavy JavaScript. AI crawlers like GPTBot, OAI-SearchBot, PerplexityBot, and Google-Extended obey robots.txt, so your directives there, plus parseable pages, govern reach.
How do I control which AI crawlers can access my site?
Use robots.txt directives targeting each AI user-agent by name, because the engines run several distinct crawlers for different jobs. OpenAI uses GPTBot, OAI-SearchBot, and ChatGPT-User; Perplexity uses PerplexityBot and Perplexity-User. Allow or disallow each one deliberately, since blocking the training crawler does not block the live-search crawler that earns citations.
Continue Reading:
More On On-Page AEO:
- The On-Page AEO Playbook: Build Pages AI Quotes
- Entity SEO: How to Become a Known Thing to AI
- The FAQ Block, Rebuilt for Answer Engines
- How to Write a Definition AI Will Quote Verbatim
- Comparison Tables and Formats LLMs Love to Cite
More from TDM Insights:
- How AI Answer Engines Choose Their Sources
- How to Get Cited by Perplexity
- Answer Engine Optimization: What AEO Actually Is
- Technical SEO Checklist for Founders, by Severity
- Why Isn’t My Site Indexed? A Founder’s Diagnostic
Explore TDM Insights Categories: