llms.txt: what it is and whether it actually does anything
llms.txt is a proposed markdown file that lists a site's main content for large language models. What it can and cannot do, who actually reads it, and what demonstrably does affect whether an assistant uses your page.
llms.txt is a proposed markdown file at a site’s root that lists the site’s main content for large language models. It was proposed by Jeremy Howard in 2024, and its own documentation calls it a proposal, which is the accurate word. It grants nothing, denies nothing, and as far as the providers’ own documentation shows, no major provider publicly commits to reading it. This post explains what it is, what it is not, and what actually does get your site read.
The honest summary first. llms.txt is cheap to publish and costs nothing to maintain, so publishing one is fine. It is a bet, not a channel. The work that demonstrably matters is different: being reachable by the retrieval crawlers, serving the content in the HTML, and writing pages that can be quoted on their own.
What llms.txt is
The proposal: a site places a markdown file at /llms.txt (or any path, covering the pages under that path). The file holds an H1 with the site’s name, a short blockquote summary, and lists of links to the content worth reading. The format is deliberately plain, because a language model reads it as text.
The idea is that an agent fetches the file, sees where the useful content is, and follows the links. The file stays small enough to fit in a context window. The detail lives behind the links and is fetched only when needed.
What llms.txt is not
It is not robots.txt. robots.txt is a permission file that tells crawlers what may and may not be fetched. llms.txt grants and denies nothing. It has no enforcement. A crawler can ignore it entirely, and nothing in the file can stop a crawler from visiting the page it names.
People conflate the two because both are plain text files at the site root. They answer different questions. robots.txt says who may enter. llms.txt says where the good content is, to whoever asks.
The honest part: no major provider publicly documents using it
As far as the operators’ own documentation shows, no major provider publicly commits to reading third-party llms.txt files. OpenAI, Anthropic and Google all publish llms.txt files for their own developer documentation, which shows the format is real enough to dogfood. But publishing a file for your own docs is not a commitment to fetch the file on everyone else’s site. The distinction is the whole point of the honest section: the labs use the format, and none of them documents treating other sites’ llms.txt files as an input to answers.
If this changes, it will show up in the same places: the provider’s documentation of how its crawler works. Until then, publishing an llms.txt is a bet that the format wins the tooling race, not a channel with measurable return.
What demonstrably does affect whether an assistant can use your page
The retrieval crawlers are the fact that carries the weight: when a user asks a question, the assistant fetches pages through its search crawler. OAI-SearchBot serves ChatGPT search, Claude-SearchBot serves Claude with web sources, and PerplexityBot serves Perplexity. If those crawlers cannot reach your page, no llms.txt can fix it. If they can, the assistant can cite you.
The content must exist in the HTML. A page that renders its body only after JavaScript runs presents an empty shell to a crawler that does not execute the script. The assistant needs the content in the response, not in a bundle.
Clean and stable URLs let a cited page stay the cited page. A URL that works today and 404s tomorrow breaks the citation. Real headings give the assistant a structure to quote. Answer-shaped content, a page that makes its point in the opening paragraph and then expands, is what gets lifted into an answer. FAQPage and Article structured data give the page’s question and answer an explicit form an assistant can read directly.
The verdict
llms.txt is a proposal, not a standard, and its own site says so. That is not a criticism: robots.txt and sitemap.xml both started as conventions and became things everyone relies on, and this one may too. We publish one at stillindexed.com/llms.txt, and we treat it as a low-cost bet, not a source of traffic. Publishing the file costs one deploy and risks nothing. Do that if you want. Put the real effort into item four: the retrieval crawlers, the HTML content, the stable URLs, the real headings, and the structured data.
Where these facts come from
The proposal, its author and its status are taken from
llmstxt.org, which describes itself as “a proposal to
standardise on using an /llms.txt file to provide information to help agents
use a website” and notes that the AI labs publish llms.txt files for their own
developer documentation. Nothing on that page, and nothing in the crawler
documentation published by
OpenAI or
Anthropic,
states that they read llms.txt files from other sites. If that changes, this
post is wrong and we will update it.
Questions
- What is llms.txt?
- It is a proposed markdown file placed at a site's root, for example stillindexed.com/llms.txt, that lists the site's main content for large language models. It was proposed by Jeremy Howard in 2024. The file is guidance: it tells an agent where the useful content is, and grants or denies nothing.
- Is llms.txt like robots.txt?
- No. robots.txt tells crawlers what they may and may not fetch. llms.txt tells an agent where the content is. It has no enforcement and no way to block anything. A site can publish both, and the two do not conflict, because they answer different questions.
- Do AI providers actually read llms.txt?
- No major provider publicly documents reading third-party llms.txt files. OpenAI, Anthropic and Google publish llms.txt files for their own developer documentation, which shows the format is real, but publishing your own file is not a commitment to fetch anyone else's. Treat llms.txt as a bet, not a channel.
- What does affect whether an assistant can use my page?
- The retrieval crawlers must be able to reach the page: OAI-SearchBot, Claude-SearchBot and PerplexityBot. The content must exist in the HTML, not only after JavaScript runs. URLs should be clean and stable. Headings should be real headings. And the page should be quotable on its own, because an answer quotes the page, not the surrounding navigation.
- Does llms.txt help with search rankings?
- No. Search engines index HTML pages, and llms.txt is neither a ranking factor nor a page that competes for a query. It exists for agents, not for the search index, and no search engine documents treating it as ranking input.
- Should I publish an llms.txt file?
- Publishing one costs almost nothing and blocks nothing, so doing it is harmless. The mistake is treating it as a source of traffic or adoption. It is a low-cost bet. The effort that actually matters goes into the retrieval, the HTML, and the structured data on the pages themselves.