The short answer
- llms.txt is a proposed convention, published in 2024, for a markdown file at a website’s root that gives language models a short summary of the site and links to its key pages.
- Major AI providers have not committed to reading llms.txt when they crawl sites or answer questions, so any benefit is uncertain.
- robots.txt controls which crawlers may visit, a sitemap lists URLs for discovery, and llms.txt is a short, curated guide to the site written for a model to read.
llms.txt is a proposed convention for a plain markdown file, placed at /llms.txt on a website, that gives language models a concise summary of the site and links to its most useful pages. It was put forward in 2024 by Jeremy Howard of Answer.AI, and many documentation sites have since published one. None of the major AI providers has committed to using it in search or in answers, so adding one costs little and its benefit is uncertain.
Why llms.txt was proposed
Language models have a limited context window, which is the amount of text they can take in at once. Web pages are full of material that wastes it, such as navigation, scripts, cookie banners and adverts. The idea behind llms.txt is that a site owner offers a short, clean map of the site in markdown, a simple text format that models handle well, so a model or tool can find the important information quickly.
The proposal was aimed mainly at the moment a question is asked. A developer working with a coding assistant, for example, can point it at a library’s llms.txt to bring the right documentation pages into the conversation. That remains its clearest use today.
How an llms.txt file is structured
The specification at llmstxt.org sets out a simple order of sections. Only the first is required.
- A title. A level-one heading with the name of the site or project, written as
# Name. - A summary. A blockquote, written as a line starting with
>, giving a short description with the key facts a reader needs to understand the rest of the file. - Optional detail. Paragraphs or lists with more context, such as who the site is for or how to use the sections that follow. This part contains no headings.
- Sections of links. Each starts with a level-two heading, such as
## Servicesor## Guides, followed by a markdown list of links in the form- [Page name](url): one-line note. - An Optional section. A section headed
## Optionalholds secondary links that a tool can skip when it needs a shorter context.
For a small business, a file might open with # Example Roofing Ltd and a one-line summary of what the company does and where it works. Sections would then list the service pages, the areas covered, the most useful guides and the contact page, each with a short note on what the page contains. A line in the services section might read - [Flat roofs](https://example.com/flat-roofs): EPDM and GRP flat roofs, repairs and replacements.
The proposal also suggests offering a clean markdown version of individual pages at the same address with .md added. Some documentation platforms also publish an llms-full.txt file that contains the full text of the documentation in one place.
llms.txt, robots.txt and sitemaps compared
The three files are often mentioned together, but they do different jobs for different readers.
| robots.txt | sitemap.xml | llms.txt | |
|---|---|---|---|
| Purpose | Tells crawlers which paths they may fetch | Lists URLs for search engines to discover | Summarises the site and points to key pages |
| Format | Plain text rules | XML | Markdown |
| Status | Formal standard, RFC 9309 | Long-established protocol supported by the major search engines | Community proposal, not formally adopted |
| Who reads it | Crawlers, before fetching pages | Search engine crawlers | Tools and models, usually when someone points them to it |
| Coverage | Rules for the whole site | Every URL you want indexed | A curated selection |
| Controls access | Yes, for crawlers that follow it | No | No |
The last row matters most. llms.txt has no power to allow or block anything. If you want to control whether AI crawlers can fetch your pages for training or for search, that is done in robots.txt, as explained in AI crawlers and robots.txt.
Whether AI providers use llms.txt
As of late 2026, none of the major AI providers has publicly committed to reading llms.txt when its crawlers visit a site, or when its answer engine chooses what to cite. Google’s guidance on appearing in its AI search features doesn’t ask for one. You can check your own server logs to see whether anything requests the file, and which user agents do.
Some AI companies publish llms.txt files for their own developer documentation, which shows the format is useful for that purpose. It doesn’t mean their crawlers look for the file on other people’s sites. Coding assistants and other tools that can fetch web pages will read it when a user asks them to, which is the use the proposal had in mind.
This could change. If a major provider announces support, the file will matter more, and a site that already has an accurate one will have lost nothing by being early.
Deciding whether to add one
For most business websites it is a low-cost, low-certainty addition. Writing a good one takes an hour or two for a small site, and it does no harm. It won’t replace the things that make you easier for answer engines to find and quote, which are clear pages, content in server-rendered HTML, consistent facts about your business across the web, and passages that answer questions directly. Those are covered in how to become the source an AI answer cites.
For a site with only a handful of pages, the file adds little that a model couldn’t get from the pages themselves. The case grows with the size of the site and the amount of material a reader has to sift through.
It is a clearer yes for software products with developer documentation or a public API. Developers already paste documentation into AI tools, and a well-kept llms.txt makes that quicker and more accurate.
How to write an llms.txt that helps
- Write the summary as plain facts covering what you do, for whom, where, and what sets you apart.
- Link only to the pages that answer the questions customers ask, and leave the rest to your sitemap.
- Give each link a one-line note saying what the page contains.
- Use the same company name and description as on your site and your Companies House record.
- Keep it in step with the site. Generate it from your content when the site is built, if you can, so it never describes a page that has gone.
- Serve it as plain text at the root, and check that it loads at
yourdomain.com/llms.txt. - Make it a summary of the site, and never add claims that the site itself doesn’t make.
Review it whenever you add a service, retire a product or restructure the site, in the same way you would update the sitemap. An out-of-date llms.txt is unlikely to be read often, but when it is read, it should be right.
Questions people ask
Is llms.txt an official standard?
No. It is a community proposal published in 2024, with a specification at llmstxt.org. It hasn’t been adopted by a standards body, unlike robots.txt, which is defined in RFC 9309.
Do ChatGPT and Google read llms.txt?
Neither OpenAI nor Google has committed to using llms.txt for search or for choosing sources. A tool that can browse may read the file if you give it the address, in the same way it would read any page.
Will llms.txt improve my rankings or AI citations?
There is no evidence that it does today. It has no known effect on Google rankings, and no major answer engine has said it uses the file to choose sources. Treat it as a small, low-risk addition.
Can llms.txt stop AI companies using my content?
No. llms.txt contains no rules for access or training. To control AI crawlers, use robots.txt directives for each crawler’s user agent, and bear in mind that robots.txt relies on crawlers choosing to follow it.