An llms.txt file is a plain text file, written in Markdown and placed in a website’s root directory, that lists a site’s most important pages for large language models (LLMs) to read. Jeremy Howard of Answer.AI proposed the format in 2024, pitching it as a curated shortcut: instead of an AI system crawling an entire site and guessing what matters, the site’s own owner points it straight at the pages worth reading.
The file itself stays deliberately simple. A short description of the site sits at the top, followed by a list of links grouped under plain Markdown headings, each with a line or two explaining what that page covers and why an AI system might want it.
How LLMs.txt Differs from Robots.txt and a Sitemap
Robots.txt tells crawlers where they’re not allowed to go. An XML sitemap lists every page a site wants indexed, exhaustively and without much editorial judgment. LLMs.txt does neither of those jobs. It curates rather than restricts or catalogues, offering a short, hand-picked reading list rather than either a gate or a full map.
That distinction matters because of how language models actually retrieve information at the moment a person asks them a question. Rather than crawling a live site in real time the way a search engine does, most systems answering a query pull from content gathered well before that conversation started, or from a search index built separately. An llms.txt file is a bet on a future where that changes, where AI systems fetch a site’s own curated summary on demand rather than relying only on what they indexed months earlier.
LLMs.txt and AI Search Visibility
Adoption sits somewhere between promising and unproven. Plenty of major sites, especially in the developer tools and documentation space, have already published one, and doing so costs nothing beyond the time it takes to write it. What’s missing is clear public confirmation that the large AI providers actually fetch and act on the file in a consistent, meaningful way, rather than it showing up in crawl logs occasionally without any evidence it shaped a single answer.
For now, publishing an llms.txt file sits closer to a low-cost bet than a proven tactic, worth doing for its own sake if a site already keeps a tidy content inventory, but not something to prioritise over the fundamentals still driving most AI-generated answers: clear, well-structured content, solid technical accessibility for crawlers, a genuine reputation for expertise, and the kind of real-world signals search engines already trust. Whether llms.txt becomes a real standard or fades into a well-intentioned footnote likely depends on choices AI companies haven’t made public yet, not on anything a site owner does today.