What Is llms.txt and How to Create One
AIWhat llms.txt is, how to create and publish one, what the August 2026 v2 spec changed, and what the data says about whether AI models actually read it.
Eugenijus Denisov
Key Takeaways
-
The file guides AI crawlers to your core content and ignores administrative pages.
-
The August 2026 version 2 update added link relations and HTTP headers so agents can locate the file and Markdown page versions instantly.
-
Checking server logs shows which bots request the file, but it cannot measure search impact since Google confirmed in 2026 that it ignores the format for AI search and AI Overviews.
The llms.txt file directs large language models to the most valuable information on your site. The proposal received a major update in August 2026, meaning most older tutorials are already out of date. We will explain how to build this file and examine the evidence on whether platforms actually read it.
What Is an llms.txt File?
An llms.txt file is a plain text file formatted as a Markdown document placed on your website to give AI agents a curated map of your best content. Rather than providing full page details for large language models, the file pairs URLs with short, one-line summaries or just page titles.
Jeremy Howard of Answer.AI proposed the format on 3 September 2024, and it received a version 2 update on 10 August 2026. This file provides no access control for AI systems, offers no SEO ranking signal, and serves a completely different purpose than an XML sitemap.
What Changed in the llms.txt v2 Spec
The August 2026 update changed how AI systems discover the file and handle file extensions. Originally, publishers had to append .md to the end of a full URL. Now, you can simply replace the .html extension with .md to point bots to a Markdown file version of a page.
The updated rules also define subpaths to help AI assistants navigate. A file now covers only the pages under its specific path, so a site owner who controls just a single section of a documentation platform can participate.
Furthermore, the revision clarifies that search bots should actively search the file for what they need instead of treating it as a static context window.
The biggest change is discoverability. The community introduced standard link relations so AI tools can find the file without guessing. You can add these to your page header :
<link rel="describedby" type="text/markdown" href="/llms.txt">
Alternatively, you can provide the link relations as an HTTP response header, which works perfectly for non-HTML files:
Link: </llms.txt>; rel="describedby", </page.md>; rel="alternate"; type="text/markdown"
You can configure this header directly in your web server or CDN to avoid editing individual HTML pages.
llms.txt vs. robots.txt vs. sitemap.xml
| Feature | robots.txt | sitemap.xml | llms.txt |
|---|---|---|---|
| Purpose | Access control for crawlers | Broad URL catalog with metadata | Curated reading list for agents |
| Format | Plain-text directives | XML | Markdown |
| Audience | Search engines and most crawlers | Search engines | Large language models and AI agents |
| Scope | What bots may crawl | All or most URLs | Most important pages only |
| Content guidance | None | Minimal, lastmod and priority | Yes, descriptions and context |
An llms.txt file does not override robots.txt. AI crawlers must still respect your existing site permissions, even if you explicitly include a blocked URL in your curated reading list.
llms.txt Format and Example
The specification defines a strict order for the file structure:
- H1 heading: The project or site name, which is the only mandatory element.
- Blockquote, optional: A short summary of the project.
- Prose, optional: Additional details without any headings.
- H2 sections, optional: Markdown lists containing links.
Each list item must take the form of a name link. You can optionally add a colon and a short note after the link.
# Acme Developer Platform
> Acme provides APIs for payment processing and subscription management.
These developer docs explain how to integrate our checkout flow and manage customer data.
## API reference
- [Payments API](/docs/payments.md): Endpoints for processing credit cards.
- [Subscriptions API](/docs/subscriptions.md): Endpoints for recurring billing.
## Optional
- [Changelog](/changelog.md): Recent updates to the platform.
- [Support](/support.md): Contact information for our engineering team.
In this example, the H1 heading is the only required element. The blockquote, prose, and H2 sections are officially defined parts of the specification rather than informal conventions. The ## Optional heading, however, is just a convention. The version 2 specification gives it no mechanical meaning, so you should not expect agents to automatically skip it.
How to Create an llms.txt File
- Choose your most important pages, especially if you run a documentation site. Include developer docs, API references, and key product pages. Exclude login-gated content, duplicated pages, and temporary promotions.
- Group your URLs under descriptive H2 sections to provide context. Write a one-line description for every link that explains exactly what the page contains. Avoid vague phrases like “learn more”. Link to Markdown file equivalents wherever they exist.
- Place the file in your root directory for site-wide coverage, or in a subpath like /docs/llms.txt for a specific section. Add the
<link rel=describedby>HTML tag or Link: HTTP header so bots can find it. - Verify that the URL returns an HTTP 200 status code with a text/plain or text/markdown content type. Check that your firewall allows bot access.
- Test your file by feeding it to AI systems. Ask user queries about your content to evaluate whether the bot answers accurately based solely on the structure and links you provided.
Does Google Use llms.txt?
Google incorporates the file into its Lighthouse Agentic Browsing category, which evaluates how well a site serves automated agents. Instead of a traditional 0-to-100 score, the audit issues a fractional pass ratio. If your site lacks an llms.txt file entirely, Lighthouse marks the check as “not applicable”. You only receive a failing mark if the file exists but returns an error or breaks formatting rules.
The category also checks whether your accessibility tree is well-formed and your layout remains stable (CLS). These metrics directly impact your score because AI agents rely on a stable layout and a clean accessibility tree to understand and click on page elements. Finally, Lighthouse looks for WebMCP and ai-catalog.json implementations, but these checks are currently just informational and will not lower your score.
How to Check if AI Crawlers Read Your llms.txt
You can track reads by checking your server or CDN logs for direct requests to /llms.txt and /llms-full.txt. However, do not filter your logs for major web crawlers like GPTBot or ClaudeBot. Major AI providers do not proactively crawl this format. Furthermore, Google-Extended will never appear in your logs because it is simply a robots.txt control token, not an actual crawler .
Instead, log analysis will show that requests primarily come from individual users running local AI tools, CLI tools, or Python scripts. Because these user-driven agents rely on your file to navigate on the fly, you must keep your links current. Stale URLs will send agents to dead pages and break their workflow, so always update your llms.txt file after any site restructuring.
FAQ
Does llms.txt improve SEO rankings?
No. The file does not improve traditional search engine rankings. Google and Bing use standard algorithms to rank pages for both traditional and AI search.
What is the difference between llms.txt and llms-full.txt?
The standard file provides a curated list of links and summaries to guide large language models across your site. The llms-full.txt file concatenates all your essential web content into one massive document, giving an agent everything it needs in a single request.
Can I block AI crawlers with llms.txt?
No. The file provides no mechanics to block bots from your site. You must configure your robots.txt file and set up firewall rules to prevent automated scraping .
Where should llms.txt go on my site?
Place the file in the root directory of your domain to cover your entire site. You can also place it in a subpath to restrict its scope strictly to the documents under that directory.
Does Google use llms.txt?
Google includes a check for the file in its Lighthouse Agentic Browsing audit. However, Google explicitly confirmed in 2026 that it ignores the format for Search and AI Overviews.
How do I know if anything is reading my file?
You can check your server access logs for direct requests to the file. Expect to see requests from individual users running local AI tools, Python scripts, or CLI tools rather than major web crawlers.
