The llms.txt file and the XML sitemap

The llms.txt file is an emerging, not universally adopted standard that offers a structured Markdown summary of your site for generative AI systems; the XML sitemap, on the other hand, remains the established and widely supported standard for telling search engines the full set of pages to crawl. The two play different roles and complement each other rather than compete.

What llms.txt is

Proposed in 2024, llms.txt is a Markdown text file placed at the site root (yourdomain.com/llms.txt), meant to give a generative AI a condensed, readable summary of the site's content: an overview, main sections, links to the most important pages or documents. The underlying idea is to offer a compact entry point that's easier for a language model to process than a full site crawl.

It isn't an official standard adopted or confirmed by the major AI providers — OpenAI, Anthropic, Google, and Perplexity haven't made any formal commitment to systematically use it. It's a community proposal, championed in particular by parts of the developer ecosystem, gaining traction without being any guarantee of adoption.

What it delivers today (and its limits)

In practice, publishing an llms.txt doesn't guarantee better citation in AI answers: no major provider has confirmed that its models or crawlers systematically consult this file when responding to a user query. What can reasonably be expected from it today:

• A signal of good practice and clear structuring, useful should tools come to rely on it more broadly in the future.
• A useful resource for cases where a developer or a specifically configured AI agent explicitly looks for this file (for example, certain coding assistants that use it to understand a project's documentation).
• A clarifying exercise in its own right: writing an llms.txt forces you to synthesize what actually matters on the site, which has value even independent of whether an AI actually reads it.

It in no way replaces the XML sitemap, robots.txt, or properly implemented Schema.org markup, which remain the best-established levers today.

How to generate and structure an llms.txt

The proposed format stays simple, in Markdown:

# Site name

> Short one- or two-sentence description of what the site does and its value.

## Documentation
- [Page name](https://yourdomain.com/page): one-line description

## Resources
- [Resource name](https://yourdomain.com/resource): one-line description

For a content site (like a blog or a resource center), favor a list of your most foundational guides or articles rather than an exhaustive list of every page — the goal is to synthesize, not duplicate the sitemap. No universal automatic generator exists today for every CMS: writing it manually, or semi-automatically by pulling your main titles and meta descriptions, remains the most reliable method.

A refresher on the XML sitemap and best practices

The XML sitemap, meanwhile, is a standard that's been supported by every major search engine for a long time. A few best practices worth checking:

• Cap at 50,000 URLs and 50MB uncompressed per file: beyond that, use a sitemap index referencing multiple sitemaps.
• Only include canonical, indexable, up-to-date URLs: a sitemap containing noindex pages, redirects, or 404s sends a negative quality signal.
• Update it automatically through the CMS rather than by hand, to avoid drift from what's actually published.
• Include a reliable last-modified date (<lastmod>), since some engines use it to prioritize re-crawling.

Where to place these files and how to keep them current

Both files follow the same location logic: at the domain root, directly reachable (yourdomain.com/llms.txt, yourdomain.com/sitemap.xml), with no authentication or redirect in the way. For the sitemap, automatic updates through the CMS are almost always possible and preferable. For llms.txt, which remains a more editorial document, a quarterly manual review is usually enough to reflect newly published foundational content — no need to regenerate it every time a new article goes live if the site's overall structure hasn't changed.

Two files, complementary rather than competing

The XML sitemap says "here are all my pages, go crawl them"; llms.txt, in its current intended use, says more "here's the essential to understand if you need to summarize my site quickly." Publishing both costs virtually nothing in implementation time and covers both approaches without having to bet on the future adoption of either.

With robots.txt, the sitemap, and llms.txt in place, the technical layer for controlling crawlers is complete. What's left is understanding more precisely who reads you and why, then moving on to the technical SEO basics to look at how content itself gets interpreted once it's reachable.