llms.txt: The robots.txt of the AI Era
Thirty years after robots.txt, a new file emerges with the same logic: llms.txt. What it does, how to write it, and why its coherence with robots.txt is decisive.
In 1994, site owners started placing a robots.txt file at the root of their domains. It told web crawlers what to index and what to ignore. Nobody forced them. It became a universal standard because it solved a real problem: giving bots the instructions they needed to do their job properly.
Thirty years later, a new file is emerging with the same logic: llms.txt.
Its purpose? Give language models the essential information about your site, structured in a way they can process efficiently — without having to piece it together themselves.
What llms.txt Does
The llms.txt file lives at the root of a domain, just like robots.txt. It contains a structured description of the site — in markdown, readable by LLMs — with key information about the entity, its products, its values, its trusted sources, and its strategic pages.
It can include things like:
- What the company does, in one precise sentence
- Main products or services and their characteristics
- Target audience
- The most important pages to know about
- Authors or experts associated with the site
- Links to third-party sources that mention the site (press, studies, comparisons)
The principle: rather than letting an LLM reconstruct an incomplete or distorted image of your brand from scattered fragments, you give it a structured, verifiable, coherent information base to work from.
Why It Matters for GEO
LLMs have two ways of knowing a brand.
The first is training. Everything ingested before the model's knowledge cutoff. That's where well-known brands live — established facts, information repeated across many sources. For recent, niche, or poorly represented brands, this layer is partial or inaccurate.
The second is real-time search. When the model has access to search tools, it goes looking for current information. And what it finds first are the best-structured, most easily parseable sources.
llms.txt plays on both levels. It improves the quality of information available to the model in real time, and it structures that information in a way that maximizes the likelihood the model uses it in its responses.
What a Poorly Written llms.txt Does (or Doesn't Do)
An empty or generic llms.txt is a missed opportunity. An auto-generated llms.txt without calibration is sometimes worse — it can contain vague descriptions that reinforce an inaccurate image.
The common mistakes:
Description too vague. "Innovative company in the digital sector" tells an LLM nothing. "GEO platform that tracks brand citations in ChatGPT, Gemini, and Perplexity and recommends content actions based on fan-out queries" says a great deal.
No named entities. LLMs operate on entities — names, products, technologies, locations, people. An llms.txt without clear entities is barely usable.
Missing third-party sources. Pointing to press articles, comparisons, and studies mentioning your brand reinforces the credibility of the information in the model's eyes.
Misalignment with the rest of the site's content. If the llms.txt says one thing and the site's pages say another, the model arbitrates based on the strongest signals — not necessarily in your favor.
Contradictory robots.txt. Blocking AI crawlers in robots.txt while maintaining a llms.txt is incoherent. Both files need to be aligned.
llms.txt and robots.txt: Coherence as a Condition
The llms.txt doesn't work in isolation. It connects with robots.txt, with Schema.org structured data, with the overall quality of the site's content.
The coherence between robots.txt and llms.txt is a point often neglected. Some sites block AI crawlers in robots.txt by instinct — data protection reflex — without realizing it completely neutralizes the llms.txt. Others allow crawling but have an llms.txt that contradicts their product pages.
Vurto includes a dedicated module for auditing and rewriting both files together — robots.txt and llms.txt — to ensure the instructions given to models are coherent, complete, and aligned with the overall GEO strategy. It's one of the starting points of any serious GEO audit.
How to Write an Effective llms.txt
The recommended structure is simple:
# [Company Name]
[Precise description in 2-3 sentences: what you do, for whom, with what difference]
## Products / Services
- [Product 1]: [functional description in one sentence]
- [Product 2]: [functional description in one sentence]
## Target Audience
[Description of the primary audience: sector, size, role, problem solved]
## Key Pages
- [URL page 1]: [what's there]
- [URL page 2]: [what's there]
## Reference Sources
- [Press article link]
- [Comparison link]
- [Study citing the brand]
This isn't a fixed format — the standard is evolving fast. But the underlying logic is stable: precise information, named entities, links to third-party sources, coherence with the rest of the site.
Where the Market Stands in 2026
llms.txt adoption is growing. Among the most GEO-advanced sites, it's become a baseline reflex, just like robots.txt or a sitemap.
For brands that haven't done it yet, it's one of the GEO actions with the best effort-to-impact ratio. A few hours of work, a well-written file, a robots.txt coherence check — and you give LLMs a reliable information base about your brand that they didn't have before.
In a world where generative engines build their responses from whatever signals are available, controlling those signals at the source is basic digital hygiene.
Vurto audits and rewrites llms.txt and robots.txt files to ensure coherence and maximize your site's readability for LLMs.