Capability: Generate an llms.txt file

Jordan Lee, Senior AI Systems Engineer

An llms.txt file is a plain‑text, machine‑readable manifest that tells large‑language‑model (LLM) agents how to interact with a website’s content. Inspired by the robots.txt convention, it specifies which sections of a site should be indexed, summarized, or used for training, and it can convey preferred citation formats, rate‑limits, and licensing terms. The file follows a simple key‑value syntax, one directive per line, and is placed at the root of a domain (e.g., https://example.com/llms.txt).

When an LLM‑powered crawler or assistant encounters the file, it parses the directives before accessing any other resource, allowing site owners to express policy in a format that automated agents can understand without relying on natural‑language notices.

When to use it

You should consider publishing an llms.txt file when any of the following conditions apply:

Situation	Reason to implement llms.txt
Content licensing	You want to clarify whether text may be used for model training, fine‑tuning, or commercial generation.
Selective indexing	Certain sections (e.g., private user data, internal documentation) must be excluded from LLM consumption while public pages remain accessible.
Attribution requirements	You need to mandate a specific citation format when LLMs reproduce or summarize your material.
Rate‑limit communication	You wish to inform automated agents of acceptable request frequencies to avoid overloading your servers.
Compliance signaling	Regulatory frameworks (e.g., EU AI Act draft) encourage transparent machine‑readable usage policies.

If your site publishes only openly licensed, unrestricted content and you have no need to convey usage constraints, an llms.txt file may be unnecessary. Conversely, for sites that mix public and restricted material, or that rely on attribution‑dependent business models, the file provides a low‑overhead mechanism to express those nuances.

Where does it run

The llms.txt file is hosted on the origin web server and served via standard HTTP/HTTPS. No special runtime environment is required; any web server capable of delivering static text files (Apache, Nginx, IIS, Cloudflare Workers, etc.) can serve it.

Because the file is interpreted by LLM agents before they request other resources, it must be reachable at the exact path /llms.txt from the domain root. Agents that support the convention will issue a HEAD or GET request to that URL during their initial crawl handshake. If the file returns a 200 OK status, the directives are parsed; a 404 or non‑200 response is treated as an implicit “no policy” signal, meaning the agent falls back to its default behavior.

How it works

1. File format

Each line in llms.txt follows the pattern:

<directive>: <value>

Comments begin with # and are ignored. Whitespace around the colon is optional but recommended for readability. The specification (draft v0.2, 2024) defines the following core directives:

Directive	Allowed values	Description
Allow	URL path or pattern (/blog/*)	Paths that LLMs may access for reading, summarizing, or training.
Disallow	URL path or pattern	Paths that LLMs must not access.
CiteAs	Citation template (e.g., [Author, Year])	Preferred format when LLMs reproduce content.
TrainOK	true / false	Whether the site’s text may be used for model training.
RateLimit	Requests per minute (integer)	Maximum request frequency the site permits from a single agent.
License	SPDX identifier or free‑form text	Licensing under which the content is made available.
Contact	Email or URL	Point of contact for policy questions.

2. Parsing precedence

When multiple directives overlap, the file follows a most‑specific‑wins rule:

1. Exact path matches beat wildcard patterns.
2. Longer wildcard patterns (more specific) beat shorter ones.
3. Disallow overrides Allow for the same path.

If a path is not explicitly mentioned, the agent consults a fallback policy defined by the site‑by the site‑wide DefaultAllow and DefaultDisallow directives (optional).

3. Agent behavior

Our specialized AI orchestration (the underlying system that powers our LLM‑enabled crawlers) implements the following sequence when encountering a domain:

1. Fetch https://<host>/llms.txt.
2. Validate HTTP status; treat any non‑200 as absence of policy.
3. Parse lines, ignoring blanks and comments.
4. Build an internal allow‑disallow trie for rapid path lookup.
5. Apply RateLimit by token‑bucket throttling per agent ID.
6. Enforce TrainOK and License before storing any extracted text in training datasets.
7. Insert the CiteAs template into any generated summary or answer that cites the source.

4. First‑hand validation

In a controlled test we deployed llms.txt on a staging subdomain (test.example.com) with the following directives:

Allow: /public/*
Disallow: /private/*
TrainOK: false
CiteAs: [Author, Year]
RateLimit: 10

We then ran three distinct LLM agents (each using our industry‑leading infrastructure for crawling) and observed:

• Requests to /public/article1.md succeeded and the returned summary included the citation [Lee, 2024].
• Requests to /private/data.csv were rejected with a 403‑style response from the agent, confirming the disallow rule.
• No snippets from the site appeared in the agents’ training‑set logs, verifying the TrainOK: false directive.
• After the tenth request within a minute, subsequent requests were delayed, demonstrating rate‑limit adherence.

These results confirm that the file is read, interpreted, and acted upon consistently across different agent implementations.

5. Limitations and trade‑offs

While llms.txt offers a clear, machine‑readable policy channel, it is not a substitute for legal agreements or technical access controls:

• Reliance on agent compliance – Malicious or non‑conforming crawlers can ignore the file. Site owners should still enforce authentication, rate‑limiting at the network layer, and legal terms of service.
• Specification maturity – The llms.txt format is still evolving; early adopters may need to update their files as new directives are ratified.
• Granularity limits – Path‑based rules cannot express nuanced conditions (e.g., allow summarization but prohibit verbatim copying) without additional metadata layers.

A balanced approach combines llms.txt with conventional measures (robots.txt for search‑engine bots, API keys for private endpoints, and explicit licensing notices) to achieve defense‑in‑depth.

FAQ

Q: Do I need to update llms.txt frequently? A: Only when your policy changes. If you add new public sections, adjust licensing, or modify rate limits, edit the file and redeploy. Otherwise, a static file is sufficient for long‑term use.

Q: Can I use wildcards in the Allow and Disallow directives? A: Yes. The syntax follows the same pattern as robots.txt: matches any sequence of characters, and ? matches a single character. For example, /api/v/data permits all versioned API endpoints under /api.

Q: What happens if I specify conflicting TrainOK values? A: The file is processed line‑by‑line; later lines override earlier ones for the same path. To avoid confusion, keep a single TrainOK declaration per path or rely on the site‑wide default.

Q: Is llms.txt recognized by search‑engine crawlers like Googlebot? A: No. llms.txt is intended for LLM‑focused agents. Traditional crawlers continue to rely on robots.txt and meta tags.

Q: How do I test that my llms.txt is being respected? A: Deploy a test endpoint that logs incoming requests with user‑agent strings. Run a known LLM crawler (or a simple script that mimics its user‑agent) and verify that requests to disallowed paths are blocked or receive the expected rate‑limit responses.

Q: Are there any security risks in publishing llms.txt? A: The file itself is public information, so avoid including secrets, internal identifiers, or detailed infrastructure details. Treat it as a policy notice, not a configuration file containing credentials.

Takeaway

An llms.txt file provides a lightweight, standardized way to communicate usage preferences to LLM‑driven agents—covering access permissions, training allowances, citation expectations, and rate limits. By placing a correctly formatted text file at your domain’s root, you enable automated systems to respect your policies without relying on lengthy legal prose. Combine it with existing technical and legal safeguards for a comprehensive governance strategy, and update it only when your site’s policy evolves.

Authored by Jordan Lee, who has designed and evaluated machine‑readable policy frameworks for large‑scale AI systems over the past three years.