Skip to content

AI Agents

llms.txt, the file for AI crawlers

llms.txt is a Markdown file that points AI tools to your key pages. See the format, an example and what Google says.

Free agent readiness scan · no credit card

yourbrand.com /

robots.txtsitemap.xmlllms.txtindex.html

llms.txt

# YourBrand> CRM for small agencies## Key pages- [Pricing](/pricing)- [Docs](/docs)- [API](/api)
On this pageDefinition
  1. 01Definition
  2. 02Comparison
  3. 03Format
  4. 04Example
  5. 05How to
  6. 06Google
  7. 07Decision
  8. 08FAQ

llms.txt in one line

An llms.txt file is a Markdown file at yoursite.com/llms.txt that tells large language models what the site is and links to its most useful pages. It was proposed by Jeremy Howard in September 2024 and is still a proposal, not an official web standard. Google Search ignores it (Google Search Central), so it will not change your rankings or your AI Overviews.

01 · Definition

What is llms.txt?

llms.txt is a plain text file, written in Markdown, that sits at the root of a site next to robots.txt and sitemap.xml. It gives an AI agent a curated map: the site name, a short summary, and lists of links to the pages that explain the product, the docs or the policies. The llms.txt proposal starts from a simple problem: web pages are built for people, and their navigation, ads and scripts make the useful text hard for a model to pull out.

The file is meant for the moment a model answers a question, not for training. The proposal expected that llms.txt "would mainly be useful for inference rather than training". That is why the file is most common on developer documentation. The llms.txt site points out that OpenAI, Anthropic and Gemini publish llms.txt files for their own developer docs.

Publishing a file is not the same as reading one. The crawler pages from OpenAI, Anthropic and Perplexity describe robots.txt controls for their bots and say nothing about their bots reading llms.txt (checked September 28, 2026).

You will also see the name written as llm.txt, llms txt or llms-txt. The file the proposal describes lives at /llms.txt, with an s.

02 · Comparison

llms.txt vs robots.txt and sitemap.xml.

robots.txt tells crawlers what they may fetch, sitemap.xml lists the URLs you want indexed, and llms.txt picks a short set of pages and describes them for a model. Only robots.txt is a control. llms.txt is a suggestion that a tool may or may not read.

Job
robots.txtAllow or block crawling
sitemap.xmlList URLs for discovery
llms.txtSummarize the site and point to key pages
Format
robots.txtPlain text rules such as User-agent and Disallow
sitemap.xmlXML
llms.txtMarkdown: a title, a summary and link lists
Who reads it
robots.txtSearch and AI crawlers, including OpenAI's bots and Anthropic's bots
sitemap.xmlSearch engines
llms.txtTools and agents that choose to fetch it; OpenAI, Anthropic and Perplexity do not document reading it
Effect in Google Search
robots.txtControls what Googlebot crawls
sitemap.xmlHelps Google find URLs
llms.txtNone. Google Search ignores it

03 · Format

The llms.txt format.

The llms.txt format has one required part, an H1 with the site or project name. Everything after it is optional, but the specification sets the order below. The file can sit at the root or under a path such as /docs/llms.txt, where it covers the pages below that path.

H1 title
Required?Yes
What to writeThe name of the site or project, on a line that starts with #.
Blockquote summary
Required?No
What to writeOne or two sentences on a line that starts with >, holding the facts a model needs first.
Details
Required?No
What to writeShort paragraphs or lists with anything else worth knowing about the site.
H2 file lists
Required?No
What to writeSections that start with ##, such as Docs or Policies. Each item is a Markdown link to a page, followed by a colon and a short note.
Optional section
Required?No
What to writeA final ## Optional section for links an agent can skip when it needs a shorter context.
.md page copies
Required?No
What to writeThe proposal also suggests clean Markdown copies of pages at the same URL plus .md, so an agent can read them without the HTML (llmstxt.org).

04 · Example

An llms.txt example, line by line.

Here is a complete llms.txt file for a fictional outdoor store. Each line is shown in bold with what it does next to it. Swap example.com for your own domain.

  1. 01

    # Example Outdoor Co

    The H1. The only line the format requires.

  2. 02

    > Example Outdoor Co. sells tents, sleeping bags and camping stoves online

    The blockquote summary. Put the facts a model should know first here.

  3. 03

    Orders ship from the company warehouse. Prices on product pages include tax

    Optional details in plain sentences. Keep them factual and current.

  4. 04

    ## Products

    Starts the first file list.

  5. 05

    - Tents: sizes, weights and materials

    A link with a short note. Pointing to a clean .md copy follows the proposal's suggestion; a normal page URL also works.

  6. 06

    - Sleeping bags: temperature ratings and fill types

    One line per page that answers a real customer question.

  7. 07

    ## Policies

    A second file list for the pages agents check before recommending a store.

  8. 08

    - Returns: how returns and refunds work

    Policy pages are short and factual, which makes them good candidates.

  9. 09

    ## Optional

    Everything below this heading can be skipped when an agent needs a shorter context.

  10. 10

    - Company history: background on the brand

    Useful but secondary, so it goes last.

For a larger, real example, the llms.txt proposal walks through a file for the FastHTML project with Docs, Examples and Optional sections.

05 · How to

How to create an llms.txt file.

To create an llms.txt file, list the pages that matter, write them into the format above, and upload the file to your site root so it loads at yoursite.com/llms.txt.

  1. 1

    Pick the pages that answer real questions

    Choose the pages a model would need to describe you correctly: what you sell or build, docs, pricing, and policies such as shipping and returns. A short, curated list beats a copy of your sitemap.

  2. 2

    Write the file in Markdown

    Start with the H1 and a one or two sentence summary, then group links under H2 headings with a short note on each. Follow the order in the specification and put secondary links under ## Optional.

  3. 3

    Start from a draft if it helps

    AISEO's llms.txt generator takes one page URL and returns that page's main content as clean text you can copy or download as a .txt file. Edit the output into the title, summary and link-list format before you publish it.

  4. 4

    Upload it to the root

    Save the file as llms.txt in UTF-8 plain text and serve it at https://yoursite.com/llms.txt. Chrome's Lighthouse llms.txt audit flags the page if your server returns an error for the file; a missing file is marked not applicable.

  5. 5

    Keep it in sync

    Update the file when pages move, prices change or policies are rewritten. A stale summary or a broken link sends an agent to the wrong answer.

06 · Google

Does Google use llms.txt?

No. Google Search does not use llms.txt. Google's AI optimization guide, last updated July 10, 2026, lists llms.txt files under things you don't need to do, because Google Search itself doesn't use them. It adds that creating one will "neither harm nor help" your visibility or rankings.

That includes AI Overviews and AI Mode. The same guide says you don't need new machine-readable files, AI text files, markup or Markdown to appear in Google's generative AI features. Earlier, in June 2025, Google's John Mueller wrote that no AI system currently used llms.txt and that server logs show chatbots fetching pages but not the file (Search Engine Roundtable).

Chrome is a separate product. Its Lighthouse audit for llms.txt sits under agentic browsing and only fails when your server errors on the file. That is a check for AI agents working in a browser, not a Google Search ranking signal.

07 · Decision

Should you add an llms.txt file?

Add an llms.txt file if it is cheap for you and your site has docs, an API or policies that an agent may need. Do not add it expecting more rankings or AI citations: OpenAI, Anthropic and Perplexity do not document reading it, and Google says Search ignores it.

  • 01

    It costs little and does no harm

    It is one text file. Google says adding it will neither harm nor help your Google Search visibility (Google Search Central).

  • 02

    It helps the tools that do fetch it

    Chrome's Lighthouse documentation notes that without the file, agents may spend more time crawling a site to work out its structure. A short map of the right pages saves that step for the tools that read it, which is the use case the proposal was written for.

  • 03

    It does not replace the basics

    AI engines still read your actual pages. Crawlable pages, clear answers and accurate policies matter far more than a summary file. The generative engine optimization page covers that work.

  • 04

    It needs upkeep

    A file that lists old prices or retired pages gives agents wrong facts. If nobody will maintain it, a smaller file is better than a stale one.

08 · FAQ

llms.txt questions.

What is an llms.txt file?

An llms.txt file is a Markdown text file at the root of a website that gives AI models a short summary of the site and links to its key pages. It follows a 2024 proposal by Jeremy Howard and is optional.

Where do I put llms.txt?

Put it at the root of your domain so it loads at https://yoursite.com/llms.txt. The proposal also allows a file under a path, such as /docs/llms.txt, which covers the pages below that path.

Is llms.txt mandatory?

No. Nothing requires it. Chrome's Lighthouse audit marks a missing file as not applicable, and Google Search ignores the file whether it exists or not.

Is llms.txt actually used?

Its intended use is for agents and developer tools to fetch it when they work with a site, and AI labs publish one for their own docs. The crawler pages from OpenAI, Anthropic and Perplexity do not say their bots read it, and Google Search does not use it.

What is llms-full.txt?

llms-full.txt is not part of the llms.txt proposal. It is a convention from documentation platforms: Mintlify describes it as a single Markdown file holding all of a site's documentation text. Docs platforms can generate it for you; on other sites you can join your pages' Markdown into one file.

Does llms.txt help SEO?

Not in Google. Google says llms.txt will neither harm nor help visibility or rankings in Google Search, including AI Overviews and AI Mode (Google Search Central).

Is llm.txt the same as llms.txt?

People use both names for the same idea. The proposal uses /llms.txt, with an s, so use that file name.

Can I add llms.txt on WordPress or Shopify?

Yes, if your platform lets you serve a plain text file at the root of your domain. Check whether your theme, a plugin or an app can publish a file at /llms.txt, then open that URL in a browser to confirm it loads as plain text.

Related topics

Sources

  1. 01The /llms.txt file (proposal, v2) · llmstxt.org (Jeremy Howard), September 3, 2024; modified August 10, 2026
  2. 02Optimizing your website for generative AI features on Google Search · Google Search Central, July 10, 2026
  3. 03llms.txt audit (Lighthouse) · Chrome for Developers, May 5, 2026
  4. 04Google Says No AI System Currently Uses LLMs.txt · Search Engine Roundtable, June 18, 2025
  5. 05Simplifying docs for AI with /llms.txt · Mintlify, November 20, 2024
  6. 06Overview of OpenAI crawlers · OpenAI, Checked September 28, 2026
  7. 07Does Anthropic crawl data from the web, and how can site owners block the crawler? · Anthropic, April 7, 2026
  8. 08Perplexity crawlers · Perplexity, Checked September 28, 2026