What is llms.txt?
llms.txt is a proposed convention, published in 2024, for a Markdown file at the root of a website that gives large language models a short, curated description of the site and links to its most useful pages. It starts with an H1 name, a blockquote summary, optional details, and H2 sections of links. It does not replace your sitemap or robots.txt; it is a reading guide for AI tools.
Does llms.txt improve AI visibility?
Treat it as cheap hygiene, not a ranking lever. Support among AI products is uneven and no major assistant has committed to using it for ranking or citation. What reliably helps AI assistants describe you correctly is the same set of facts being consistent across your site, structured data, and third-party sources they already cite.
Which AI crawlers should I allow?
It depends on what you want. Many companies allow AI search crawlers (so they can be cited in ChatGPT search, Claude or Perplexity answers) while blocking crawlers used only for model training. Some tokens, such as Google-Extended and Applebot-Extended, are not crawlers at all: they only control whether content already fetched by Googlebot or Applebot may be used for AI models. Blocking them does not affect normal Google or Apple search.
Where do these files go?
https://yourdomain.com/llms.txt— plain text or Markdown, UTF-8.https://yourdomain.com/robots.txt— one file per host; subdomains need their own.