What is the llms.txt file, and what is it used for?

If you work in digital marketing or SEO, you’ve probably seen someone recommend creating an llms.txtfile to “prepare your website for AI.” The idea is circulating in groups, newsletters, and even technical audit tools.

The problem is that much of this advice doesn’t hold up when compared to what search engines themselves say on the subject.

This article explains what the llms.txt file is, who proposed this idea, how it actually works, and, most importantly, whether it’s worth your team’s time.

What is the llms.txt file?

The llms.txt file is a text file in Markdown format, hosted in the root directory of a domain (such as yoursite.com/llms.txt), which provides a summary of the website and links to the pages considered most relevant.

The idea is to provide a language model with a condensed and “clean” version of the content, without the clutter of menus, ads, and scripts found on a typical HTML page.

This approach stems from a real-world problem. Language models operate within a limited context window—that is, a finite amount of text they can process at a time.

However, transforming an HTML page full of navigation elements, JavaScript, and visual elements into text that is useful for an AI is an imprecise process, as described in the original llms.txt specification.

In practice, the file follows a fixed structure: an H1 heading with the name of the website or project, a short quote summarizing what that domain offers, and then lists of links organized into sections, each pointing to Markdown documents with more detail. It is essentially an editorial summary, written to be read by a machine.

Before continuing, an important point: llms.txt is not a web standard recognized by organizations such as the W3C, noris it a requirement of any search engine. It is a market proposal, open and still under discussion.

Who created llms.txt, and what is the original purpose of this file?

The llms.txt file was proposed by Jeremy Howard, co-founder of Answer.AI and fast.ai, in September 2024.

The original goal was to help development tools and technical documentation deliver more direct content to LLMs at the time of inference—that is, when a user asks an AI for help with a code library, for example.

According to Howard’s own text on llmstxt.org, the idea arose from a practical limitation: software documentation sites are difficult to process in their entirety, and much of the content (menus, footers, banners) is irrelevant to a language model trying to answer a technical question.

Not surprisingly, the first documented adopters were mostly technology companies and SaaS products with extensive documentation, as noted in Search Engine Land’s coverage of the standard’s launch.

Howard’s intended use has always been closer to an “instruction manual for code agents” than to an SEO tactic for blogs and corporate websites.

This origin matters because it explains much of the current confusion: llms.txt was created to solve a technical documentation problem and was later generalized by the SEO as if it were a ranking tool or a citation tool for AI search engines. That was not its original purpose, and that is not what it delivers today.

How does the llms.txt file work in practice?

In practice, llms.txt functions as an index: the website publishes the file in the domain’s root directory; a language model (or a tool that queries that model) can retrieve this file and use the links it contains to quickly understand the content’s structure, without having to crawl the entire site.

The specification also suggests that relevant pages have a version in plain Markdown, available at the same URL as the original page with the .md extension at the end. This allows AI to read a version without visual formatting, containing only the text and the heading structure.

Unlike robots.txt, llms.txt neither blocks nor allows crawling. It is a reading suggestion, not an access rule.

And, unlike sitemap.xml, it does not list all the pages on the site, but rather a curated selection of what the file’s author considers most important for an AI to understand.

This point is key to understanding the format’s limitations. Since the file’s content is written and selected by the website itself (that is, it is a statement by the domain owner about what is relevant), there is no independent verification that this summary accurately reflects the page’s actual content.

It is a self-declared signal, similar in spirit to the old “keywords” meta tag that the search engine industry abandoned years ago because it was easily manipulable.

Caption: Conceptual representation of AI agents extracting simplified data from an llms.txt file.

Does Google use llms.txt to rank pages?

No. Google confirms, in its own official AI optimization documentation, that it is not necessary to create machine-readable files, AI text files, specific markup, or additional Markdown to appear in Search, including generative AI features.

Google’s own documentation goes further and treats the issue as neutral, not as prohibited: it is permissible to maintain an llms.txt file for other services that use it, which neither helps nor hinders visibility on Google, because Search ignores this file.

This stance is not merely the isolated opinion expressed in a third-party article. Gary Illyes, from Google’s Search Relations team, has publicly stated that the company does not support llms.txt and has no plans to start using it, according to a report by Search Engine Roundtable.

John Mueller, also from Google’s Search Relations team, was even more direct in a recent response reported by Search Engine Journal.

He compared llms.txt to the old keywords meta tag—a self-declared signal from the website itself about what it thinks is relevant—and noted that, even after years of the file’s existence, no relevant AI system actually uses it. According to Mueller, the benefit of the file remains “purely speculative.”

In other words, if an llms.txt file appears on one of Google’s own properties, that does not constitute an endorsement of the format.

Mueller has already clarified that this happened because an internal content system began adding the file automatically, and some teams simply failed to remove it—which does not represent any strategic position on the part of the company.

Do other AI systems and search engines crawl llms.txt?

Not consistently or in significant volume. An analysis by SE Ranking of approximately 300,000 domains, published by Search Engine Journal, found that only 10.13% of the analyzed websites used llms.txt, and no statistical correlation between having the file and being cited more frequently in LLM responses.

The most revealing finding of this study is technical: when the “has llms.txt” variable was removed from the statistical model used to predict AI citations, the model’s accuracy improved. In other words, the presence of the file was introducing noise into the data, not a signal.

The same material points out that OpenAI’s crawler documentation focuses on robots.txt rules for the OAI-SearchBot, without mentioning llms.txt as a citation or ranking factor.

Some server logs show that GPTBot occasionally retrieves the file, but this does not occur with any significant frequency nor does it appear to influence which domains are cited.

Here’s a summary of the current situation among the main public references:

Table: Consolidated overview based on public statements and the most recent available adoption study on the topic.

This table helps explain why the recommendation to create an llms.txt file "to ensure visibility in AI" tends to be, at best, an exaggeration, and at worst, an empty promise.

Is there any real benefit to creating an llms.txt file?

Yes, but within a much narrower scope than the SEO market usually leads people to believe.

The llms.txt file can be helpful in specific contexts involving technical documentation and software products—which is what it was originally designed for—rather than as an optimization tactic for search engine rankings.

For documentation platforms (code libraries, APIs, SaaS products with many technical pages), the format offers a cost-effective way to provide condensed context to AI-assisted development tools, such as code editors with built-in AI.

In this scenario, the file’s consumer is an internal tool used by the developer, not a public search engine.

There is also no evidence that maintaining the file harms the website. As Google itself describes, the file is ignored by Search, so it neither helps nor harms rankings.

This means that the decision to create an llms.txt file or not is, technically, low-risk, but also offers a low expected return for most content websites, blogs, and institutional pages.

The most common mistake is to treat this documentation infrastructure as if it were part of an SEO or GEO (Generative Engine Optimization). These are different objectives, with distinct audiences and tools.

When is it worth implementing llms.txt on your website?

It’s worth doing on a case-by-case basis when the website is a technical documentation platform, a SaaS product with a public API, or a developer tool whose users rely on AI assistants to look up documentation directly. Outside of this context, the effort outweighs the benefit.

A practical guideline: if your support team already receives questions from developers about how to integrate an AI assistant with your documentation, or if a specific third-party tool explicitly requires an llms.txt file to work better, then it makes sense to invest a few hours in this.

If your website is an institutional blog, a course page, a lead generation landing page, or any content aimed at human readers who arrive via organic search, the llms.txt file won’t solve the problem you’re likely trying to address, which is appearing more frequently in search engine results and AI assistant responses.

That’s achieved through a different approach, as we’ll see below.

What should you do instead of using llms.txt to gain visibility in AI?

The short answer is: the same well-executed SEO work, with structural adjustments for AI. Optimizing to be cited by generative AI tools doesn’t depend on a special file, but rather on original, extractable, and reliable content built on a sound technical foundation—just like any content that ranks well in traditional search.

Some approaches with real backing—unlike llms.txt—include understanding how crawling by LLMs and ensuring that major AI crawlers (such as retrieval bots used by conversational search tools) aren’t blocked in robots.txt, since blocking these agents removes your site from the responses.

It’s also important to understand how AI platforms themselves break down a question into multiple sub-questions before generating a response—a process known as query fan-out: covering a topic broadly—rather than just the main question—increases the chance of appearing in one of these subquestions.

From a practical perspective regarding writing and structure, the factors that most increase the likelihood of being cited in generative AI responses are citing reliable, named sources, including quotes from experts, and using verifiable statistics with sources—techniques that any content team can apply without relying on any special files.

This is the kind of tactic detailed in how to do SEO and get cited by AIs.

Finally, if your website already receives visits from autonomous AI agents (assistants that browse and perform tasks on behalf of the user), the technical priority is to ensure that these agents aren’t blocked and that the interface is usable by them—a topic explored in greater depth in agent-based SEO.

This tends to be more valuable, in terms of results, than any self-declared text file.

Frequently Asked Questions About the llms.txt File

So, is it worth investing time in the llms.txt file?

For most content websites, corporate blogs, and fundraising pages, no. The llms.txt file was created to solve a specific technical documentation problem, and major search engines and AI platforms—including Google—have already publicly confirmed that they do not use it to rank or cite content.

Investing hours of development time in this file, hoping to gain visibility, tends to be a waste of time.

The real effort in AI optimization lies elsewhere: a clear content structure, direct answers at the beginning of each section, comprehensive coverage of the topic, cited reliable sources, and a solid technical SEO foundation without crawling bottlenecks.

It is this work—not a self-declared text file—that actually appears in AI citation analyses.

If your team is deciding where to invest in the next optimization cycle, this is exactly the kind of prioritization that warrants a technical discussion before any implementation.

A mkt4edu works with SEO based precisely on this logic: identifying what truly drives rankings and citations in AI for educational institutions, rather than chasing unsubstantiated trends.

Contact the team via mkt4edu’s contact page and find out where your team’s technical efforts will generate real returns.