← All articles

Do You Need an llms.txt File? What Google, OpenAI and Perplexity Actually Say

By Rei · CiteMonthly · Published 16 October 2026

Short answer: For Google Search — including AI Overviews and AI Mode — you do not need an llms.txt file. Google’s own generative AI optimization guide says Google Search “doesn’t use” these files, and that creating one will “neither harm nor help your site’s visibility or rankings in Google Search, as Google Search ignores them” [1]. OpenAI’s and Perplexity’s crawler docs tell site owners how to manage access with robots.txt (and published IP ranges). In the pages we opened on 16 Oct 2026, neither documents reading a third-party site’s /llms.txt when deciding what to show in ChatGPT search or Perplexity results [4] [6]. An llms.txt is a community proposal for giving AI agents a short, curated map of a site — useful for documentation and coding agents, optional for a typical small business website.

What is an llms.txt file?

It is a markdown file, usually at https://yoursite.com/llms.txt, proposed as a conventional place for LLM-friendly site information. The proposal lives at llmstxt.org and was published by Jeremy Howard (Answer.AI) on 3 September 2024; a v2 revision is dated 10 August 2026 [3].

According to the proposal, a conforming file:

  • starts with an H1 naming the project or site (the only required section);
  • may include a blockquote summary and short background;
  • then lists curated links under H2 headings, each as [name](url) with optional notes [3].

The same proposal suggests offering clean markdown versions of important pages (for example by appending .md or replacing the extension with .md), and advertising them with standard link relations such as rel="alternate" type="text/markdown" [3].

It is not a replacement for robots.txt or sitemap.xml. The proposal itself says robots.txt is about what automated access is acceptable, while llms.txt is meant to be used on demand when an agent needs a guided path into the site; a sitemap lists indexable URLs and is often too large to fit in a model’s context [3].

What does Google say about llms.txt?

Google addresses this directly in its mythbusting section for generative AI search.

In Google’s Guide to Optimizing for Generative AI Features on Google Search, under “LLMS.txt files and other ‘special’ markup,” Google says you do not need to create “new machine readable files, AI text files, markup, or Markdown” to appear in Google Search, including its generative AI capabilities, “as Google Search itself doesn’t use them.” It adds that creating and maintaining LLMS.txt files for other services is fine, and that doing so “will neither harm nor help your site’s visibility or rankings in Google Search, as Google Search ignores them” [1].

The same guide’s takeaways list “creating unnecessary AI text files (like llms.txt)” among tactics you can ignore for Google Search [1].

Google’s separate page on AI features and your website makes the same practical point without naming the filename: “You don’t need to create new machine readable files, AI text files, or markup to appear in these features,” and there is “no special schema.org structured data that you need to add” [2]. Eligibility for supporting links in AI Overviews or AI Mode is the same as for Search with a snippet: the page must be indexed and snippet-eligible [2].

Bottom line for Google: spend your time on crawlable, useful pages — not on an llms.txt for ranking in Google’s AI features.

Do OpenAI’s crawler docs mention llms.txt?

OpenAI’s “Overview of OpenAI Crawlers” page is about robots.txt tags and user agents, not about requiring an llms.txt on your site [4].

What OpenAI documents for site owners:

  • OAI-SearchBot — used to surface sites in ChatGPT’s search features. Sites opted out “will not be shown in ChatGPT search answers, though can still appear as navigational links.” OpenAI recommends allowing it in robots.txt and allowing its published IP ranges [4].
  • GPTBot — may crawl content used in training generative foundation models; disallowing it is the documented way to opt out of that use. Settings are independent of OAI-SearchBot [4].
  • ChatGPT-User — user-triggered fetches; “robots.txt rules may not apply”; “not used to determine whether content may appear in Search” [4].
  • Changes for search can take about 24 hours after a robots.txt update [4].

The crawler page does mention llms.txt — but as OpenAI’s own documentation index, not as a file you must publish for ranking: “For the complete documentation index, see llms.txt. Markdown versions of documentation pages are available by appending .md to the page URL” [4]. We opened developers.openai.com/llms.txt on 16 Oct 2026; it is a curated markdown index of OpenAI’s developer docs [5].

What we did not find: any statement on that crawler page that ChatGPT search reads your website’s /llms.txt when choosing citations.

Do Perplexity’s crawler docs mention llms.txt?

Same pattern. Perplexity’s crawlers page tells webmasters to manage access with robots.txt user-agent tags, notes that changes may take up to 24 hours, and recommends allowing PerplexityBot (plus published IPs) if you want to appear in Perplexity search results. PerplexityBot “is not used to crawl content for AI foundation models.” Perplexity-User handles user-triggered visits and “generally ignores robots.txt rules” [6].

At the top of that same docs page, Perplexity points AI agents to its own documentation index: “Fetch the complete documentation index at: https://docs.perplexity.ai/llms.txt” [6]. We opened that file; it is a large curated list of Perplexity API and docs pages in markdown [7].

What we did not find: any instruction that PerplexityBot or Perplexity search requires or ranks third-party sites by whether they publish /llms.txt.

So who is llms.txt actually for?

The proposal’s own framing is clearer once you separate two different uses:

  1. Labs publishing an index of their documentation so coding agents can find the right page quickly. OpenAI and Perplexity both publish a docs llms.txt (we opened them on 16 Oct 2026) [5] [7]. The proposal page also notes that other AI labs publish ones for their own developer docs [3].
  2. You publishing a map of your own site so an agent pointed at your domain can find key pages without scraping navigation chrome.

The proposal says agents are expected to “view or search llms.txt to find the information they need, then follow the relevant links,” and that the format has been used most heavily for software documentation [3]. That is a different problem from “will ChatGPT name my bakery nearby?”

Our inference (not a vendor quote): for search visibility, the crawler docs that matter talk about robots.txt and IP allowlists, not about this file [4] [6]. If you run public API or product docs that coding agents hit often, an llms.txt matches what the proposal was designed for [3].

When might a small business still add one?

Google says maintaining LLMS.txt for other services is fine and will not hurt Google rankings [1]. So the decision is about effort, not about Google penalty risk.

Consider adding a simple /llms.txt if you already have clear service and location pages and want one short file that links them for agents, or if your docs platform generates one automatically (the proposal lists integrations; we did not audit each product) [3].

Skip or postpone it if your time is better spent allowing the right crawlers in robots.txt, fixing CDN bot blocks, keeping your Google Business Profile accurate, and writing pages that answer real customer questions. Those are the controls the vendor pages we opened actually document [1] [2] [4] [6].

What should you do instead for AI search visibility?

A practical order of work, based only on what the cited pages say:

  1. Stay crawlable for search bots. Allow OAI-SearchBot and PerplexityBot (and their IPs) if you want to appear in those products’ search answers; use Googlebot/robots.txt controls for Google Search [4] [6] [2]. We covered the checklist in Is your website blocking ChatGPT?.
  2. Meet Google’s normal eligibility bar for AI features: indexed, snippet-eligible content; no special AI markup required [2].
  3. Write unique, useful pages rather than chasing “AEO/GEO hacks.” Google groups unnecessary llms.txt files with other tactics you can ignore for Google Search [1].
  4. Keep local details accurate where you use Google Business Profile or Merchant Center; Google notes those products can help local and product information show in AI responses as well as other Search results [1].
  5. Optional: add /llms.txt later if you have documentation-style content or want a curated agent map — knowing Google will ignore it for Search ranking [1] [3].

Frequently asked questions

Will adding llms.txt help me rank in Google AI Overviews?

Not according to Google. Google Search ignores these files for visibility and rankings [1]. Appearing in AI Overviews or AI Mode has no additional requirement beyond normal Search eligibility with a snippet [2].

Does ChatGPT require llms.txt to cite my site?

OpenAI’s crawler documentation does not say so. It documents robots.txt controls for OAI-SearchBot and related agents, and points to OpenAI’s own docs llms.txt as a documentation index [4] [5].

Does Perplexity require llms.txt?

Not in the crawler docs we opened. Perplexity documents PerplexityBot / robots.txt / IPs for search visibility, and publishes its own docs llms.txt as an index for agents reading Perplexity’s documentation [6] [7].

Is llms.txt an official web standard?

It is a community proposal hosted at llmstxt.org, not something Google’s Search documentation requires [3] [1]. Treat it like a useful convention some sites and agents follow — not like a mandatory Search file.

Can llms.txt hurt my SEO?

Google says creating one will neither harm nor help Google Search rankings because Search ignores it [1]. Do not put confidential information in a public /llms.txt.

Want to know what ChatGPT and Perplexity say about you?

CiteMonthly runs a free "Does ChatGPT mention you?" check. We ask ChatGPT and Perplexity the questions your customers ask, then email you their answers: who gets named, which sources they cite, and whether you're one of them. No card, no account. Email citemonthly@gmail.com to request your free check.

Sources

  1. Google Search Central, "Google's Guide to Optimizing for Generative AI Features on Google Search". https://developers.google.com/search/docs/fundamentals/ai-optimization-guide (opened 16 Oct 2026)
  2. Google Search Central, "AI features and your website". https://developers.google.com/search/docs/appearance/ai-features (opened 16 Oct 2026)
  3. llms-txt / Answer.AI, "The /llms.txt file, v2". https://llmstxt.org/ (opened 16 Oct 2026)
  4. OpenAI, "Overview of OpenAI Crawlers". https://developers.openai.com/api/docs/bots (opened 16 Oct 2026)
  5. OpenAI Developers, documentation index (llms.txt). https://developers.openai.com/llms.txt (opened 16 Oct 2026)
  6. Perplexity docs, "Perplexity Crawlers". https://docs.perplexity.ai/docs/resources/perplexity-crawlers (opened 16 Oct 2026)
  7. Perplexity docs, documentation index (llms.txt). https://docs.perplexity.ai/llms.txt (opened 16 Oct 2026)