Awesome GEO

Does llms.txt help AI search? How to make one

By · Updated

The note on the Awesome GEO llms.txt tool says llms.txt will not get you cited, because no assistant has said it reads the file for citations. It adds that llms.txt is useful for docs sites and coding agents. If you run a docs site, the steps below build the file and check who requests it.

This page is written by the maker of Awesome GEO, which is built and maintained by Dmitry (Dee) Kargaev.

Google says its search ignores the file. Its guide to generative AI features, as Searchable quotes it, says keeping an llms.txt file “will neither harm nor help your site’s visibility or rankings in Google Search, as Google Search ignores them.” Searchable also found no public statement from OpenAI, Anthropic, Perplexity or Microsoft that their AI search products read llms.txt when choosing sources.

Studies have measured it. Beamtrace reports that SE Ranking, studying 300,000 domains, found no relationship between having the file and how often AI cites a domain. It also reports that Ahrefs found 97% of the valid files in its sample of 137,210 domains received zero requests in a month. Beamtrace quotes John Mueller of Google comparing the file to the keywords meta tag and saying none of the AI services have said they use it.

OtterlyAI put the file on an experiment website and watched AI bot traffic for 90 days: 0.1% of AI crawler requests touched /llms.txt. Superlines says accounts that had the file for over eight months showed no measurable uplift in AI search visibility. Its view is that implementing the file does not hurt but is not something to prioritize heavily.

The case for the file is about agents that fetch pages on demand. Documentation.AI, which hosts docs sites, writes that the file helps agents that fetch pages while someone waits for an answer. On its hosted sites, those agents read one page for every 3.4 that people read. Its figures count HTML page views, not requests for llms.txt itself. Searchable adds that documentation sites such as Anthropic’s and Stripe’s publish one so coding assistants can load clean docs on request. Omnia notes that Chrome’s Lighthouse 13.3 added an experimental Agentic Browsing category, and that Chrome’s documentation warns that agents without the file may spend more time crawling a site to understand its structure.

Mostafa ElBermawy gives the practical view: publish it if your buyers evaluate you through coding agents, or if your CMS generates it for free, and do not expect citations from it. He adds that nothing probes for the file, so you should link to it.

Before you start

You need a docs site and the address of its documentation. The llms.txt tool is free and builds the file from a docs sitemap. These steps follow the tool as read on 11 November 2026.

Make the file with the llms.txt tool

  1. Open the llms.txt tool. You see the fields “Project name”, “Docs URL”, “Summary” and “Sitemap”, and the button “Read pages”.
  2. Enter your project’s name in “Project name”.
  3. Add the address of your documentation to “Docs URL”.
  4. Describe the project in “Summary”.
  5. Fill in “Sitemap” if you know the address. If the field is empty, the tool tries /sitemap.xml, /sitemap-index.xml and /sitemap_index.xml on the docs host.
  6. Select “Read pages”. The section “Pages” lists your pages in two groups, “Docs” and “Optional”, each page with a checkbox. Up to 12 shallow Docs pages are checked to start. If your sitemap is an index, the tool reads only the first three child sitemaps, and it lists at most 500 pages.
  7. Check or uncheck pages. Pages whose paths contain words such as changelog, blog or legal go into Optional. By convention, Searchable says, an Optional section holds secondary links that a tool can skip when it is short on space. Superlines names listing every page instead of curating as a common mistake.
  8. Use the “Copy” button on the output labelled “/llms.txt”.
  9. Save the text as a file named llms.txt at the root of your domain. Searchable says it goes there, in the same place as robots.txt, so it loads at example.com/llms.txt. The proposal also allows one in a subpath such as example.com/docs/llms.txt. Open the address in a browser and the Markdown loads as plain text.

Check whether anything requests the file

The Crawler log reader is a free tool that finds AI crawler requests in a server access log, including requests for /llms.txt.

  1. Paste log lines into the reader, drop a file onto it or press “Choose file”. It accepts combined-format lines from nginx and Apache and JSON lines from Caddy and Cloudflare. It runs in your browser and nothing is uploaded. “Try the sample log” loads sample data first.
  2. Read the section “AI crawlers”. It shows a request total and, for each crawler, its operator, its three most requested paths, “Requests” and “Last seen”. If no AI crawler is in the log, the tool says “No AI crawler requests in this log.”
  3. Read the section “Files”. For /llms.txt it shows how many requests the file got, which crawlers asked for it and the latest status code.

ElBermawy cautions that a fetch is not a read. A request shows that the bytes moved, and it does not show whether anything acted on them.