I added llms.txt so you don't have to
Last week, during an SEO tidy-up, I added an /llms.txt to this blog — the tidy little Markdown file that's supposed to hand AI crawlers a clean map of your content. It took ten minutes and it felt responsible, like flossing. Then I went looking for evidence that anything reads it. What I found is that I did this mostly for my own peace of mind, and I might as well tell you so you can skip the guilt.
What it is, briefly
llms.txt was proposed by Jeremy Howard of Answer.AI in September 2024: a file at your site root offering a curated, link-rich summary of your key pages, so a language model can find and cite you without wading through rendered HTML. There's usually a companion /llms-full.txt with the whole thing concatenated. It's a sensible idea. It's also, as a spec, entirely a community proposal — no IETF, no W3C, no one obligated to honour it.
The awkward part: nobody's fetching it
The clearest voice here is Google's John Mueller, and he hasn't been subtle. On Reddit in April 2025: "none of the AI services have said they're using LLMs.TXT (and you can tell when you look at your server logs that they don't even check for it)." He compared it to the old keywords meta tag — a signal the site owner controls, which is exactly why search engines learned to ignore it. On Bluesky two months later: "FWIW no AI system currently uses llms.txt."
You could dismiss that as one Googler's opinion, except the data lines up. A March 2026 study by Trakkr scanned nearly 38,000 AI-cited domains and found sites with llms.txt averaged 6.8 citations versus 6.7 without — a difference of statistical nothing (p=0.85). Their conclusion: "whatever drives AI citations, it isn't llms.txt." Adoption itself is still tiny and skewed — a mid-2026 crawl of the top 10,000 sites found around 5–6% had one, and the ones that do tend to be smaller sites seeking visibility, not the established domains models actually cite. As one researcher put it, the products that would theoretically read these files haven't even added one to their own domains.
A file nobody is required to read, on a web where crawlers already ignore the files they are asked to respect, is a bet on goodwill — not a mechanism.
How to check for yourself (the honest test)
Mueller's challenge is the right one: look at your own logs. Grep your access log for GET /llms.txt and bucket by user-agent. If AI crawlers cared, you'd see GPTBot, ClaudeBot, PerplexityBot, CCBot and friends fetching it. The tell is that they'll happily crawl your actual pages and never touch the file you made specifically for them.
I'll be straight about my own attempt: my blog's container recycles its logs on every deploy, so I don't have a long enough window to make a fair claim yet. I'm not going to wave around a "zero fetches!" screenshot from an access log that's a few hours old — that would be exactly the kind of tidy-but-hollow evidence this post is complaining about. So I've left the file up and I'll actually watch, over a real stretch of weeks, and report back if anything surprising shows.
The case for keeping it anyway
I want to steelman it, because the skeptics can overshoot. There's one place llms.txt is genuinely useful today: tools you point at your own content. If you feed an IDE assistant or a docs chatbot your site, clean Markdown beats parsing bloated HTML — fewer tokens, less truncation, a summary you control instead of one a model guesses. That value is real and it doesn't depend on GPTBot's goodwill at all. Add the "it's cheap and it positions you for if they adopt it" argument and you've got a perfectly reasonable reason to spend the ten minutes.
The kicker
Here's what tipped me from "harmless" to "eyes open." In August 2025 Cloudflare caught Perplexity using undeclared stealth crawlers — rotating user-agents and IPs to slip past sites that had explicitly blocked PerplexityBot in robots.txt. If crawlers will route around a directive file when it suits them, the odds they'll voluntarily read a helper file with nothing but etiquette behind it are about what you'd expect. robots.txt is a request that's increasingly ignored; llms.txt is a request nobody agreed to in the first place.
So: I added llms.txt so you don't have to feel bad about not bothering. Keep it if you use agentic tools on your own docs, or if ten minutes of future-proofing helps you sleep. Just don't expect it to move a single citation in an AI answer this year. The web that reads it doesn't exist yet, and might not show up.