In September I added an llms.txt file to every Djangify site. It felt like the first useful thing I had done about AI search. The file is plain text in Markdown, it describes a site and its products for machines to read, and it builds itself and keeps itself up to date. I told people about it. I wrote it into the documentation.
Then in the autumn I kept running into the same sentence on forums and in newsletters: nobody reads it. Some people said AI companies had never said they would. Others said their own logs showed no visits. I had a choice. I could ignore it and carry on feeling pleased, or I could go and find out what was true. This is what I found, and it changed what I tell Djangify owners to spend their time on.
- Google says you do not need an llms.txt file, special markup or any new machine-readable file to appear in its AI features.
- A published server-log study found llms.txt was fetched very rarely by AI assistants, with one coding assistant as the main exception.
- ChatGPT, Perplexity and Claude each use their own crawlers, and you control them through robots.txt.
- What helps in practice: pages that can be crawled and indexed, real text in the HTML, structured data that matches the page, and a clear explanation of what each product is for.
- llms.txt is harmless and automatic on Djangify, but it is a small extra and not the plan.
The sentence I kept reading
The first thing I did was go to the source I trust most for search, which is Google's own documentation. It has a page about how a site appears in its AI features, meaning AI Overviews and AI Mode. The page is short and it is direct. There are no extra requirements for appearing in those features. The normal rules of search still apply. A page has to be indexed and eligible to appear in search with a snippet. And you do not need to create new machine-readable files, AI text files or special markup to be included.
That is Google, which is one company. I wanted to know about the others, and about what the logs say, because a statement of policy is one thing and what crawlers really do is another.
What the evidence says about llms.txt
The best evidence I could find came from a studio called Evil Martians, who logged every request to their own site on the server, which does not depend on JavaScript, and wrote up what they saw. They first ran the study from May to July 2026 and re-measured in September and October, and the write-up was updated on November 4, 2026. You can read it yourself: Which AI actually reads your site?
Here is what stood out to me. Their two llms.txt files were fetched about 600 times in total. Only about 47 of those came from named AI assistants, and 43 of the 47 came from one tool, Claude Code, which is a coding assistant. Most of the rest came from SEO crawlers, scanners and scripts. The chat assistants they measured mostly asked for ordinary HTML pages. The write-up also quotes Google's John Mueller as saying that no AI system currently uses llms.txt, and the logs mostly agree with him.
What that suggests
llms.txt probably helps when
- A coding assistant reads your software documentation
- A developer points a tool at your site on purpose
- You want a tidy, machine-friendly summary kept up to date
It does not seem to help when
- You hope ChatGPT will start recommending your product because of it
- You expect it to change how Google shows your pages
- You are relying on it instead of normal search basics
This is one site's logs over a few months, and AI companies change what their crawlers do without telling anyone. Treat it as the best evidence I could find in November 2026, not as a permanent law.
What AI systems actually use
If llms.txt is not the route in, what is? The answer turned out to be less exotic than I expected. Each assistant has its own crawler and its own way of finding pages, and the companies publish what those crawlers do. I read the documentation from OpenAI, Perplexity and Anthropic, and the table below is built from it.
| Crawler | Run by | What it is for |
|---|---|---|
| Googlebot | Google Search, which also feeds AI Overviews and AI Mode | |
| OAI-SearchBot | OpenAI | Finding pages to show in ChatGPT search. Blocking it keeps your site out of ChatGPT search answers |
| ChatGPT-User | OpenAI | Opening a page when a person asks ChatGPT a question that needs it |
| GPTBot | OpenAI | Collecting material that may be used to train models |
| PerplexityBot | Perplexity | Surfacing and linking sites in Perplexity results. It is not used for training |
| Claude-SearchBot | Anthropic | Indexing pages to improve Claude's search results |
| Claude-User | Anthropic | Opening a page when a person asks Claude a question that needs it |
| ClaudeBot | Anthropic | Collecting material for model training |
Two things in that table surprised me. First, the training crawlers and the search crawlers are different. You can allow a company's search crawler and still decline its training crawler, so a decision about one is not a decision about the other. Second, the crawlers that fetch a page because a person asked, such as ChatGPT-User and Perplexity-User, may not follow robots.txt in the same way, because a human triggered the visit. What decides whether you are findable in an assistant's search is mostly the search crawler, and that one reads robots.txt like any other.
The links are worth opening, because these details change: OpenAI's crawler guide, Perplexity's crawler guide and Anthropic's crawler guide.
What I found in my own files
With the table in front of me, I opened the demo site's robots.txt, expecting to find a mistake. It was fine. It allows everything public, keeps crawlers out of the admin, accounts and checkout areas, and names the AI crawlers I had added in September. Because the main rule allows every crawler, the search crawlers in the table are allowed as well.
Then I opened the demo's llms.txt, the file I had been so pleased about. It lists the products and the blog posts with a short description of each, and it carries the founder bio from the settings. It does its job. It is just a job that, as far as anyone can show, almost no AI assistant is asking for.
The honest conclusion was uncomfortable but useful. I had put effort into a file that mostly helps a few coding tools, and I had barely looked at what the search crawlers see when they arrive at a product page. That is where the work is.
What I do now instead
These are the things that stay true whichever assistant is fashionable this month. None of them is clever. All of them are about making a page easy to read for anything that arrives, human or machine.
- Make sure the page can be found and indexedGoogle says a page has to be indexed and eligible to show with a snippet. Submit your sitemap and check in Search Console that your product pages are indexed.
- Check robots.txt does not shut the search crawlers outLook for OAI-SearchBot, PerplexityBot, Claude-SearchBot and Googlebot. If you block them on purpose, you are choosing not to appear.
- Put the words in the HTMLMany AI crawlers do not run JavaScript, so text that only appears after a script runs may never be seen. A Djangify page is built on the server and arrives as ready-made HTML.
- Describe each product plainlyProduct Knowledge has four fields: the problem it solves, who it is for, what makes it different and the words a buyer would search. They feed the structured data on the page.
- Answer the questions buyers really askAdd five to eight questions with short, direct answers. They show on the page and in the page's FAQ data, and both are written from the same source so they cannot disagree.
- Say who made itEach product page credits its creator in its structured data, so the page names the person behind the work.
Google's guidance on structured data is worth remembering here. It should match what a visitor can see on the page. Djangify writes the visible questions and the structured data from the same source, for exactly that reason. The longer explanation is in my earlier post on making a digital site understandable to AI, and the setup page is AI Discoverability: Product Knowledge and llms.txt.
Try it on one product page
Pick your best product and work through this once. It takes about an hour the first time. Your ticks stay in this browser.
Make one product page easy to read for machines
- Check it is indexedIn Search Console, inspect the product address.
- Read your robots.txtOpen yourdomain.com/robots.txt and look for blocks on search crawlers.
Ask your AI to check it
Paste the file into a chat and ask it to tell you, in plain words, whether any of the crawlers in the table are blocked.
- Fill in the four Product Knowledge fieldsProblem, audience, difference and search terms, in plain language.
- Add five to eight real buyer questionsUse the ones you answer in emails again and again.
- Run the Site AuditCheck the AI Discoverability section for this product.
- Read the page as a strangerWould someone who has never heard of you understand what this is and who it is for?
Here is my robots.txt file: [paste the file]. Tell me in plain words whether it blocks Googlebot, OAI-SearchBot, PerplexityBot or Claude-SearchBot from my public pages. Then tell me whether it blocks GPTBot or ClaudeBot, and explain that this is a separate choice about training. Do not change anything. Just explain.
Then score how readable your site is to something that cannot ask you a question. It is quick and it points at what to fix first.
How easy is your site to read for a machine?
- My product pages are indexed in Google.
- My robots.txt does not block the search crawlers.
- My product pages say plainly what the product is, who it is for and what problem it solves.
- My product pages answer real buyer questions in visible text.
- My pages say who made the product.
- Early daysStart with whether your product pages are indexed. Nothing else matters until they are.
- Getting therePick the lowest score and fix that one thing this week.
- In good shapeKeep the basics current and add new products as you make them.
You can see a finished product page with this structure on the demo, in this demo product. The page you are reading is built from Djangify content blocks. The free ones are explained in this free article, and the guide and the self check above are premium blocks from Djangify Content Blocks Premium.
Questions
Should I delete my llms.txt file?
There is no need. On Djangify it builds itself and costs you nothing, and some coding assistants do fetch it. Just do not treat it as your AI search plan.
Should I block the training crawlers?
That is your choice, and it is a separate choice from search. Blocking GPTBot or ClaudeBot is about training. Blocking OAI-SearchBot or Claude-SearchBot affects whether you appear in those assistants' search answers.
Does structured data guarantee a recommendation?
No. Google notes that meeting the conditions does not guarantee crawling, indexing or showing. Structured data helps a system understand a page. It does not promise a result.
Is this different from ordinary SEO?
Mostly not. For Google's AI features the guidance is that standard search practice applies. Writing clear pages for people turns out to be what helps machines too.
- Google says no special files or markup are needed for its AI features, and one published log study found llms.txt rarely fetched by AI assistants.
- ChatGPT, Perplexity and Claude each use their own search crawler, and robots.txt controls whether they can read you.
- Indexed pages, real HTML text, matching structured data and a plain description of each product do the work.
- llms.txt is a harmless automatic extra, not the strategy.
Make Your Digital Site Understandable to AI
The Product Knowledge, Q&A and creator credit features in detail.
Read more →A Weekly Routine for Running Your Digital Business With AI
Where the audit and the drafting fit into a week.
Read more →Why Your Site's Lighthouse Score Matters
The speed and accessibility basics that every crawler benefits from.
Read more →