An llms.txt file is a short, Markdown document placed at a website’s root that gives AI systems a curated shortlist of its most important pages. It works alongside AI crawler control in robots.txt, and most UK businesses still have neither properly set up. Below, we cover what llms.txt does, how it differs from robots.txt, and the steps worth taking before assuming either file is working.
What Is llms.txt?
llms.txt is a plain text file, written in Markdown and saved at yourdomain.co.uk/llms.txt. It points large language models towards a website’s most useful content, rather than leaving them to work through menus, adverts and scripts themselves. It was proposed in September 2024 by Jeremy Howard of Answer.AI, and the specification is publicly available at llmstxt.org. A typical file starts with an H1 brand name and a short summary, followed by Markdown links to key pages.
llms.txt vs Robots.txt: What Is the Difference?
Think of robots.txt as an access rules file and llms.txt as a reading list. robots.txt follows the established Robots Exclusion Protocol and tells crawlers which parts of a site they may fetch, and most major bots, including Googlebot, GPTBot and ClaudeBot, generally respect it, though a handful of crawlers are known to ignore it, so it is not a guaranteed block. llms.txt works differently. It has no enforcement mechanism and simply provides a curated set of links that a provider may or may not choose to read.
Why AI Crawler Control Matters for UK Websites
UK websites can receive requests from dozens of bots, each with a different purpose. A single blanket robots.txt rule could block crawlers that contribute to search visibility while still allowing others that primarily collect content for training.
Which AI Crawlers Actually Visit Your Site?
Some of the AI crawlers worth knowing by name include:
- GPTBot (OpenAI training)
- OAI-SearchBot (ChatGPT search)
- ClaudeBot and Claude-SearchBot (Anthropic)
- PerplexityBot (Perplexity’s answer engine)
- Google-Extended (Gemini training, no effect on Google Search rankings)
- CCBot (Common Crawl, feeds open-source models)
How to Create an llms.txt File
Creating an llms.txt file is fairly straightforward. Write a Markdown document with your brand name as the H1, add a short summary, then include labelled links to the pages you want to highlight. Upload it to your root folder so it is available at /llms.txt.
The practical steps are:
- Shortlist your priority URLs and core services.
- Structure it as H1, a brief summary, then grouped links.
- Publish it at the root, rather than in a subfolder, so it sits at /llms.txt.
- Review it quarterly and update it when pages change.
Does llms.txt Actually Help UK Websites?
Here is the part worth knowing before you spend too much time on it: Google has stated that Search does not use llms.txt, and creating one will neither help nor harm rankings in Google Search or its AI features. That means it should not be presented as a Google ranking tactic.
Its more established value is with AI coding assistants and developer documentation. For most UK service businesses, proper AI crawler control in robots.txt currently has a more practical role than llms.txt on its own.
Optimise Your Website for AI Crawlers with Haarty Hanks
Getting AI crawler control right involves more than uploading a text file. You need to check your robots.txt rules, decide which bots you want to allow or block, and make sure your content is structured clearly enough for Google and AI answer engines to interpret. Our AI SEO Optimisation service covers these areas. Get in touch with our team for an AI crawler audit today.
FAQs
Is llms.txt the same as robots.txt?
No. robots.txt controls which crawlers may access which parts of a site, while llms.txt curates which content an AI system should treat as most important. They serve different jobs and work best used together, not as substitutes for one another.
Does llms.txt improve my Google rankings or AI Overviews?
No. Google has confirmed that Search does not use llms.txt, so creating one carries no ranking benefit or penalty. OpenAI, Anthropic and Perplexity have not confirmed that llms.txt affects citations or visibility in their consumer-facing answers either, though some AI providers reference the format in their own developer and agent documentation, so any wider benefit remains unproven rather than ruled out entirely.
How do I create an llms.txt file for my website?
Write a Markdown file with your brand name as the H1, a short summary, and grouped links to your priority pages, then upload it to your site’s root directory so it is accessible at yourdomain.co.uk/llms.txt.
Which AI crawlers should I allow on my website?
Most UK businesses benefit from allowing search and citation bots such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, while deciding separately whether to allow or block training bots such as GPTBot, ClaudeBot and Google-Extended based on their own content and data preferences.
Do I need both an llms.txt file and a robots.txt file?
Yes, if you want proper AI crawler control. robots.txt sets the access rules for every bot on your site, including AI crawlers, while llms.txt is an optional, additional layer that curates content for the AI systems that choose to read it.
Is llms.txt worth setting up for a small UK business website?
It costs very little to add and does no harm, so it is a reasonable low-effort addition, but it should not replace solid technical SEO, clear site structure and proper AI crawler control in robots.txt, which currently have far more proven impact.
