{"id":1679,"date":"2026-09-04T10:12:58","date_gmt":"2026-09-04T10:12:58","guid":{"rendered":"https:\/\/www.haartyhanks.com\/blog\/?p=1679"},"modified":"2026-09-21T10:17:40","modified_gmt":"2026-09-21T10:17:40","slug":"llms-txt-ai-crawler-control-uk-websites","status":"publish","type":"post","link":"https:\/\/www.haartyhanks.com\/blog\/llms-txt-ai-crawler-control-uk-websites\/","title":{"rendered":"llms.txt and AI Crawler Control: The Setup Most UK Websites Still Do Not Have"},"content":{"rendered":"

An llms.txt file is a short, Markdown document placed at a website’s root that gives AI systems a curated shortlist of its most important pages. It works alongside AI crawler control in robots.txt, and most UK businesses still have neither properly set up. Below, we cover what llms.txt does, how it differs from robots.txt, and the steps worth taking before assuming either file is working.<\/span>\u00a0<\/span><\/p>\n

What Is llms.txt?<\/span>\u00a0<\/span><\/h2>\n

llms.txt is a plain text file, written in Markdown and saved at yourdomain.co.uk\/llms.txt. It points large language models towards a website’s most useful content, rather than leaving them to work through menus, adverts and scripts themselves. It was proposed in September 2024 by Jeremy Howard of Answer.AI, and the specification is publicly available at<\/span> llmstxt.org<\/a><\/strong>. A typical file starts with an H1 brand name and a short summary, followed by Markdown links to key pages.\u00a0<\/span>\u00a0<\/span><\/p>\n

llms.txt vs Robots.txt: What Is the Difference?<\/span>\u00a0<\/span><\/h3>\n

Think of robots.txt as an access rules file and llms.txt as a reading list. robots.txt follows the established Robots Exclusion Protocol and tells crawlers which parts of a site they may fetch, and most major bots, including Googlebot, GPTBot and ClaudeBot, generally respect it, though a handful of crawlers are known to ignore it, so it is not a guaranteed block. llms.txt works differently. It has no enforcement mechanism and simply provides a curated set of links that a provider may or may not choose to read.<\/span>\u00a0<\/span><\/p>\n

Why AI Crawler Control Matters for UK Websites<\/span>\u00a0<\/span><\/h3>\n

UK websites can receive requests from dozens of bots, each with a different purpose. A single blanket robots.txt rule could block crawlers that contribute to search visibility while still allowing others that primarily collect content for training.\u00a0<\/span>\u00a0<\/span><\/p>\n

Which AI Crawlers Actually Visit Your Site?<\/span>\u00a0<\/span><\/h2>\n

Some of the AI crawlers worth knowing by name include:<\/span>\u00a0<\/span><\/p>\n