llms.txt is not robots.txt — and treating it that way costs you citations

We keep finding the same configuration: robots.txt disallowing GPTBot, PerplexityBot and similar agents, alongside a carefully maintained llms.txt at the site root. Someone read that llms.txt helps AI visibility, published one, and separately someone else applied a blanket AI crawler block because it seemed prudent.

The result is a site that has locked the door and posted directions to it. The crawler never arrives to read either file's contents, and the effort spent curating llms.txt produces nothing. Neither person was wrong about their own piece; nobody checked how the two interacted.

This is worth stating plainly because the two files are discussed together constantly, in a way that implies they are alternatives or complements. They are neither. One is access control; the other is a content pointer with no enforcement behind it.

robots.txt is a decades-old convention that well-behaved crawlers honour. AI crawlers from the major providers publish their user agent strings and respect the directives, which makes it the only mechanism you have for allowing or blocking them at the fetch stage.

The decision it encodes is strategic rather than technical. Blocking AI crawlers removes you from the answers your buyers read. For most businesses that is a worse outcome than the training exposure it prevents, because the content is a route to the sale rather than the product itself.

The genuine exception is publishing, where the content is the product. There, restricting access or negotiating licensing is a legitimate commercial position, and we say so on our media and publishing page even though it runs against the advice we give everyone else.

The proposal is a markdown file at the site root listing your most useful content in a clean, readable form, so a model consuming it gets a curated view rather than crawling navigation and boilerplate. As an idea it is sensible and the effort involved is small.

What it does not do is grant permission, withhold permission, or oblige anyone to read it. No major engine has committed to honouring it, and there is no reliable public evidence that major models fetch it. Publishing one is a low-cost bet on a convention taking hold, not a visibility strategy.

Treating it as access control is the specific error. It cannot stop training, cannot restrict use, and cannot substitute for a robots.txt decision. If someone tells you llms.txt protects your content, they have misunderstood what it is.

Make the robots.txt decision deliberately and write down why. Allow AI crawlers unless you have a specific commercial reason not to, and if you block them, know what you are trading away. This is a five-minute conversation that most organisations have never actually had.

Then check the interaction. Fetch your own robots.txt, list every AI user agent you block, and confirm that matches your intent rather than a template someone copied. This is where the contradiction usually surfaces.

Publish llms.txt if maintaining it is genuinely easy, and treat it as housekeeping rather than as a channel. The work that actually moves generative visibility is making pages retrievable and earning presence in the third-party sources models cite — neither of which is a file at your root.