Governing AI crawlers comes down to three decisions: who gets your content, what they may do with it afterward, and how you verify the requester is the crawler it claims to be. llms.txt answers none of them. It is a content-distribution convention proposed by Jeremy Howard on September 3, 2024 at llmstxt.org, it has never been adopted by any standards body, and Ahrefs' June 15, 2026 log study of 137,210 domains found that 97% of llms.txt files received zero requests during May 2026. What actually determines crawler access is robots.txt (RFC 9309, September 2022, Standards Track), the default policy your CDN is quietly changing underneath you, and enforcement at the network layer. Here is where each one's authority begins and ends, and where each one bites.

What llms.txt specifies, and who actually reads it

The spec is small. llmstxt.org defines: an H1 with the project name (the only required section), a blockquote summary, zero or more non-heading Markdown sections, then zero or more H2-delimited file lists of URLs. The ## Optional section carries special semantics — a consumer working with a constrained context budget may skip everything under it. The file is Markdown, served from the domain root.

One correction before anything else. A large volume of 2026 content asserts that the W3C published a working draft to standardize llms.txt in June of that year, complete with specific provisions like a version header line and a prohibition on subpath variants. No such document exists in the W3C TR index. I checked. Do not cite it, and do not restructure your file to satisfy provisions that were invented by content farms. As of this writing llms.txt has no RFC, no W3C Technical Report, and no standards-body endorsement of any kind.

The log data

Ahrefs' methodology matters here, because it avoids the usual failure mode of these studies. They probed the root path across 137,210 domains, required an HTTP 200 response, and confirmed the body was actually Markdown rather than an SPA fallback serving HTML. 38,360 domains (28%) passed. That adoption rate is not generalizable to the open web — the sample skews toward sites running Ahrefs analytics, which skews toward SEO-aware operators.

Of those files, 97% received zero requests in May 2026. Within the 3% that saw any traffic at all, 96% of requests came from bots, distributed as: SEO audit tools 21.7%, unidentified 14.9%, general web crawlers 13.1%, technology-profiling tools 11.6%, and industry participants studying llms.txt itself 12.1% combined (GEO/AEO tools 5.8%, llms.txt discovery bots 3.6%, research bots 2.7%). AI bots accounted for 19.5% combined — agents 10.5%, training crawlers 5.3%, assistants 2.5%, and retrieval bots, the category that actually composes AI search answers, 1.1%.

The most decisive finding is a negative one: on domains without an llms.txt, AI bots never probed for the path. Not once. They are not building a prior about whether a given site has one. This kills the most common justification for shipping the file — "it costs nothing, and maybe someone reads it later." Nobody is checking, so there is no option value accruing.

Where the vendors actually stand

Google's AI features documentation (last updated 2025-12-10) leaves no room: "You don't need to create new machine readable files, AI text files, or markup to appear in these features." The same page positions robots.txt directives for Googlebot as the site owner's control over crawling, Google-Extended as the opt-out for AI training and grounding in Google's other systems, and nosnippet / data-nosnippet / max-snippet / noindex as the controls over what gets displayed. llms.txt appears nowhere in that control surface.

There is a misreading worth naming explicitly, because it drives most of the remaining enthusiasm. OpenAI's own developer documentation serves /llms.txt and offers Markdown variants by appending .md to page URLs. That is OpenAI acting as a publisher, feeding clean docs to coding tools. It is not evidence that OpenAI's crawlers consume anyone else's llms.txt. Anthropic's crawler documentation does not mention llms.txt at all. A company publishing the file tells you nothing about whether that company's bots read yours.

When it is actually worth shipping

One case holds up: your readers are coding agents. A documentation site giving Cursor, Claude Code, or Cline a clean entry point — no navigation shell, no cookie banner, no sidebar chrome to parse — is a real use case, and it is what the 10.5% agent share in the data above represents.

But put the effort in the right place. The thing that actually gets fetched is not the index file, it is the .md twin of each page. The index only tells a tool where to look. If you can ship one of the two, ship the .md variants.

Two corollaries. An llms.txt auto-generated from your sitemap is worthless — the entire value of the format is curation, and dumping every URL into it is equivalent to not doing it. And llms-full.txt is not in the 2024 proposal; it is a downstream convention from documentation platforms, so do not treat it as a spec requirement.

robots.txt: the parts that bite in production

RFC 9309 is Standards Track, published September 2022. Four provisions cause real incidents.

5xx means complete disallow. The spec requires that on a 500–599 response, the crawler MUST assume complete disallow. If your robots.txt is rendered dynamically by your application framework, a single bad deploy takes your entire site out of every compliant crawler's index. Recovery is not immediate either: crawlers MAY cache and SHOULD NOT use a cached copy for more than 24 hours, so you inherit up to a day of lag after the fix. The spec does allow treating a 30-day outage as merely unavailable, but that timescale is irrelevant to anyone operating a business.

The operational consequence: serve robots.txt statically and monitor its status code independently of your homepage. This is the single highest-value line in this article, because it is the one that converts directly into lost traffic.

4xx means full allow. On 400–499 the crawler MAY access any resources on the server. A 404 is not a conservative default; it is an open door. Deleting robots.txt does not tighten anything.

Longest match wins, not first match. The spec states the most specific match MUST be used, where specificity is measured in octets. Engineers carrying intuitions from older, order-sensitive parsers routinely write Allow/Disallow pairs that resolve to the opposite of what they intended.

The parsing limit MUST be at least 500 kibibytes. "At least" means a crawler is free to read only the first 500 KiB. Programmatically generated robots.txt files that grow past this get silently truncated, and every rule after the cutoff quietly stops existing.

One hazard that is not in the RFC: sites whose error pages return HTTP 200 with an HTML body. A failed robots.txt fetch then parses as a pile of invalid lines, which is functionally equivalent to allowing everything — and your monitoring shows a healthy 200 the whole time.

Segment by purpose, not by vendor

The major crawlers split their user agents by purpose, and your policy should follow that split rather than blocking company names wholesale.

OpenAI runs GPTBot (foundation model training), OAI-SearchBot (surfacing sites in ChatGPT search results), ChatGPT-User (user-initiated actions in ChatGPT and custom GPTs), and OAI-AdsBot (validating the safety of pages submitted as ads, explicitly not used for training). IP ranges are published per-bot at endpoints like openai.com/gptbot.json.

Anthropic runs ClaudeBot (collecting content that may contribute to training), Claude-SearchBot (improving search result quality), and Claude-User (fetching on behalf of a user's question). The IP list is at claude.com/crawling/bots.json, and the documentation states the bots honor robots.txt and support the Crawl-delay extension.

Google separates Googlebot from Google-Extended, where the latter is the opt-out for Gemini training and grounding and has no effect on Search ranking.

The distinction that matters most: ChatGPT-User and Claude-User are user-initiated fetches, not crawlers. OpenAI's documentation says directly that because ChatGPT-User is triggered by a user request rather than automatic crawling, robots.txt may not apply to it. Blocking that class is approximately blocking a human who pasted your link into a chat box. You lose the referral and gain nothing.

By the same logic, if you want to be cited in AI answers, you cannot block OAI-SearchBot or Claude-SearchBot. The defensible default is: allow search-class and user-triggered fetchers, deny training-class crawlers.

User-Agent: *
Content-Signal: search=yes, ai-train=no
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

Content-Signal: cheap, worth adding, not enforcement

Cloudflare announced the Content Signals Policy on September 24, 2025. It adds a line to robots.txt expressing preferences about what happens to content after it has been accessed — a dimension robots.txt itself has never covered.

Three signals are defined: search (building a search index and returning hyperlinks and short excerpts), ai-input (feeding content into models in real time — RAG, grounding, generative search answers), and ai-train (training or fine-tuning models). Values are comma-delimited yes/no. Content-Signal is a directive inside a user-agent group — it applies to the group it sits in, which is why the snippet above places it under User-Agent: * rather than trailing after the last bot-specific group, matching Cloudflare's own example.

There are no universal defaults. The policy text is explicit that if an operator omits a signal, they neither grant nor restrict permission for that use. This inverts the intuition most people bring to it, and it is the sharpest semantic difference from robots.txt's allow/deny model.

What it adds over robots.txt is legal framing rather than technical force. Cloudflare positions the signals as "express reservations of rights under Article 4 of the European Union Directive 2019/790" — the text-and-data-mining reservation under the EU DSM Directive. The cost of adding it is effectively zero, since an RFC 9309-conforming parser ignores the unknown directive and your existing rules are unaffected.

Cloudflare says the rest itself: these are preferences, crawlers may ignore them, and you should pair them with WAF rules and Bot Management for actual protection. Treat it as a declaration, not a switch.

Your CDN's defaults are moving

This is the one section that changes your production behavior whether or not you do anything.

From July 1, 2025, Cloudflare began defaulting new domains to block AI crawlers, and launched pay per crawl in private beta. The mechanism is worth knowing even if you never enable it: HTTP 402 Payment Required, with crawler-price, crawler-exact-price, and crawler-max-price headers negotiating the amount and crawler-charged confirming the charge on a successful 200. Crawler identity is established through Web Bot Auth HTTP message signatures — requests carry signature-agent, signature-input, and signature.

The July 1, 2026 update classifies traffic into Search, Agent, and Training, alongside Transact, Data Collection, Security Testing, SEO, Ads Verification, Social/Link Preview, Feed Fetching, and Monitoring. Effective September 15, 2026, all new domains onboarding to Cloudflare default to: Training and Agent blocked on ad-monetized pages, Search still allowed. Existing customers can opt out through Security settings before that date.

The provision to internalize: mixed-use crawlers are allowed or blocked according to all of their behaviors. A crawler that performs both Search and Training gets blocked entirely when you block Training, and Cloudflare names Googlebot as the example. A toggle that reads in the UI as "don't let them train on my content" can therefore take your organic search traffic with it. Run it in dry-run or log-only mode first, review the actual matched user-agent list, then enforce.

The same day, Cloudflare introduced Pay Per Use, arguing that crawl count is a crude proxy for value — their figure is that more than 50% of crawl traffic from good bots re-fetches pages that have not changed, which is exactly the traffic conditional requests are supposed to collapse (see ETags, Preconditions, and CDN Caching for APIs). A page might be crawled once and cited in thousands of answers, or crawled repeatedly and never used. Emerging models include Ceramic.ai's pay-per-query and You.com's pay-on-demand. This is early; do not bake it into architectural assumptions.

The standards work: track it, don't build on it

Two IETF working groups are relevant. Neither has produced an RFC.

AIPREF (AI Preferences). draft-ietf-aipref-vocab-06 was published April 28, 2026, is an Active Internet-Draft with intended status Proposed Standard, and expires October 30, 2026. It is not an RFC, and its IESG state remains "I-D Exists." The vocabulary is deliberately minimal — the -06 text names train-ai (using an asset in the production or refinement of a model that can generate content) and search (applications whose primary purpose is selecting assets and directing users to their location). Values are allow (y) and disallow (n); absent a stated preference the value is "unknown," and the specification explicitly takes no position on what default should be assigned to unknown.

The more consequential fact is about the other half. draft-ietf-aipref-attach-04 stopped at October 28, 2025 and is expired. That was the document defining how preferences attach to content in HTTP, and it was the one slated to update RFC 9309 to carry usage preferences in robots.txt.

So the current state is: the vocabulary is advancing and the attachment mechanism has lapsed. There is no IETF-consensus way to express an AIPREF preference today — the same expired-draft-plus-vendor-divergence shape as the Idempotency-Key header. That gap is precisely why vendor schemes like Content-Signal exist and will persist for a while — the standards process left a hole and a vendor filled it.

Web Bot Auth (webbotauth). A chartered working group with 10 active Internet-Drafts and zero RFCs — and every one of those drafts is still an individual submission; there is not a single adopted draft-ietf-webbotauth-* document yet. "HTTP Message Signatures for automated traffic" (draft-meunier-webbotauth-httpsig-protocol) reached -01 on August 6, 2026. The approach builds on RFC 9421 (HTTP Message Signatures, February 2024, Standards Track): a bot operator generates an Ed25519 keypair, publishes the public key as JWKS at /.well-known/http-message-signatures-directory on a domain it controls, and signs every outbound request (verifying signatures like these, replay windows included, is the same server-side problem covered in Webhook Signatures and Replay Protection). The origin verifies the signature and knows cryptographically which operator sent it.

Why identity beats declaration

On August 4, 2025 Cloudflare published an analysis accusing Perplexity of using undeclared crawlers to evade no-crawl directives — switching to a generic user agent impersonating Chrome on macOS when its declared bots were blocked, and sourcing requests from IPs across unrelated ASNs. Cloudflare put the volume at 3–6 million requests per day across tens of thousands of domains, and removed Perplexity from its Verified Bots program.

The useful lesson is not that one company misbehaved. It is that governance keyed on a User-Agent string has no structural enforcement power. The UA is a self-report, and self-reports can change. robots.txt and Content-Signal live at the declaration layer: their value is expressing intent and reserving legal rights, not preventing access. Only network-layer blocking combined with cryptographic identity constitutes enforcement. That is what makes Web Bot Auth the standards work worth watching — it converts "this is GPTBot" from a forgeable string into a verifiable fact.

Until it ships, the strongest available control is verification against published IP ranges. Endpoints like openai.com/gptbot.json and claude.com/crawling/bots.json give you a checkable source of truth. A request whose UA matches but whose source IP falls outside the published range is an impersonator, and you should treat it as one.

What to actually ship

  1. Serve robots.txt from static hosting and monitor its status code separately. A 5xx is a site-wide incident, not a config nit. While you are there, confirm your error pages do not return 200 with an HTML body.
  2. Segment by purpose. Deny training-class crawlers; allow search-class and user-triggered fetchers. Blanket-blocking by company name costs you citations and referrals for no gain.
  3. Add Content-Signal. Near-zero cost, real value as a DSM Article 4 rights reservation. Do not expect it to stop anything.
  4. Audit your CDN's September 15, 2026 default change, specifically whether the mixed-use crawler rule catches Googlebot. Dry-run before enforcing.
  5. If you want enforcement, do it at the network layer: WAF rules plus verification against official IP ranges. Never trust the UA alone.
  6. Ship llms.txt in exactly one situation — your documentation site serves coding agents. Prioritize per-page .md variants over the index file. Do not ship it for "AI visibility"; the log data does not support it and Google states plainly that Search does not use it — see the GEO/AEO field guide for what actually moves that needle.
  7. Track AIPREF, don't implement it. The vocabulary is still a draft and the attachment mechanism has expired. There is nothing stable to build against yet.