One Line of robots.txt Now Blocks AI Training — Here's What It Actually Does

For two years, keeping AI companies off your content meant playing whack-a-mole. You named GPTBot. Then ClaudeBot. Then Google-Extended, then CCBot, then the dozen other crawlers that appeared whenever a lab shipped something new. Four rule sets, maintained forever, all saying the same thing: stay out.

That mess is what Friday’s announcement actually fixes. Google, Anthropic, OpenAI, and Microsoft have committed to honoring a single robots.txt directive, proposed through the IETF, that lets a publisher opt out of generative AI training with one rule instead of four. Anthropic’s standards lead Kevin Kelley wrote the proposal. If you run a site and you’ve been putting this off because the per-crawler version was tedious, this is the version worth ten minutes of your afternoon.

Here’s the line, and what it honestly will and won’t do for you.

The snippet

Open your robots.txt and add a user-agent group for the new token, __usage_ai-training:

User-agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

That’s the whole idea. Disallow: / blocks AI training use site-wide. The Allow: line carves out exceptions, like a press kit you want models to see. You can get more surgical: block /archive/ but allow /blog/, or whatever split matches how you actually want your work used.

Before this, the same intent meant a separate block for each provider’s crawler, and a chore of keeping that list current every time a new bot launched. Now one token covers the four labs that agreed to honor it.

What it actually blocks

Scope matters here, and the scope is narrower than the headlines suggest.

The token covers training only. It does not stop crawling. It does not affect search indexing — your Google rankings are untouched. And it does nothing about data already scraped. If a lab’s crawler took your archive in 2025, this line does not claw it back.

That last point is where most coverage goes quiet. Every publisher that spent the last two years getting scraped has already been scraped. This directive covers what happens from each lab’s start date forward. Worth setting. Not worth celebrating.

Four labs, four timelines

The commitment is real, but the rollout is staggered to the point of comedy:

• Anthropic honors the token now.
• Microsoft starts February 1, 2027.
• Google says early 2027.
• OpenAI says “the coming months,” unspecified.

So for the next few months, the practical effect for most publishers is close to zero. One of four labs is live. You’re writing a rule that mostly takes effect next year. That doesn’t make it pointless — it makes it early. Add the line now and you stop thinking about it.

There’s also a wrinkle in how Google treats it. When you set __usage_ai-training, Google reads it as a block on Google-Extended, the crawler behind Gemini’s training data. Publishers who wanted that could already set it per-crawler. Google’s implementation, in other words, is a rename of an existing control, not a new one. Fine. Renames that save you maintaining four lists are still worth having.

A sign on the gate, not a lock

Here’s the part I’d underline for anyone deciding whether this changes their licensing strategy. Nothing in it creates a legal duty.

The token is a proposal moving through the IETF, not a finished standard. Participation is voluntary. If a lab decides to ignore your Disallow line, your remedy is whatever it was last month: public pressure, negotiation, or a lawyer. The directive formalizes an etiquette rather than an enforcement mechanism.

I think it’s still meaningful, for a specific reason: it makes violations legible. When everyone maintains their own crawler list, a lab can quietly launch a new bot and shrug. When there’s one agreed token and a lab trains on a site that set it, that’s a clean, public broken promise. Opt-out systems only work when skipping them is conspicuous. This one now is, at least for the four names on the commitment.

But if your content is your main asset and you’ve been weighing formal licensing deals, this doesn’t replace them. Treat it as hygiene, not strategy.

What to do now

1. Add the __usage_ai-training group to robots.txt today. Two minutes. Keep your existing per-crawler rules in place until all four labs are actually enforcing — belt and suspenders through 2027.
2. Decide your exceptions deliberately. Press kit, public docs, anything where being in a model helps you: Allow: those paths. Everything else stays out.
3. Remember search is unaffected. If your SEO traffic depends on Google crawl, this line does not touch it. The number of people who will worry about this is not zero.
4. If you’ve licensed content to any of these labs already, check your contracts before flipping anything — a blanket Disallow could conflict with a deal that permits training use.
5. Circle back in February 2027. That’s Microsoft’s start date and roughly Google’s. When all four are live, you can retire the old per-crawler blocks and keep just the token.

The modest win, stated modestly

One line of intent replacing four rule sets is a real improvement for anyone who publishes online. It’s also, on its own, a small one — voluntary, forward-only, and enforced by reputation rather than law.

The honest summary: the plumbing finally exists, three of four labs will honor it eventually, and deciding what your content is worth didn’t get any easier. Add the line. Keep the receipts. The negotiation is still coming.

Based on reporting from Engadget and the IETF proposal by Anthropic’s Kevin Kelley.