Anthropic Wants a Law That Can Block an AI Model — But Only If a Judge Agrees

Anthropic has quietly been shopping a draft AI safety bill around Washington, and the headline number is wild: a model could be blocked from release if there's a "foreseeable, imminent risk" that more than 50 people could die, or that a billion dollars in property could be destroyed. On paper that sounds like the strongest AI law anyone has proposed. Read the fine print and it starts to look like something else entirely.

Politico broke the story, and TechCrunch's Lucas Ropek flagged the detail that actually matters. The bill does create a mechanism to stop a dangerous model — but it also explicitly bars any federal agency from halting a model on its own authority. The government would have to go ask a judge in Washington, D.C. for permission first. So the "block" in the headline comes with a courtroom attached.

What the draft actually says

The catastrophic scenarios are narrow and specific. The risk has to stem from one of three things: the creation or release of a biological weapon, the model evading its developer's controls, or a cyberattack. The 50-deaths and $1-billion-in-damage thresholds mirror California's SB 53, which was recently signed into law and uses the same numbers to define catastrophic risk. So Anthropic isn't inventing new math here — it's nationalizing a standard California already set.

Under the draft, the danger assessment wouldn't come from a government agency at all. It would come from "independent evaluators" registered with the U.S. Commerce Department. Those evaluators could be paid by the AI labs themselves, as long as the payment isn't contingent on the results. If an evaluator concludes an imminent catastrophic risk exists, they get 72 hours to notify Commerce, which then has to petition a D.C. judge to stop the release.

The obvious hole

Saying "you can pay the auditor, just don't pay them based on the verdict" is the kind of rule that sounds airtight until you picture how it works in practice. Nobody has explained how you'd prove that a lab's money didn't influence an evaluation, or who audits the auditors. DeepMind chairman Demis Hassabis floated a similar idea months ago — an AI regulator modeled on FINRA, the self-policing body the finance industry runs for itself. That model has always had the same structural question hanging over it: how independent is a watchdog funded by the industry it watches?

The other thing worth noticing is what the bill forbids. Preventing federal agencies from acting unilaterally is a genuine concession to the labs. It means even if a regulator sees something alarming, it can't pull a model unilaterally — it needs a court. In a world where the danger is supposed to be "imminent," routing the response through a judge is a deliberate speed bump.

Why this lands oddly right now

The risks the draft focuses on feel a little detached from what's actually been going wrong lately. The real anxiety in tech this year has been agentic hacking — AI programs disobeying operators and going after websites on the open internet. That's not bioweapons or a billion dollars in damage; it's messy, mundane, and already happening. A bill built around civilizational scenarios doesn't obviously address it.

There's also a pattern here that's hard to ignore. AI labs keep warning that their own products could end civilization, while simultaneously resisting mandatory oversight. Anthropic's proposal is a step further than most industry suggestions — Nvidia's Jensen Huang recently argued the industry needs no new laws at all — but it's still a framework the industry wrote for itself. Anthropic's spokesperson confirmed the draft to Politico and said it reflects the company's Advanced AI Framework, which it's been pushing with lawmakers since June.

My take

A draft bill is not a law, and Congress has a long history of ignoring even popular legislation. But the shape of this one is instructive. The industry isn't fighting regulation anymore — it's writing it. And what it wrote is a system where the labs pay the evaluators and the only entity that can actually stop a release is a judge.

That may be a reasonable design. Courts exist to prevent regulators from acting on vibes, and a 72-hour clock plus judicial review is more process than most AI proposals get. But if you're going to sell a bill on the promise that it can block a deadly model, the public deserves a straight answer to one question: how fast can that actually happen when it needs to?