Anthropic wants the government to have the power to block an AI model from shipping. Anthropic also wants that power to be narrow enough that it almost never gets used. Both of those things are in the same draft bill, and that tension is the whole story.
POLITICO obtained the proposal, which Anthropic has been pitching quietly to US lawmakers. Reuters and Bloomberg Law picked it up on October 16, 2026. The core mechanism: the government could step in and stop a model’s release if that model is determined to imminently risk killing or injuring more than 50 people, or causing more than $1 billion in damages. The draft also calls for independent evaluators to be part of the regulation process. An Anthropic spokesperson described the contents as sensible policies Congress should act on.
I review tools for a living. I spend my days figuring out whether a product does what its marketing says it does. So when a model vendor writes its own safety rules, my instinct is the same as when a vendor writes its own benchmark — read the thresholds first, and read them carefully.
Thresholds are product decisions dressed as policy
Fifty casualties. One billion dollars. Imminent risk. Those are not moral categories, they are specifications. And like any spec, what matters is everything that falls below the line.
Think about what an AI model actually does wrong in practice. It helps someone write a convincing phishing campaign. It spits out confidently wrong medical information. It automates a hiring screen that quietly filters out a protected class. It gets wired into a customer service stack and starts making commitments the company never authorized. None of those scenarios kill 50 people. Most of them do not cost a billion dollars, at least not in a single traceable event. They cost a lot of smaller amounts, spread across a lot of people, over a long period of time, with no clean attribution.
Under a threshold like this one, that entire category of harm sits outside the trigger. The bill is built for catastrophe. The daily failure modes I actually see in tooling are not catastrophes. They are erosion.
I am not saying catastrophe rules are pointless. Mass-casualty risk from a frontier model is worth a legal mechanism, and Anthropic proposing one is more than most of its competitors have put in writing. But a reviewer learns to notice when a safety claim is scoped to the failure that was never going to happen anyway.
The independent evaluator question
This is the part of the proposal I actually want to see developed, because it is the part that could change my job.
Right now, evaluating a model means taking the vendor’s word for a lot of things. Vendors publish their own model cards, run their own red-team exercises, and pick their own benchmarks. Independent testing exists, but it runs on whatever access the vendor decides to grant. If a bill establishes independent evaluators as a real part of the regulatory process, with actual access, that changes the information available to everyone downstream — including people like me, and including the teams deciding what to put into production.
The draft says independent evaluators should be involved. It does not, from what has been reported, say who selects them, who funds them, what access they get, or whether their findings become public. Those four details are the entire difference between meaningful oversight and a compliance checkbox. I have reviewed enough tools with “third-party audited” badges to know how much work that phrase can be doing.
What to actually watch
POLITICO’s reporting notes the draft would limit the circumstances in which the government could block a model, and would preempt some existing rules. That second piece deserves attention. A federal standard that replaces a patchwork of state rules is a real benefit for anyone building across jurisdictions. It is also, depending on where the lines land, a way to trade many small constraints for one large and distant one.
For teams picking AI tooling, none of this changes your evaluation process this quarter. A bill pitched quietly to lawmakers is a long way from a law. What it does tell you is where a major vendor thinks the floor should be, and that floor is set at mass casualty. Everything between “works as advertised” and “kills 50 people” remains your problem to test for.
So keep testing. Keep logging what the model does when it is wrong, not just when it is dangerous. And if independent evaluation does become a real requirement with real teeth, that is the development worth tracking — not the headline number, but who gets to check the work.
đź•’ Published: