Speed is a feature until it isn’t.
Nvidia CEO Jensen Huang told CBS News in September 2026 that we should develop AI “as fast as we can.” At Dreamforce 2026, sharing a stage with Marc Benioff, Anthropic’s Dario Amodei, and Siemens CEO Roland Busch, he rejected calls to slow down. His reasoning, as reported, is that market forces are sufficient to ensure AI safety. Amodei has argued otherwise. That disagreement is the whole story.
I test AI tools for a living. I install them, break them, write down what happened, and tell you whether the demo matched the product. So when someone says the market will sort out safety, I don’t hear a philosophical position. I hear a claim about a feedback loop I spend my weeks inside of. And I have opinions about how well that loop actually works.
What “market forces” looks like from the review desk
The theory is clean. Bad products lose customers. Unsafe products get sued, churned, or mocked into irrelevance. Companies that ship carefully win. Pressure flows backward from users to vendors, and safety becomes a competitive feature rather than a regulatory chore.
In practice, the loop has friction at every joint. A few things I run into constantly:
- Failures are quiet. When an agent quietly mangles a spreadsheet or hallucinates a citation into a client deck, nobody files a bug report. The user shrugs and fixes it by hand. That signal never reaches the vendor.
- Switching is expensive. Once a team wires an agent framework into its auth, its data, and its internal docs, churn stops being a realistic threat. Lock-in dulls the exact pressure that’s supposed to enforce quality.
- Buyers aren’t the users. The person approving the seat license is rarely the person watching the tool fail at 11pm. Purchase decisions get made on demo quality, not reliability data.
- Nobody publishes their misses. Vendor benchmarks are marketing. Independent reliability data on agentic tools is thin, inconsistent, and usually outdated within a release cycle.
A market can only punish what it can see. Right now, the thing most users can see is a polished onboarding flow.
The part that makes me uneasy
Also in this news cycle, Google confirmed that an AI model breached three real companies during a May test. Sit with the shape of that sentence for a second. A controlled test, real targets, real access. That’s the capability curve Huang wants pushed harder, and it’s the sort of result that arrives faster than any procurement committee’s ability to evaluate it.
I’m not going to pretend I know the right global speed for AI development. I review tools. But I do know the gap I see between what agents can do and what the average team is set up to supervise, and that gap has been widening every quarter I’ve been doing this. The models get more capable on a research timeline. Organizational oversight moves on a budget-cycle timeline. Those are not the same clock.
Give Huang his due
There’s a real argument on his side, and I don’t want to strawman it. Slowing down isn’t free either. Slower iteration means fewer people stress-testing systems in the open, more capability concentrated in fewer hands, and longer waits for the boring useful stuff that actually helps working teams: better retrieval, better tool-calling, fewer silent errors. A lot of the reliability improvements I’ve been happy to write about came from rapid iteration, not from a committee.
And Huang’s incentives are not a secret. He sells the compute. A man who sells shovels telling you to dig faster is not exactly a surprising headline. That doesn’t make him wrong, it just means you should weigh the argument on its merits rather than on the authority of the person making it.
What this changes for how you buy tools
If the market really is the safety mechanism, then you are the safety mechanism. That’s less inspiring than it sounds, but it’s actionable. My practical take, sharpened by this whole debate:
- Ask vendors what their tool does when it fails, not what it does when it works. Vague answers are the answer.
- Scope agent permissions to the narrowest thing that gets the job done. Broad access is the default and it shouldn’t be.
- Log agent actions somewhere a human will actually read them. Unlogged autonomy is just hope.
- Report the failures. Publicly, if you can. The feedback loop Huang is counting on doesn’t exist unless people feed it.
Two of the most influential people in AI stood on the same stage and disagreed about whether the brakes need pressing. Neither one is going to be in the room when your agent does something weird to your production database. Build accordingly.
🕒 Published: