Imagine buying a toolkit where the manufacturer refuses to show you the safety data sheet. You’re told the tools were tested, that experts were consulted, that everything meets standards — but you can’t see which standards, what was measured, or how the results were interpreted. You’d probably return that toolkit. And yet, that’s essentially what the White House is asking the public to accept with its new AI model evaluation framework.
What We Know
The White House has reviewed a new framework for evaluating advanced AI models with major players including OpenAI, Anthropic, and Microsoft. Three sources familiar with the discussions told Axios that the administration does not plan to publicly release this framework. The decision has drawn immediate criticism from transparency advocates and AI safety researchers who argue that secret evaluation criteria undermine the very purpose of evaluation.
This comes in the context of a broader pattern. In June 2026, Trump signed an executive order calling on AI companies to submit their models to the US government for vetting and patching. Earlier in the year, reporting from The Wall Street Journal revealed that the administration had directed the Center for AI Standards and Innovation (CAISI) to pause public reports on its AI testing work. So we have a government that wants to inspect AI models but doesn’t want the public to know how those inspections are conducted or what they find.
Why This Matters for Anyone Evaluating AI Tools
I review AI toolkits for a living. My entire job depends on transparent, reproducible evaluation criteria. When I tell you a tool works or doesn’t work, you can see my methodology. You can disagree with my benchmarks. You can replicate my tests. That’s how trust gets built.
A secret evaluation framework inverts this logic entirely. It asks us to trust outcomes without understanding process. For those of us in the business of helping developers and teams pick the right AI tools, this creates a dangerous information vacuum. If the government is quietly certifying or flagging models behind closed doors, and we can’t see the rubric, how do we contextualize our own findings? How do companies building on these models assess their own risk?
Secret Standards Aren’t Standards
Let me be direct: an evaluation framework that nobody outside a closed room can scrutinize is not a standard. It’s a checklist held by gatekeepers. Standards work because they’re shared, debated, refined through public input, and applied consistently. The moment you remove public accountability from that process, you’ve created something else — something closer to regulatory theater.
Anthropic CEO Dario Amodei has publicly stated that people still don’t grasp how close we are to AI systems that outperform any human at any cognitive task. If that assessment is even partially correct, the stakes around evaluation methodology are enormous. The public deserves to understand what “safe enough” means to the people making that determination.
Who Benefits from Opacity
When evaluation criteria stay private, the companies being evaluated benefit most. They face less public pressure because nobody outside the process can point to specific failures or gaps. The evaluators benefit too — they avoid criticism of their methodology. The only party that loses is the public, which includes developers, businesses, and researchers building on top of these systems.
From my seat as a toolkit reviewer, I’ve seen what happens when vendors self-certify or undergo opaque audits. The results almost always skew favorable. Not because of outright fraud, but because hidden frameworks tend to measure what’s convenient rather than what’s critical.
What I’d Want to See
If I were designing an evaluation framework for advanced AI models — which is essentially what I do at a smaller scale every week — I’d want it to include:
- Published evaluation criteria with version histories
- Clear definitions of what constitutes a pass or fail
- Independent third-party verification
- Public disclosure of which models were tested and general outcome categories
- A comment period for researchers and civil society before finalization
None of that requires revealing classified national security methods. You can publish a rubric without publishing every intelligence source that informed it.
My Take
I’ve called tools “baffling” before in my reviews — usually when a product makes choices that actively work against its stated purpose. This situation earns that label. An AI safety framework designed to build trust that simultaneously refuses transparency is working against its own stated goal. You cannot build public confidence through secrecy. That’s not how confidence works.
For now, those of us reviewing AI tools independently will keep doing our work in the open. Someone has to.
🕒 Published: