Remember when a chess engine beating a grandmaster was the whole story? For a few years there, the benchmark was the product. You could buy the engine, run it on your machine, and lose to it personally. That loop — claim, release, verify, adopt — is the one I’ve built a review practice around. You ship something, I run it, I tell people whether it works.
That loop just broke, and I want to be honest about what it means for anyone who evaluates tools for a living.
What actually happened
On October 6, 2026, OpenAI published 722 mathematical manuscripts on GitHub. The credit goes to an unnamed internal frontier model. Alongside the manuscripts came supporting proof artifacts, Lean formalizations, and ten abridged summaries of the model’s reasoning.
The company says it plans to responsibly release the model behind the results, and framed the whole exercise around the importance of evaluating internal models to speed up progress in mathematics and other sciences.
This follows the September 9, 2026 announcement, where OpenAI said its system solved the Navier-Stokes existence and smoothness problem after roughly 88 hours of compute using up to 10,000 coordinating AI agents, and resolved over 100 long-standing open problems within 24 days of training. Scientific American described a field already in shock. ABC News coverage of the September claim pulled over a million views.
Those are the facts I have. Everything past this point is my read on them.
The Lean files are the part I care about
Most AI announcements ask you to trust a number. This one ships formalizations. That’s a meaningful difference, and it’s the detail I’d point to if someone asked why this release isn’t just another demo reel.
Lean formalizations are machine-checkable. A proof assistant either accepts the argument or it doesn’t, and it doesn’t care who wrote it or what their valuation is. For a reviewer, that’s the closest thing to a clean test use that mathematics offers. The artifacts are on GitHub, which means the verification work doesn’t depend on OpenAI’s cooperation. Anyone with the toolchain can start checking.
Compare that to the usual pattern, where a lab reports a score on a benchmark it also helped shape, and the only available response is a shrug. Here, the homework is attached.
The part that doesn’t pass my review standard
You cannot use this model. You cannot name it. You cannot test it on your own problems, compare it against alternatives, or find out where it falls apart.
That last one matters most. The thing I spend most of my time doing in reviews is finding failure modes — the prompt shapes that break a tool, the edge cases where confident output is wrong output. With 722 published manuscripts and no model access, I can examine the successes in detail and learn nothing about the misses. How many attempts produced these 722? What did the model get wrong, or get confidently, plausibly wrong? That’s not in the release.
Ten abridged summaries of reasoning, for 722 manuscripts, is a ratio worth sitting with. It’s a window, not a door.
What this does to tool evaluation
There’s a structural shift here, and it isn’t about math specifically. Labs are starting to publish outputs from systems they don’t publish. The deliverable is the result, and the tool stays in the building.
For a site like this one, that creates an awkward gap. I can assess work products. I can’t assess a toolkit I can’t open. And “OpenAI plans to responsibly release the model” is a plan, not a release date. I’ve seen enough plans.
If this becomes the norm, reviewers end up in the position of art critics rather than product testers — describing what came out, speculating about the process. That’s a worse job and a less useful service.
How I’d approach it if you’re deciding what to believe
- Treat the Lean formalizations as the verifiable core. They’re checkable now, by people who aren’t OpenAI.
- Treat the compute and agent-count figures from September as the company’s account of its own run. Interesting, unreplicated.
- Treat “plans to release” as unscheduled. Reassess when there’s a model you can query.
- Watch what working mathematicians say over the coming months. They’re the actual review panel here, not tech press.
My honest take
This is a serious release with a genuine gap in the middle of it. Shipping proof artifacts and formalizations is the right instinct, and it’s more than most labs offer. Withholding the system that produced them means the most important question — is this reliable, or is this 722 lucky outputs from an unknown number of swings — stays open.
I’d rather have the 722 manuscripts than not have them. I’d rather have the model. Until then, what we’ve got is evidence that something impressive exists, filed in a repository, with the thing itself still behind the door.
🕒 Published: