Benchmarks test a sliver of actual work.
That’s the short version of a study from Mercor that I’ve been chewing on since it landed. They built something they call a Benchmark Map, overlaying 29 selected AI benchmarks onto the US Department of Labor’s O*NET taxonomy, which catalogs 923 occupations and 18,797 distinct tasks. Then they measured how much of real American economic activity those benchmarks actually touch. The answer: roughly 90% of US working hours go toward tasks with little or no benchmark coverage at all.
I review AI tools for a living. I read benchmark scores the way other people read nutrition labels, with a mix of trust and suspicion. This study is the moment the suspicion wins.
What the numbers actually say
Out of 923 occupations in the taxonomy, only 208 have even a single task backed by at least five benchmark prompts. Five prompts. That is the bar, and most jobs in the American economy can’t clear it.
The coverage that does exist isn’t spread evenly. It clusters around high-wage, AI-exposed work: software engineering, law, the kind of roles that already dominate AI discourse. Lower-wage occupations and jobs rooted in the physical world sit almost entirely unmeasured. Middle-income knowledge work and smaller job categories get thin treatment too.
So when a model posts a strong score, you’re learning something real about a narrow band of tasks that happened to interest benchmark authors. You are not learning much about the work most people get paid to do.
Why this matters for anyone picking tools
Here is the practical problem I run into constantly. A reader emails asking whether a given assistant will handle their workflow. They run a dental practice, or manage a warehouse shift schedule, or do municipal permitting. There is no benchmark for any of that. There is a benchmark for competitive programming problems and bar exam questions.
Vendors know this. Benchmark scores are marketing assets, and the incentive is to optimize against the tests that exist rather than the work that exists. That’s not fraud, it’s just how measurement works when the measurement is incomplete. But it means the gap between “scores well” and “works for you” is wider than the marketing implies, and the Mercor map gives us a rough size for that gap.
Ten percent. That’s the share of working hours where benchmark results carry real signal. For the other 90%, you are flying on vibes, demos, and whatever you can test yourself.
What I’m changing about how I review
A few adjustments I’m making, and that I’d suggest for anyone evaluating tools:
- Treat benchmark scores as a floor, not a forecast. A high score tells you the model is competent at something. It doesn’t tell you it’s competent at your something.
- Build your own small test set. Ten real tasks from your actual week, run against two or three tools, scored by you. That beats any public leaderboard for your specific decision.
- Be more skeptical of claims about non-knowledge work. If a product promises to transform operations, logistics, field service, or care work, ask what evidence exists. The study suggests the answer is often very little.
- Watch where the benchmarks are growing. New evaluations get built where attention and money go. The coverage skew isn’t random, and it tells you which industries vendors are chasing.
The uncomfortable part
There is a version of this finding that reads as reassuring. Most work isn’t measured, therefore most work isn’t automated, therefore relax. I don’t think that follows. Unmeasured doesn’t mean unaffected. It means we have no idea, which is a different and arguably worse position to be in.
The flip side is equally uncomfortable for the hype crowd. If you’ve been reading benchmark progress as a proxy for economic transformation, that proxy just got a lot shakier. Saturated benchmarks in a handful of high-wage domains are not the same thing as broad capability across the economy. Mercor’s own framing is blunt about it: this makes it very hard to know how good AI models are at the work Americans actually do.
Both the optimists and the doomers have been arguing over a scoreboard that covers a tenth of the field.
Where I land
I want more benchmarks, and specifically more boring ones. Evaluations for scheduling, intake, dispatch, documentation, the unglamorous middle of the economy. That work is harder to benchmark because it’s messier, more contextual, and less likely to generate a press release. Which is exactly why nobody’s built it.
Until then, my advice stays the same but with more conviction behind it. Run your own tests. Trust what you can reproduce on your own tasks. And read every benchmark chart with the question this study makes unavoidable: measured against what, exactly, and does it look anything like my Tuesday?
đź•’ Published: