3,400 tokens per second. That’s the number Nvidia is putting on the Groq 3 LPX, the inference accelerator that came out of its $20 billion Groq acquisition and has now entered full production. For anyone who has watched an AI agent chew through a multi-step task at a pace that makes you check whether your Wi-Fi died, that figure lands somewhere between exciting and slightly unbelievable.
I review tools for a living, which means I spend a lot of time separating what a spec sheet promises from what actually shows up in your workflow. So let me split this into the part that matters right now and the part that matters later.
What’s actually shipping
Nvidia announced the Groq 3 accelerator and the Groq LPX rack at GTC 2026 as part of the Vera Rubin platform. Jensen Huang framed it as an inference-specific chip that works alongside the Rubin GPU rather than replacing it. That distinction is the most useful thing in the whole announcement. This is not a general-purpose GPU that also does inference. It’s a chip built for one job: low-latency interactions with large models.
Full production is a real milestone. It means the design is locked and the fab pipeline is moving. But full production and full availability are different animals. As of April 2026, the Groq 3 LPU sits in early-access preview. Broad cloud availability is expected in late 2026, once Nvidia and its partners finish building out LPX rack deployments.
So if you are reading this hoping to swap your inference endpoint over next week, the honest answer is no. You are waiting on rack rollouts, and rack rollouts have a way of slipping.
Why latency is the right thing to optimize
Here is where I think Nvidia read the room correctly. The generative AI buildout of the last few years has been throughput-obsessed. How many requests per second, how many concurrent users, how much batch you can stuff through a card. That made sense when the dominant workload was a human typing a prompt and reading a paragraph.
Agent systems break that assumption. An agent doesn’t send one request. It sends a chain of them, each one waiting on the last. Plan, call a tool, read the result, revise the plan, call another tool. Throughput-centric architectures struggle to balance that pattern, because the bottleneck isn’t how much work the chip can do in aggregate. It’s how fast it can turn around a single small request while the rest of the pipeline sits idle.
Every 200 milliseconds of per-step latency multiplies across a twelve-step agent run. That is the difference between a tool you use and a tool you open, start, and then go make coffee. If the LPX genuinely cuts that per-step cost, the practical effect on agent tooling is bigger than the raw token number suggests.
What I’d want to see before believing the number
3,400 tokens per second is a headline figure, and headline figures come with conditions that never make it into the headline. The questions I’d be asking a vendor:
- Which model size produced that number, and at what precision?
- Is that single-stream or aggregated across concurrent requests? For agent work, single-stream is the number that matters.
- What’s the time-to-first-token, not just the sustained rate? Agents feel slow at the start of each step, not the middle.
- What does the cost per million tokens look like against current-generation GPUs like the H100 once you factor in the rack?
None of that is in the public material yet, and I’m not going to guess at it. But those four answers determine whether this is a meaningful upgrade for the average team or a specialized part for hyperscalers with latency SLAs.
The strategic read
Spending $20 billion to acquire the company that popularized the language-processing-unit approach, then folding that silicon into your own platform as a companion to your flagship GPU, is a fairly clear statement of intent. Nvidia is treating inference as a separate market with separate physics, not as a leftover use for training hardware. Given that inference is where the actual money in deployed AI ends up, that’s a reasonable bet.
It also quietly concedes something. If a general GPU were the best answer for low-latency inference, you wouldn’t need a dedicated part for it.
My take for now
File this under promising and unavailable. The architecture targets a real problem that agent builders complain about constantly, and the fact that it’s in production rather than on a roadmap slide is worth something. But early-access preview in April with broad availability projected for late 2026 means most of us are evaluating a press release, not a product.
If you’re building agent tooling, plan your architecture so inference is a swappable layer. When LPX capacity shows up on your provider of choice, you want to be able to point at it and measure, not rewrite. That’s the practical move available today. The benchmarks can wait until there’s something to benchmark.
đź•’ Published: