\n\n\n\n Conference Stages Are the Worst Place to Learn What AI Tools Actually Do - AgntBox Conference Stages Are the Worst Place to Learn What AI Tools Actually Do - AgntBox \n

Conference Stages Are the Worst Place to Learn What AI Tools Actually Do

📖 5 min read•804 words•Updated Aug 30, 2026

Anthropic and OpenAI sharing a stage at TechCrunch Disrupt 2026 is going to generate a lot of coverage, and almost none of it will help you pick a tool. I say that as someone who spends most of his working hours testing these things until they break. The gap between what gets announced on a stage and what survives contact with a real codebase is the single most reliable pattern in this industry.

That is not a knock on Disrupt. Conferences are useful. They are just useful for something other than evaluation.

What a stage appearance actually tells you

When the two most-watched labs in the field agree to appear at the same event, the signal is about positioning, not product. It tells you both companies think the developer and founder audience is worth courting directly. It tells you they want to be seen as peers rather than as one leader and one challenger. Those are real signals. They are also signals about the companies, not about whether their models will handle your retrieval pipeline without hallucinating a schema.

The lineup around them makes the same point. Amjad Masad, CEO and co-founder of Replit, is joining the Disrupt Stage. Replit is a genuinely interesting case study in agentic coding because it has shipped to a large user base and absorbed the complaints that come with that. But a stage conversation with Masad will be about vision and trajectory. It will not be about the failure modes I care about, like what happens on the fourth iteration of a refactor when context starts drifting.

The demo problem

Every tool I review looks excellent for about twenty minutes. That is not deception. It is the natural result of building a demo path and polishing it. The interesting behavior lives outside that path.

Here is what I actually look for when a new model or agent framework lands, and none of it is visible from an audience seat:

  • How it behaves on the second and third attempt after a failure, not the first success
  • Whether error messages tell me what went wrong or just that something did
  • Cost per completed task, not cost per token, because retries are where budgets die
  • What breaks when I hand it a codebase with inconsistent conventions and half-finished migrations
  • Whether the documented behavior matches the observed behavior after a silent model update

You cannot get any of that from a forty-minute panel. You get it from a week of unglamorous testing and a spreadsheet.

Agents are the part worth watching

The most telling item in this news cycle is not the headliners. Runable raised $21M on the bet that AI agents can move from building businesses to growing them. That is a specific, testable claim, and it is a much harder one than it sounds.

Building is bounded. There is a spec, and either the thing exists or it does not. Growing is open-ended, involves judgment calls with delayed feedback, and punishes confident wrong answers. Every agent I have tested handles bounded tasks meaningfully better than open-ended ones. So a company raising real money on the open-ended half is either seeing something in their internal numbers that the rest of us have not, or it is pricing in optimism. I would like to know which. That question is more useful to a practitioner than any keynote.

How to use an event like this

If you are attending, or following along, the practical move is to treat announcements as a to-do list rather than a verdict. Write down the claims. Then go test them against your own workload, because your workload is the only benchmark that matters to you.

Disrupt also has a participatory side this year that is easy to overlook. Applications are open to host Side Events, and audience voting decides which sessions make it to the TechCrunch Founder Summit. Side events and audience-chosen sessions tend to be where people talk about what did not work, partly because the stakes are lower and partly because the audience is smaller. In my experience those rooms are where you learn about the rate limit that ruined someone’s launch week, which is the kind of detail that never makes it into a main-stage conversation.

My honest expectation

I expect the Anthropic and OpenAI sessions to be well-produced, quotable, and light on anything that would change how I score a tool. I expect the Replit conversation to be more concrete, because Masad has shipped agentic products at scale and that tends to make people specific. And I expect the most useful information of the whole event to come from side conversations that nobody records.

None of that makes the event pointless. It just means the stage is where the story starts. The review comes later, from people willing to run the thing until it fails.

🕒 Published:

🧰
Written by Jake Chen

Software reviewer and AI tool expert. Independently tests and benchmarks AI products. No sponsored reviews — ever.

Learn more →
Browse Topics: AI & Automation | Comparisons | Dev Tools | Infrastructure | Security & Monitoring
Scroll to Top