Agent frameworks are easy to market and hard to compare.
Every framework can make a demo look clean. Every framework can show a loop where a model calls a tool, observes the result, and tries again. That does not tell a builder what happens when the task gets messy, the repo is real, the API surface matters, and failure has to be debugged.
That is why the ADK Arena paper is useful.
The authors evaluate Agent Development Kits through a method they call LLM-as-a-Developer. Instead of manually building every agent, they use an LLM coding agent that learns each framework API from available material, writes the agent code, and repairs it through a validation loop until tests pass.
Then they compare what happens when the developer is held constant and the framework changes.
That is a much more interesting question than “which framework has the best README?”
Framework choice changes the outcome
The paper evaluates 51 popular Python agent frameworks across 204 agent-benchmark pairs, using benchmark adapters for SWE-bench, tau2-bench, Terminal-Bench, and MCP-Atlas.
The headline is not that one framework wins.
The headline is that no single framework dominates.
Generation succeeds in 57 percent of runs. The cost to generate an agent varies by 5.6x across frameworks, from about $0.60 to $3.40 per agent in the reported setup. The best single-benchmark ADK agents resolve up to 80 percent of tasks, while the median framework resolves 32 percent.
That is a wide spread.
It means the framework is not just a wrapper around the model. The design of the API, the available abstractions, the validation path, and the way tools are expressed can all influence whether a working agent actually emerges.
For builders, that matters.
If the agent fails, the answer is not always “use a smarter model.” Sometimes the framework made the easy thing hard, hid the wrong state, encouraged the wrong decomposition, or made repair loops expensive.
Benchmarks need workflow pressure
Agent evaluation has a habit of drifting toward toy tasks.
That is understandable because toy tasks are easier to run, easier to score, and easier to explain. They are also less useful when the real product has to operate across repositories, terminals, tools, service APIs, and error recovery.
ADK Arena is closer to the kind of pressure that matters because it treats agent development as a workflow. The coding agent has to learn the framework, produce working code, handle failures, and pass validation.
That makes usability measurable.
An agent framework is not only its runtime behavior. It is also how difficult it is for another agent, or a human, to build with it correctly.
That is going to matter more as teams increasingly ask agents to create, modify, and maintain agent systems themselves.
There is no settled stack yet
The practical takeaway is simple: do not assume the agent framework market is settled.
The category is still young. The best framework for one benchmark may not be the best framework for another. Documentation helps, but the paper also finds that source code, documentation, and model knowledge can substitute for one another more than expected in some settings.
So the serious builder’s response should be measurement.
Instrument the workflow. Test your own tasks. Track cost, success rate, repair loops, failure modes, and handoff quality. Choose the framework that makes your real work easier to complete and easier to inspect.
Agent frameworks are not neutral plumbing.
They shape the work.
Source: arXiv:2606.05548