Agents are not just longer chat sessions; they are messier infrastructure workloads.
NVIDIA’s June 12 technical post on AA-AgentPerf is useful because it names the problem clearly. Traditional inference benchmarks are not enough for agents because agent workloads are made of trajectories: reasoning calls, tool calls, observations, retries, variable context lengths, and non-deterministic paths through a task.
That means the hard question is not only “how many tokens per second?” It is “how many useful agent sessions can this system support while keeping latency inside a real service target?”
The unit is the concurrent agent
AA-AgentPerf measures how many concurrent AI agents an inference system can support while meeting service-level objectives for output speed and time-to-first-token. It uses prerecorded agentic coding trajectories with interleaved reasoning and tool use, plus simulated tool-call latency, so the benchmark is closer to the shape of real coding agents than a clean prompt-completion loop.
That is a big shift.
If agents become everyday software infrastructure, data centers need to plan around active agent capacity, not just model throughput.
One human prompt can become a long chain of model calls, searches, edits, test runs, observations, and follow-up calls. Multiply that across thousands of developers or enterprise users and the serving problem looks very different from standard chat.
Agent economics are energy economics
NVIDIA emphasizes normalization per accelerator and per megawatt.
That may sound like a hardware-company flex, but it is the right metric to watch. Agent adoption will be constrained by latency, cost, and energy. If a platform can support more concurrent agents per megawatt, it changes the operating economics of large-scale agent deployment.
In NVIDIA’s launch-day results, GB300 NVL72 is reported as supporting far more concurrent coding agents per megawatt than H200 under the tested conditions. The point is not only that a new box is faster. The point is that agent workloads require full-stack co-design: GPUs, NVLink, MoE routing, serving runtimes, KV cache movement, CPU tool-call handling, and scheduler behavior all matter together.
Agents expose the whole system.
Benchmarks are becoming product infrastructure
The benchmark itself may be as important as the result.
When a category is young, companies brag using whatever metric makes them look best. Tokens per second. Cost per million tokens. Context window. Benchmark score. Those are useful, but agents need measurements that capture the whole loop.
The questions are practical: how much concurrency the system can handle, how latency behaves under load, what happens when tool calls interrupt the model loop, and whether long-lived sessions stay responsive.
Those questions are where agent infrastructure becomes real. AA-AgentPerf is a sign that the market is starting to measure the thing buyers will actually care about: not a model in isolation, but an agent service under pressure.
That is where the next infrastructure fight lives.
Source: NVIDIA Technical Blog