This is the agentic loop. We prompt the LLM, it evaluates and starts calling tools it needs to form an answer. Then, it gets back the information from the tool calls, it re-evaluates, and then it may call more.
We’ll dig into the specific tests and what they’re doing later in this article, but it’s good to set the stage with why our benchmark is designed the way it is. We developed the test on an Nvidia DGX Spark (GB10) featuring 128 GB of unified memory, a Blackwell GPU, and the relatively weak Arm-based Grace CPU. Above, you can see the time spent running these agent loops on the GB10 with gemma4-26b-a4b . The model isn’t super important here as long as we're able to call tools, and we’re able to re-record the tests using different models as they continue to release.
The point here is the share of time spent, not the model or the specific work the agents are doing (though that does provide useful context). Each result here is a list of sequential tasks for the agent to execute against the model, and nearly all of the time is spent on the GPU. Even relatively demanding CPU tasks, like with our coding agent, represent just a small fraction of the computational time. Running one or even a few agents isn’t a problem for the CPU when we have access to a large, powerful GPU with a sufficiently large model running on it.
Looking closer at one specific set of tasks, you can see that there are plenty of tasks for the CPU to execute during the window, but those tasks happen so quickly that it barely registers in the overall time spent.
These tasks aren’t evenly spaced, either. Depending on when the agent needs to call tools, the shape of the workload can look very different. Above, for example, you can see our document agent, which is tasked with scanning and extracting information from a set of PDF files. Nearly all of the agent time is spent at the very beginning of the task, after which it’s almost pure GPU inference.
Obviously, a more powerful GPU will churn through the inference step faster, but we’d have to create a severe imbalance between the CPU and GPU for the CPU portion of a single agentic loop to create a bottleneck — at least with these common tools. Agents don’t just handle a sequential list of tasks, however. They spawn sub-agents, and if multiple sessions are running on the same system in sandboxed environments, contention for limited resources becomes a real issue.
Instead of designing the test platform in such a way that creates a severe CPU/GPU imbalance, we looked toward concurrency. The idea is to sweep across the chip with concurrent sessions running a list of tasks in a loop in their own processes. As the sweep scales up, contention across the CPU starts to appear, and minor differences in tool calls, code execution and compilation, web browsing, and the variety of other tasks our agents do pile on.
With concurrency as our north star, we can start breaking down the design of the benchmark and how it works internally. The overall design is simple. Custom agents built with LangGraph have a list of 30 tasks to execute in sequence against a llama.cpp server running gemma4-26b-a4b. Instead of going directly to the server, however, the agents go through a proxy server that records a trace.
During live recording, the agent communicates with the proxy server, which passes a live call to the llama.cpp server. The proxy server meanwhile writes every logged request to the LLM and the response from it to an overarching JSON. Tool calls bypass the server chain and are logged to their own JSON files. Finally, for each task, a correct response is generated, hashed, and stored as its own JSON.
Recording everything allows us to remove the live model entirely from the equation. The agents still run as normal, but instead of running against a live LLM, they run against that proxy, replaying the traces captured. The requests and tool calls are all happening live, but the agents are immediately handed back a response from the trace based on the live inference capture.
Starting with the request from the agent, it’s matched by session label and hash to ensure it’s accurate before the response is looked up in the master JSON. If there’s a mismatch, the run stops and is marked as invalid. If everything checks out, the response is passed back, and the agent session(s) call the tools they need. The tool calls are compared to the JSONs generated during recording, and the recorded result is passed back. This is critical.
The test compares the live tool call to the recorded result, but it always passes back the recorded result to keep the test from drifting, even if the tool call is correct. True tool mismatches are logged. The test proceeds, but it’s marked as invalid for each level with tool mismatches.
This setup makes redesigning and expanding the benchmark simple. We can write a new persona in Python using LangGraph, or expand to other frameworks like CrewAI. Additionally, we can rerecord traces against different models in the future, perhaps some with expanded capabilities. As long as we’re communicating over an OpenAI-compatible endpoint, we can mix and match pieces of our testing to represent a broader range of workloads.
The test starts at a single worker process and scales up to the maximum thread count of the CPU, running three repeats per level. Each worker process operates off a list of 30 tasks, with new worker processes starting at the next point in the list. So, worker session one would start at task one, worker session two would start at task two, and so on. Tasks do not take an equal amount of time, so this offset creates imbalanced contention as tasks overlap.
The metric we get out of the test is sessions per second. A session is a single task completed from end-to-end, and we measure the number of sessions completed per second at each level. Naturally, at lower levels, we complete fewer sessions per second, often falling below 1, meaning the task takes longer than one second to complete. The tasks are completed sequentially. What we’re looking for is how chips diverge as sessions increase, measuring the throughput of agentic work on the CPU.
We derive two headline scores from the sweep. First is peak throughput, which is the highest sessions per second we recorded, as well as the concurrency level at which it was recorded. Next is ideal throughput, which is the point at which the slowest 5% of sessions take no longer than 1.5x the time it takes for a single session to complete at the base concurrency level. We expect that tasks will take longer as the number of sessions increases, but past that 1.5x mark, the length of the task gets significantly longer, and depending on the processor’s topology, throughput can start to degrade.
It’s worth highlighting again that there is no GPU delay in this concurrency sweep. We’ve removed the GPU entirely, throwing as much concurrent work at the CPU as possible to determine where that peak throughput point lies. However, the GPU still plays a significant role, so the results should be viewed in a realistic context.
As more agents work on the CPU, they’ll compete for resources on the chip, but this balance can easily shift. If there isn’t enough GPU horsepower, agents may queue for a slot, and that’s not effectively utilizing the compute on either processor. Our testing is more than theoretical throughput from napkin math, but it’s less than a true end-to-end system. There are too many variables to account for in that setup for us to produce any useful numbers that are broadly applicable.
However, we can run the sweep with live inference to evaluate system-level performance. For live runs, the responses handed back to agents are still served from the recording to keep runs equal. However, the request is recorded by the llama.cpp server, which generates the same number of tokens before being discarded. This accounts for the normal inference pipeline, including prefill, decode, and cache performance, as well as agents queuing for a slot.
Both numbers are important. Live sweeps are a realistic look at local AI across the entire system, while frozen replays measure the peak throughput a CPU is capable of with access to all the GPU compute (theoretically) that it could want.
We’ve defined six agent personas. A “persona” is a Python script built with LangGraph that defines what the agent can do. The agents tool calls are Python functions, defined in that main script that connects our LangGraph nodes. Each persona is working off its own unique list of tasks from local documents. For example, the research persona is able to answer questions by searching in a Wikipedia corpus of about 50,000 articles.
Answer questions about a fake retail company by searching DuckDB with SQL and write a report.
List tables, describe tables, run SQL query, make chart from query
Key considerations
- Investor positioning can change fast
- Volatility remains possible near catalysts
- Macro rates and liquidity can dominate flows
Reference reading
- https://www.tomshardware.com/tech-industry/artificial-intelligence/SPONSORED_LINK_URL
- https://www.tomshardware.com/tech-industry/artificial-intelligence/our-first-step-at-defining-a-cpu-benchmark-for-agentic-ai-how-we-can-measure-throughput-of-agentic-ai-workloads-without-any-gpu-involved#main
- https://www.tomshardware.com/my-account
- Pioneers Across Europe Deploy NVIDIA Earth-2 Open Models to Power Real-World Weather and Climate Prediction
- Save $234 on 32GB of DDR5 RAM with this 3-item AM5 Newegg bundle
- TSMC reportedly weighs 20% CoWoS price increase as Nvidia Blackwell and Rubin chips put pressure on packaging capacity
- SK hynix-owned Solidigm is plotting a new $45 billion flash memory factory for the US, report claims
- SK hynix-owned Solidigm is plotting a new $45 billion flash memory factory for the US, report claims
Informational only. No financial advice. Do your own research.