# What 320 agent runs taught me about serving LLM agents

By Li ZhenXi, 19 Oct 2026. Page: https://portfolio-emsghin2iq-as.a.run.app/writing/serving-llm-agents

> MCP tool calls were 2.7% of an agent's time, 96% of its input tokens came from the prefix cache, and the GPU sat idle most of the time it was held.

- **2.7%**: of agent time spent inside MCP tool calls
- **96%**: of input tokens served from the prefix cache
- **~7x**: more A100 time held than the agents used

For my final-year project at NTU's HyScale Lab I got ARIES, the lab's open-source framework for agent-serving experiments, running long-horizon tasks from the Toolathlon benchmark: an agent works for many turns against live apps (mail, a shop, an LMS) running in its own sandbox. Then I measured 320 runs, on a hosted model API and on Qwen3-4B served with SGLang on an A100. The question was simple: what does an agent actually cost to run, and where does the time go?

## Tool calls are cheap. Setting up the world is not.

Inside the agent phase, calls to MCP tool servers took only 2.7% of the time: the servers answer fast. That is not the same as tools being free, since the model's turns and the commands the agent runs itself are in the other 97%. The bigger surprise was outside the agent: 23% of each run's wall clock went to setting up the environment, against 2% for tasks that need a single service. A task that touches mail, a shop and an LMS has to bring all of them up and prepare them before the agent starts.

> If you want agent runs to go faster, look at environment provisioning before you look at tools.

## Every turn resends the conversation, and the cache absorbs it

An agent's prompt grows every turn: the whole history goes back to the model each time. In the runs that measured it, 96% of input tokens were served from the prefix cache, on the hosted API and on self-hosted SGLang alike. So most of a long run's prompt is work the server has already done; what costs compute each turn is the part that is new, as long as the cache keeps hitting.

That changes what matters in serving. A cache hit rate that drops, because requests land on a replica that has not seen the conversation or the cache is too small, turns cheap turns back into full recomputes.

## Four at once on one VM: 3.02x

Each run gets its own sandbox with its own copies of the apps, so runs can share a machine without seeing each other. On one 8-vCPU VM, throughput went from 13.7 runs an hour at one at a time to 23.8 at two and 41.5 at four: 1.73x and 3.02x. Successful runs scaled a little less, 2.71x at four. Along the way CPU rose from 15% to 48% and memory from 2.6 to 6.4 GiB.

## A reserved GPU that sits idle

The model ran on an A100 at NSCC, Singapore's national supercomputer, and the agents ran on a Google Cloud VM, joined by an SSH tunnel through IAP. Serving jobs reserve the GPU for their whole lifetime. They held the A100 for about seven times as long as the agent runs actually used it.

> An idle reserved accelerator is the AI version of an oversized pod: the waste is in the reservation, not the work.

## What I would measure next

- How much of the 23% setup time prepared app snapshots would remove.
- Prefix-cache hit rate across several serving replicas, with and without routing a conversation back to the same one.
- How many agent runs one GPU reservation can serve before the idle share stops dominating.

## Go further

- [The research chapter, with the charts](https://portfolio-emsghin2iq-as.a.run.app/research)
- [ARIES on GitHub](https://github.com/hyscale-lab/ARIES)
- [The Toolathlon adapter pull request](https://github.com/hyscale-lab/ARIES/pull/55)