ICCV 2025, co-author
PVChat: Personalized Video Chat with One-Shot Learning
The first video LLM that learns a person from a single reference video, so it can answer identity-aware questions about them.
Code and paperResearch: ARIES agent benchmarks
ARIES agent benchmarks
What does an AI agent actually cost to run?
My final-year project at NTU's HyScale Lab. I got ARIES, the lab's open-source framework for agent-serving experiments, running long tasks against live apps (mail, a shop, an LMS), then measured 320 runs on a hosted API and self-hosted SGLang on an A100. Since then: what MCP tool servers cost, and the whole stack running on a supercomputer without root.
- 2.7%
- of agent time is spent inside MCP tool calls
- 23%
- of wall clock goes to setup, vs 2% for single-service tasks
- 96%
- of input tokens served from the prefix cache
- ~7x
- more A100 time held by serving jobs than the agents used
Runs per hour on one 8-vCPU VM
Mean of three blocks, bars show the range
measured if it scaled perfectly
View as table
| At once | Attempts / h (range) | Passed / h | CPU | Memory |
|---|---|---|---|---|
| 1 | 13.7 (13.5-14.1) | 10.7 | 15% | 2.6 GiB |
| 2 | 23.8 (22.4-25.1) | 17.3 | 26% | 3.9 GiB |
| 4 | 41.5 (36.3-45.7) | 29 | 48% | 6.4 GiB |
Where a run's wall clock goes
Inside the agent phase, MCP tool calls are only 2.7% of the time
- 23%Environment setup
- 72%Agent working
- 4%Grading
Where an agent's time goes
48 runs with the agent's own spans switched on. The model's turns are most of it; MCP tool servers are 8.6% here.
- 72.5% Model calls
- 18.5% The harness's own tools: terminal, code, files
- 7% MCP servers that reach the internet
- 1.6% MCP servers backed by an application
- 0.5% The harness in between
What one call costs, by server
Each bar runs from the median call to the 95th percentile, on a log scale. Nine of the thirteen servers, 800 of 874 calls.
filesystem
pdf-tools
excel
emails
woocommerce
canvas
arxiv search
yahoo-finance
playwright
- does the work itself
- calls an application beside it
- reaches the internet
Servers that reach the internet are slow with long tails: a browser call takes 1.4 s at the median, and an arXiv search 11 s at the 95th percentile. Two ideas would save memory, not time: starting a server only at its first call adds about 1.2 s per server, roughly as much as all the MCP waiting in a run, and restoring an idle application from a snapshot takes about 1.5 s.
Why a long agent run doesn't recompute everything
Every turn re-sends the whole conversation. A prefix cache keeps what the server has already seen, so only the new tokens are computed.
- served from cache
- computed this turn
- one row per turn, longest prompt on the right
Tokens sent across the run100%
Tokens actually computed6.1%
Measured, not illustrated: across 320 runs, 96% of input tokens came from the prefix cache, on a hosted API and on self-hosted SGLang. This illustration reaches 94% at 24 turns.
A GPU in one place, the agents in another
Qwen3-4B served with SGLang on an A100 at NSCC, Singapore's national supercomputer, for agents running on Google Cloud.
NSCC A100
SGLang server
Qwen3-4B
Google Cloud VM
ARIES runner
task sandboxes, live apps
GPU time the serving jobs held
GPU time the agent runs used
The serving jobs held the A100 for about 7× as long as the agent runs used it. A reserved accelerator that sits idle is the AI version of an oversized pod.
The same stack on a supercomputer, without root
I added an Apptainer backend to ARIES so a whole run fits in one batch job on NSCC's ASPIRE 2A: the agent, its sandbox, and Qwen3-4B served by SGLang on the same A100 node. Merged 10 Nov 2026: 6,678 lines across 46 files.
what broke
ARIES started every part of a run as a Docker container, and an HPC cluster gives users neither Docker nor root.
what it does instead
The harness and the sandbox run as Apptainer instances, each with a fake root and its own overlay; the bridge runs as a process in the same namespace. Images are unpacked once into plain directories owned by the user, because with a packed image underneath, installing a package as "root" fails on files owned by other IDs.