Running an agent benchmark on a supercomputer, without root

My final-year project measures what long agent tasks cost to run. Until now the agents ran on a cloud VM and the model on an A100 at NSCC, Singapore's national supercomputer, joined by a tunnel, with the GPU reserved for about seven times as long as the agents used it. If the whole stack ran inside one batch job on the cluster, the model and the agents would share a node. The cluster's terms are strict, though: no Docker, no root, Apptainer without setuid, cgroups version 1, and no route from compute nodes to hosted model APIs.

Containers without a daemon, or root

ARIES runs each part of a run (the agent harness, the task sandbox, the bridge between them) as a container. The new backend runs the harness and the sandbox as Apptainer instances, each with a fake root and an overlay of its own, and the bridge as a process in the same namespace. Images are unpacked once into plain directories owned by the user. With a packed image as the lower layer, installing a package as "root" fails on files that belong to other IDs; unpacked, every file is the user's, so the fake root can change any of them.

An unpacked image forgets the configuration its OCI image carried, including its working directory. So setup reads that from the packed file before unpacking it, with a small reader written from the format's reference implementation rather than a new dependency, and a run that names no working directory starts where it would have under Docker.

A shared node is a hostile network

The bridge between agent and sandbox listens on ports with no authentication. That is fine inside a private Docker network and not fine on a node other users can log in to. So the run re-executes itself inside its own user and network namespace: outbound traffic goes through slirp4netns, and nothing outside the namespace has a route to those ports.

Measuring containers that have no cgroup

The project's whole point is measurement, and ARIES reads each container's CPU and memory from its cgroup. On this cluster cgroups are one per job and owned by root. So each instance is measured from /proc instead: the CPU of the live processes in its PID namespace, plus the CPU of every finished command, which the command's wrapper reads about itself as it exits and reports back. A command shorter than one sample is still counted. Memory is the processes' proportional set size.

A measurement you cannot take the usual way is still worth taking, as long as you write down what it leaves out.

What it leaves out is in the docs: memory excludes the page cache, the launcher's own CPU outside the instance (about 0.06 s a command) is not charged, and CPU and memory limits cannot be enforced at all without cgroups version 2, so the backend refuses them unless a profile says in so many words that they are unenforced.

The hang that looked like the agent's fault

On a rerun at NSCC an agent's git cherry-pick froze, holding the repository's lock, until the task ran out of time. It looked like a model doing something silly. It was the backend. Every command started its own fake-root daemon, which exited with the command. The agent harness saves each command's environment and restores it in the next, so a later command inherited the key of a daemon that was gone. Its messages went into a queue nothing read, and when the queue filled, the command blocked.

agent  $ git cherry-pick <commit>
         blocked in msgsnd(), holding .git/index.lock
         ... until the task's time limit

The fix is one fake-root daemon per instance, started with it, whose key every command inherits; file ownership then lasts across commands, as it would on a real root filesystem. Earlier runs had passed because they sent only read-only Git commands, or none. The test that mattered was the first one that wrote.

What runs now, and what does not

Two maintainers and two review bots went through the pull request. One change that came out of the review I would keep anywhere: if a finished command's CPU cannot be recorded, the instance's later samples now fail instead of quietly undercounting. A benchmark that reports a wrong number is worse than one that reports none.