Skip to content
Li ZhenXi, back to top

Open to 2027 roles in SRE, cloud and AI infra

I run cloud platforms in production and benchmark AI infrastructure. DevOps intern at Primustech, Computer Science at NTU.

169 merged PRs on a production IoT platform · 7 case studies with working models · 320 LLM agent runs benchmarked

~15%

cloud spend cut across monitoring, database and compute

169

pull requests merged into a production IoT platform

320

long-horizon LLM agent runs benchmarked for my final-year project

3.02x

throughput from running four isolated agent tasks at once

Where these numbers come from, and which ones you can check yourself

Production work you can play with.

Things I shipped at Primustech, each a small working model of the real system. Press the buttons.

What I work on, and the colour each one wears across this site:

Primustech · Kubernetes on GKE

Building IoT platform

Building IoT platformPrimustech, 2026

One VM to a GKE platform

Moved a live building-operations platform off a single Docker Compose VM and onto Kubernetes.

One VM, Docker Compose, one in-memory queue

Core APIMessage queueCoordinationMQTT brokerPostgreSQL

One failure domain, deployed by hand, and a queue that could not be shared.

Read the case
Building IoT platformPrimustech, 2026

A pipeline where failed merges never deploy

I own the Cloud Build delivery pipeline: build once, promote by digest, approve before release.

  1. PR checks
  2. Build image
  3. CVE scan
  4. Integration tests
  5. Approval
  6. Promote digest
  7. Deploy to GKE

A simulation of the real release gates.

Read the case
Building IoT platformPrimustech, 2026

Cutting cloud spend by ~15%

Measured first, then cut monitoring, database and compute, with every trade-off written down.

in use orphaned0 / 914 reaped
Read the case
Building IoT platformPrimustech, 2026

Alerts that mean something

Dashboards and alerts built around how this system actually fails.

zk-0

leader

zk-1

follower

zk-2

follower

Quorum held, 3 of 3 up

All quiet.

Read the case
Building IoT platformPrimustech, 2026

Remote control that fails safe

The platform could watch a building's air-conditioning. Letting it switch units on and off took a different kind of care.

control bridge

dry run

commands this hour 0/6

air-conditioning unit

26 °C from the plant's readback

selector auto

DRY_RUN=
24 °C
  1. Set DRY_RUN to "false" to arm the lane, then send a setpoint. Try the typo too.
Read the case
Building IoT platformPrimustech, 2026

Changing a live cluster in steps that can't break it

Security and reliability changes to a running platform, each split so that no single step can take it down, and rehearsed first.

  • coreno auth
  • rule engineno auth
  • http transportno auth
  • mqtt transportno auth

Cache

open

No password anywhere

The cache accepts any client inside the cluster. Flip a switch to start the change.

Read the case
Building IoT platform

Inside the IoT platform

The building IoT platform I run on GKE: how traffic moves through it, what I alert on and why, and where the money went.

Try being on call: a simulated incident on this platform

How internet traffic reaches the cluster: load balancer, WAF, ingress, Services, pods and the database.

  1. 1

    BrowserHTTPS 443

    One public hostname. Any other host, or the raw IP, gets a 404 by design.

  2. 2

    Global HTTPS load balanceranycast IP

    TLS ends here on a reserved global IP with a Google-managed certificate.

  3. 3

    Cloud ArmorWAF

    OWASP CRS rules plus a login throttle, tuned per route so legitimate traffic passes, with auth and rate limits behind them.

  4. 4

    GKE IngressL7 paths

    Path routing sends the device API, the REST and WebSocket API, and the web UI to separate Services.

  5. 5

    Service to podsClusterIP

    Core API pods are spread across three zones and across nodes, so losing a zone costs a third of capacity, not all of it.

  6. 6

    Core API to Cloud SQLprivate IP

    A Cloud SQL Auth Proxy sidecar connects over private service access using Workload Identity: no key files, no password on the wire.

AI agent platform

The AI agent platform I run.

Primustech's AI agent platform gives facility teams two agents: an operations agent that answers from the IoT platform's live readings, and a document assistant that answers from company documents. I deploy and run it on Google Cloud. Follow a request through it, see the AI infrastructure layer by layer, and switch on the lens to see which parts are mine.

Try being on call: a simulated incident on this platform

Operations agent

Answers facility managers from live building readings. A manager agent plans; a data-engineer agent queries the IoT platform and crunches the numbers.

Agno team on Cloud Run, Gemini on Vertex AI

Document assistant

Answers from a company's own documents, with citations: multi-stage RAG over Milvus, with a reranker on a GPU.

Cloud Run API and job, Milvus, Gemini via OpenRouter

Building data comes from Building IoT platform, the platform I run on GKE; it has a chapter of its own.

3
environments deployed the same way: dev, staging and production
2,014
BMS devices in the 50-storey demo building I built for the agents to query
AI agent platformPrimustech, 2026

Running an agentic AI platform

Infrastructure for the company's two AI products: an operations agent that answers from live building data, and a document assistant.

  1. Question

    from a facility manager

  2. Agent team

    Agno on Cloud Run

  3. Data

    live readings, documents

  4. Gemini

    Vertex AI, OpenRouter

  5. Answer

    traced with tokens and cost

  • Load and latency tests on the agent API
  • Token cost tracked per request
  • Models compared on latency and cost
Read the case
AI agent platform

The operations agent: a small team of AI agents that answers facility managers from live building data.

ENTRYSERVICESMODELS AND DATADELIVERY AND NETWORKFacility teamchat, web, appLoad balancerCloud Armor, TLSOperations agentAgno team, Cloud RunDocument assistantRAG API, Cloud RunDocument workerCloud Run JobIoT platformlive readings, GraphQLVertex AIGeminiOpenRouterGemini, Grok, embeddingsRerankerbge-reranker-v2-m3 on a GPUMilvusvector searchMongoDBsessions, metadataRedis queuedocument jobsCloud Storageuploaded filesLangfusetraces, tokens, costGitHubmergeCloud Buildone per environmentArtifact RegistryimagesSecret Managerkeys, parametersShared VPCfirewall, NAT, DNS
  • AI agent platform
  • Building IoT platform
  • no stripe: shared infrastructure

Step 1 of 8

A question

A facility manager asks something like "which floors ran warm this afternoon?"

ARIES agent benchmarks

What does an AI agent actually cost to run?

My final-year project at NTU's HyScale Lab. I got ARIES, the lab's open-source framework for agent-serving experiments, running long tasks against live apps (mail, a shop, an LMS), then measured 320 runs on a hosted API and self-hosted SGLang on an A100. Since then: what MCP tool servers cost, and the whole stack running on a supercomputer without root.

2.7%
of agent time is spent inside MCP tool calls
23%
of wall clock goes to setup, vs 2% for single-service tasks
96%
of input tokens served from the prefix cache
~7x
more A100 time held by serving jobs than the agents used

Runs per hour on one 8-vCPU VM

Mean of three blocks, bars show the range

measured if it scaled perfectly
02040601 at once2 at once4 at oncelinear1.00x1.73x3.02x
View as table
At onceAttempts / h (range)Passed / hCPUMemory
113.7 (13.5-14.1)10.715%2.6 GiB
223.8 (22.4-25.1)17.326%3.9 GiB
441.5 (36.3-45.7)2948%6.4 GiB

Where a run's wall clock goes

Inside the agent phase, MCP tool calls are only 2.7% of the time

  • 23%Environment setup
  • 72%Agent working
  • 4%Grading

Where an agent's time goes

48 runs with the agent's own spans switched on. The model's turns are most of it; MCP tool servers are 8.6% here.

  • 72.5% Model calls
  • 18.5% The harness's own tools: terminal, code, files
  • 7% MCP servers that reach the internet
  • 1.6% MCP servers backed by an application
  • 0.5% The harness in between

What one call costs, by server

Each bar runs from the median call to the 95th percentile, on a log scale. Nine of the thirteen servers, 800 of 874 calls.

filesystem
pdf-tools
excel
emails
woocommerce
canvas
arxiv search
yahoo-finance
playwright
  • does the work itself
  • calls an application beside it
  • reaches the internet

Servers that reach the internet are slow with long tails: a browser call takes 1.4 s at the median, and an arXiv search 11 s at the 95th percentile. Two ideas would save memory, not time: starting a server only at its first call adds about 1.2 s per server, roughly as much as all the MCP waiting in a run, and restoring an idle application from a snapshot takes about 1.5 s.

Why a long agent run doesn't recompute everything

Every turn re-sends the whole conversation. A prefix cache keeps what the server has already seen, so only the new tokens are computed.

  • served from cache
  • computed this turn
  • one row per turn, longest prompt on the right
Tokens sent across the run100%
Tokens actually computed6.1%

Measured, not illustrated: across 320 runs, 96% of input tokens came from the prefix cache, on a hosted API and on self-hosted SGLang. This illustration reaches 94% at 24 turns.

A GPU in one place, the agents in another

Qwen3-4B served with SGLang on an A100 at NSCC, Singapore's national supercomputer, for agents running on Google Cloud.

NSCC A100

SGLang server

Qwen3-4B

Google Cloud VM

ARIES runner

task sandboxes, live apps

The serving jobs held the A100 for about 7× as long as the agent runs used it. A reserved accelerator that sits idle is the AI version of an oversized pod.

The same stack on a supercomputer, without root

I added an Apptainer backend to ARIES so a whole run fits in one batch job on NSCC's ASPIRE 2A: the agent, its sandbox, and Qwen3-4B served by SGLang on the same A100 node. Merged 10 Nov 2026: 6,678 lines across 46 files.

what broke

ARIES started every part of a run as a Docker container, and an HPC cluster gives users neither Docker nor root.

what it does instead

The harness and the sandbox run as Apptainer instances, each with a fake root and its own overlay; the bridge runs as a process in the same namespace. Images are unpacked once into plain directories owned by the user, because with a packed image underneath, installing a package as "root" fails on files owned by other IDs.

How each run is wired: runner, agent harness, sandbox and its own app copies. Click to zoom

ICCV 2025, co-author

PVChat: Personalized Video Chat with One-Shot Learning

The first video LLM that learns a person from a single reference video, so it can answer identity-aware questions about them.

Code and paper

Where I've worked

From PLCs in a building's control room to Kubernetes in the cloud.

  1. Singapore

    Building IoT platform
    • Own the platform's Cloud Build CI/CD pipeline; 169 of my PRs merged so far.
    • Migrated it from one VM to GKE microservices with Terraform.
    • Wrote the safety pattern for remote control of air-conditioning and built its guards into the control bridge: deliberate arming, manual override, rate caps and readback from the plant.
    • Changed the live cluster in reversible steps: a password on the cache in two steps, a second replica for the command transport, and a register of single points of failure with measured or estimated recovery times.
    • Ran on-site control tests at a customer site, rehearsed first on a Modbus bench simulator I built.
    • GKE
    • Terraform
    • Cloud Build
    • Kafka
    • MQTT
    • Modbus
    • Grafana
    AI agent platform
    • Deployed and operated the agent and RAG services on Cloud Run, with models on Vertex AI and OpenRouter.
    • Set up per-environment CI/CD, IAM, Secret Manager and shared-VPC networking for dev, staging and production.
    • Benchmarked the stack: load and latency tests of the agent API, token cost per request, and models compared on latency and cost.
    • Built a demo building for the agents to query: 50 storeys, 2,014 BMS devices.
    • Cloud Run
    • Vertex AI
    • Cloud Build
    • Secret Manager
    • IAM

    Also

    • For a customer, outside both platforms: a read-only Modbus reader for 718 energy-meter channels, tested at full scale against a simulator.

Jul 2023 - Jun 2027

Nanyang Technological University

Bachelor of Computer Science

Exchange semesters at Tsinghua University and Luleå University of Technology.

Ask about my experience

A small retrieval service over everything on this site, with an optional LLM on top. It answers only from these facts, shows its sources, and shows its own OpenTelemetry trace.

Try one of these

Prefer your own assistant? This site is also an MCP server: point Claude, Cursor or any agent at it.

This page is a production service

The Go server that sent you this page also reports on itself. These numbers are live and refresh every five seconds. Below them: the site's own SLO as Google's uptime checks see it, how it ships (DORA metrics), and the trace of your own visit.

  1. Your browser
  2. Cloud Run: Go server
  3. Static site + /api
  4. /metrics for Prometheus

Latency of the last requests

Offline
Connecting to the status API...

This site's own SLO, measured from outside real data

Objective: 99.9% over 30 days

SLI: the share of Google Cloud uptime checks that get a 2xx from /api/health within 10 s. They run every minute from Singapore, Belgium and Oregon.

How this site ships, measured

DORA metrics, last 30 days, as of 12 Nov, 10:04 UTC

deploys to production, on 5 of 30 days
52
median lead time, commit to live (72 commits)
7.9 min
change failure rate: 0 of 52 deploys failed their live checks after taking traffic
0%
time to restore: no failed deploy to recover from
n/a

Before a release reaches you: 0 releases stopped before taking traffic, 0 releases held by the error-budget gate, 0 automatic rollbacks. Last rollback drill, 19 Oct, 12:05 UTC: traffic back on the old revision 17 s after the switch.

30 days agodeploys per day, Singapore timetoday

  1. 8533f74What this site costs: a live receipt at /costlead 10.5 min · deploy 1.9 min
  2. 9aacb41Merge pull request #5 from zhenxi0901/dependabot/npm_and_yarn/web/web-4b4c6033e8lead 11.1 min · deploy 1.8 min
  3. 1147391Old /#work-ai links land on the AI chapterlead 9.8 min · deploy 1.7 min
  4. 07a8dc0Re-review of the chapter split: 404, focus, chapter marks, live panellead 9.4 min · deploy 1.8 min
  5. b6c333dSplit the one-page site into chapterslead 10.1 min · deploy 1.7 min
  6. 7ed7870The on-call drills in the terminal: oncall iot, oncall ailead 8.3 min · deploy 1.8 min

Computed by the pipeline from this repository's commits and deploy runs as of the time above, before it built the image you are looking at. Each release starts with no traffic and is smoke-tested at its own URL first; a deploy counts as failed only if its revision took traffic and then failed the live checks, and a failure there rolls back on its own. Problems that did not come from a deploy are in the postmortems.

Your visit, traced

Measured by your own browser. Nothing is sent anywhere.

What broke here, written up properly.

Blameless postmortems from building and running this site: timeline, root cause, how it was caught, and what changed. The times and numbers come from the commits, the CI history and Cloud Run's request logs.

Want to be on the other side of one? Go on call for a simulated incident, and get a postmortem of your own response.

  1. Near miss1 Oct 2026

    “Message sent” for messages nobody would read

    For the site's first 3 h 32 min, the contact form said a message was sent while keeping it on a disk that Cloud Run throws away.

    3 h 32 minthe form could lose a message

    6 of 6 actions doneRead it

  2. Near miss1 Oct 2026

    The health check that only worked from inside

    Cloud Run's front end answers some paths ending in “z” itself, so /healthz returned 404 to every outside check while the app was healthy.

    0visitors affected: caught before launch

    3 of 3 actions doneRead it

  3. False alarm1 Oct 2026

    A failing check on a healthy site

    The smoke test reported /metrics as broken while Cloud Run's logs showed every /metrics request returning 200.

    43 / 43requests to /metrics answered 200 during the replay

    3 of 3 actions doneRead it

Writing

Longer notes on what the measurements showed. The numbers are the same ones the rest of this site uses.

Hiring for reliability?

I graduate in June 2027 and I'm looking for SRE, cloud, DevOps and AI infrastructure roles.

The role, the team, and how to reach you.