# Li ZhenXi: everything this site knows

> The facts behind https://portfolio-emsghin2iq-as.a.run.app, one section each. The site's Ask console and its MCP server (https://portfolio-emsghin2iq-as.a.run.app/mcp) answer from exactly this text.

## About ZhenXi

I'm Li ZhenXi, based in Singapore, and I'm open to 2027 roles in SRE, DevOps, cloud and AI infrastructure. I run cloud platforms in production and benchmark AI infrastructure. DevOps intern at Primustech, Computer Science at NTU. Email me at zli057@e.ntu.edu.sg.

## Numbers I can back up

The numbers behind my work, each from a pull-request history or a measurement: I have merged 169 pull requests into Primustech's production IoT platform, 4 into ARIES, the open-source agent-serving framework, and 21 into a robot-car project. I cut cloud spend by about 15%, benchmarked 320 LLM agent runs, and measured a 3.02x throughput from running four isolated agent tasks at once.

## One VM to a GKE platform

One VM to a GKE platform (Primustech, 2026). Moved a live building-operations platform off a single Docker Compose VM and onto Kubernetes. The IoT platform behind the company's facility-management product ran on one Docker Compose VM: one failure domain, hand-run deploys and no autoscaling. Split it into GKE microservices: a three-broker Kafka cluster, Cloud SQL PostgreSQL, EMQX MQTT and Redis. Provisioned everything with Terraform, behind Cloud Armor WAF and Google-managed TLS, with pod and node autoscaling. Migrated the production database, cut the public domain over, then turned on deletion protection and passed a restore drill. The platform now runs on autoscaling GKE with a WAF in front, and every change goes through code review. Tools used: GKE, Terraform, Kafka, Cloud SQL, EMQX, Redis, Cloud Armor.

## A pipeline where failed merges never deploy

A pipeline where failed merges never deploy (Primustech, 2026). I own the Cloud Build delivery pipeline: build once, promote by digest, approve before release. Builds and deploys were coupled, so what reached production was not always the image that had been tested, and infrastructure drifted from Terraform. Split build from deploy so a release promotes the exact image digest that passed its checks. Added pull-request, post-merge and Terraform drift gates, a CRITICAL-CVE image gate and weekly base-image rebuilds. Made releases and Terraform applies approval-gated and serialised per component, then moved the repository from Bitbucket to GitHub with full history. 169 of my pull requests merged so far. A red merge never reaches production. Tools used: Cloud Build, Artifact Registry, Artifact Analysis, GitHub, Terraform.

## Cutting cloud spend by ~15%

Cutting cloud spend by ~15% (Primustech, 2026). Measured first, then cut monitoring, database and compute, with every trade-off written down. Nobody had measured where the money went. Managed Prometheus alone was about US$106 a month, the third-largest line item, and climbing with no matching growth in traffic. Traced the metric bill to 914 orphaned Kafka topics, 86% of all topics, and shipped a scheduled reaper. Samples per scrape from Kafka fell from 10,386 to 2,891. Dropped three metric packages that no alert or dashboard used, a 40% cut in samples per scrape, and kept cAdvisor because three memory alerts depend on it. Right-sized Cloud SQL from 4 vCPU/16 GiB to 2 vCPU/12 GiB and moved it from regional HA to zonal, with the recovery trade-off documented. Cut Core API pod requests from 1 CPU/4 GiB to 0.5 CPU/3 GiB so pods pack onto fewer nodes, and retired a replaced ingestion lane end to end. Cut cloud spend by about 15%, without dropping a single alert. Tools used: Managed Prometheus, Kafka, Kubernetes CronJob, Cloud SQL, GKE autoscaling, Terraform.

## Alerts that mean something

Alerts that mean something (Primustech, 2026). Dashboards and alerts built around how this system actually fails. The cluster could lose ZooKeeper quorum, stall consumers or silently drop telemetry without anyone being told. Versioned Grafana SRE dashboards and Cloud Monitoring alerts for ZooKeeper quorum loss, stalled consumers and crash loops. Chose signals by failure mode: absence alerts for ingestion, silent-loss detection, consumer lag checked against throughput, HPA headroom, and Kafka memory on RSS rather than working set. Fixed a ZooKeeper leader-election livelock that appeared after a node was replaced, and kept every alert and dashboard in Terraform with zero drift. Failures that used to be silent now raise an alert, and the metrics bill went down, not up. Tools used: Grafana, Prometheus, PromQL, Cloud Monitoring, ZooKeeper.

## Remote control that fails safe

Remote control that fails safe (Primustech, 2026). The platform could watch a building's air-conditioning. Letting it switch units on and off took a different kind of care. A wrong command to a compressor is not like a wrong row in a table: it wears real equipment, or leaves a building without cooling. The first control lanes had grown rule by rule, and nothing wrote down what must hold before a lane may command real equipment. Wrote the control safety pattern each lane is measured against: ten principles, each with the finding it came from, how the platform enforces it and what the site must provide, plus a commissioning checklist and a record kept per site. Made arming deliberate. A lane commands equipment only when its dry-run setting is exactly "false"; before, any value other than "true" armed it, so a typo would have started it live. Now anything else stops the bridge at startup. Made the site's manual selector win: while a unit is on manual the platform sends nothing, retries nothing and raises no false alarm. By design, control goes back to the schedule when the selector returns to auto. Capped commands per unit per hour, held setpoints to 20-30 °C, and designed every command to be confirmed from the plant's own readback instead of the command's echo. A command that does not take gets three attempts, two minutes apart, then an alarm. Built a Modbus bench simulator, so the whole path is rehearsed before a real unit is touched, and gave each lane its own alert budget. Three control lanes are measured against one written pattern, and the bridge that drives them has 431 tests. The pattern's rule: whatever goes wrong, the equipment ends up in the site's agreed safe state, under local control. Tools used: MQTT, Modbus, Python, Kubernetes, Terraform, Cloud Monitoring.

## Changing a live cluster in steps that can't break it

Changing a live cluster in steps that can't break it (Primustech, 2026). Security and reliability changes to a running platform, each split so that no single step can take it down, and rehearsed first. The platform carried live telemetry and control traffic. Its cache accepted clients without a password, one transport pod held every gateway session, and nobody had written down what would be lost when a node went away. Every fix was a change to a system that could not stop. Gave the cache a password in two steps. First every client sends it, which the server accepts while it still requires none; then the server requires it. Neither step can break the cache, and either can be undone alone. Rehearsed a default-deny network policy before any cutover, and found the obvious rollout was not reversible: the allow rules alone closed 28 of the 46 paths that should close only at cutover, and deleting the deny rule afterwards left 43 closed. So the cutover plan gives each namespace a transition policy to fall back to, and its own window. Recorded all 16 single points of failure, each with its blast radius, a recovery time marked measured or estimated, and an owner wherever one could be named. A node upgrade had taken lanes out for between 9 seconds and 3 minutes 18 seconds, and losing the one transport pod left 88 seconds with no route for commands. Ran that transport at two replicas behind a disruption budget. In the next live node upgrades a moved session was back in 5 to 16 seconds instead of 88; on the test rig it takes 1 to 2. Set freshness objectives per data lane from 30 days of measured cadence, separating "the site is late" from "we are behind", with alerts, runbooks and synthetic cases that proved the notifications arrive. Every change is planned as steps that can be checked and undone on their own, and the platform now has a register of what breaks, for how long, and who decides. Tools used: GKE, Calico NetworkPolicy, Valkey, Terraform, Secret Manager, Cloud Monitoring.

## Running an agentic AI platform

Running an agentic AI platform (Primustech, 2026). Infrastructure for the company's two AI products: an operations agent that answers from live building data, and a document assistant. The agent and RAG services needed repeatable deploys, locked-down secrets and numbers on speed and cost. Deployed and operated the multi-agent API and the RAG services on Cloud Run, with models on Vertex AI and OpenRouter. Set up per-environment Cloud Build CI/CD, IAM service accounts, Secret Manager and shared-VPC networking for dev, staging and production. Benchmarked the stack: load and latency tests of the agent API, per-request token cost tracking, and model comparisons on latency and cost. Three environments deployed the same way, with building telemetry from the IoT platform feeding the agents. Tools used: Cloud Run, Vertex AI, Cloud Build, Secret Manager, IAM, VPC.

## DevOps Engineer Intern at Primustech: Building IoT platform

DevOps Engineer Intern at Primustech, May 2026 - now, on the Building IoT platform. Own the platform's Cloud Build CI/CD pipeline; 169 of my PRs merged so far. Migrated it from one VM to GKE microservices with Terraform. Wrote the safety pattern for remote control of air-conditioning and built its guards into the control bridge: deliberate arming, manual override, rate caps and readback from the plant. Changed the live cluster in reversible steps: a password on the cache in two steps, a second replica for the command transport, and a register of single points of failure with measured or estimated recovery times. Ran on-site control tests at a customer site, rehearsed first on a Modbus bench simulator I built. Based in Singapore. Tools used: GKE, Terraform, Cloud Build, Kafka, MQTT, Modbus, Grafana.

## DevOps Engineer Intern at Primustech: AI agent platform

DevOps Engineer Intern at Primustech, May 2026 - now, on the AI agent platform. Deployed and operated the agent and RAG services on Cloud Run, with models on Vertex AI and OpenRouter. Set up per-environment CI/CD, IAM, Secret Manager and shared-VPC networking for dev, staging and production. Benchmarked the stack: load and latency tests of the agent API, token cost per request, and models compared on latency and cost. Built a demo building for the agents to query: 50 storeys, 2,014 BMS devices. Based in Singapore. Tools used: Cloud Run, Vertex AI, Cloud Build, Secret Manager, IAM.

## DevOps Engineer Intern at Primustech

DevOps Engineer Intern at Primustech, May 2026 - now. For a customer, outside both platforms: a read-only Modbus reader for 718 energy-meter channels, tested at full scale against a simulator. Based in Singapore.

## Final-year research at NTU HyScale Lab: ARIES agent benchmarks

Final-year research at NTU HyScale Lab, Aug 2026 - now, on the ARIES agent benchmarks. Contributed a Go benchmark adapter to ARIES, the lab's open-source framework for agent-serving research. Added an Apptainer backend, so the whole agent stack runs without root on Singapore's national supercomputer (6,678 lines, merged Nov 2026). Benchmarked 320 LLM agent runs on a hosted API and self-hosted SGLang on an A100, then profiled what MCP tool servers cost across 48 instrumented runs. Found and fixed a Docker race that dropped runs under load. Based in Singapore. Tools used: Go, Docker, Apptainer, SGLang, MCP, NSCC HPC.

## Robot car: Android app and Raspberry Pi integration at NTU Multidisciplinary Design Project

Robot car: Android app and Raspberry Pi integration at NTU Multidisciplinary Design Project, Aug 2026 - Oct 2026. Built the Android tablet controller, in Kotlin, for an autonomous robot car: an STM32 drives it, a Raspberry Pi links it to the tablet, and a laptop plans the route and recognises targets in the Pi's camera feed. 21 merged pull requests. Made the tablet and the Pi agree on the map (the Pi kept planning for obstacles the tablet had cleared), and removed a planning state the app could never leave. Based in Singapore. Tools used: Kotlin, Android, Raspberry Pi, Python, STM32.

## Control Systems Intern at ENGIE South East Asia

Control Systems Intern at ENGIE South East Asia, May 2025 - Jan 2026. Configured BMS-room networks and firewalls for multi-building projects across sites. Mapped and validated 800+ BACnet/OPC points from DDCs and PLCs into SCADA. Programmed Siemens S7-1200 PLC logic and linked site data to central monitoring. Based in Singapore. Tools used: SCADA, BACnet, OPC, PLC, Networking.

## Research Intern at Shanghai Jiao Tong University

Research Intern at Shanghai Jiao Tong University, May 2024 - Aug 2024. Cut NeRF rendering time by 40% and raised image fidelity by 30% with CUDA and PyTorch. Based in Shanghai. Tools used: CUDA, PyTorch, NeRF.

## Someone opens the dashboard

How internet traffic reaches the cluster: load balancer, WAF, ingress, Services, pods and the database. How internet traffic reaches the cluster: load balancer, WAF, ingress, Services, pods and the database. Browser (HTTPS 443): One public hostname. Any other host, or the raw IP, gets a 404 by design. Global HTTPS load balancer (anycast IP): TLS ends here on a reserved global IP with a Google-managed certificate. Cloud Armor (WAF): OWASP CRS rules plus a login throttle, tuned per route so legitimate traffic passes, with auth and rate limits behind them. GKE Ingress (L7 paths): Path routing sends the device API, the REST and WebSocket API, and the web UI to separate Services. Service to pods (ClusterIP): Core API pods are spread across three zones and across nodes, so losing a zone costs a third of capacity, not all of it. Core API to Cloud SQL (private IP): A Cloud SQL Auth Proxy sidecar connects over private service access using Workload Identity: no key files, no password on the wire.

## A site gateway sends telemetry

How device telemetry reaches the cluster over MQTT and lands in the database exactly once. How device telemetry reaches the cluster over MQTT and lands in the database exactly once. Site gateway (MQTT/TLS 8883): Dials out to a public TCP load balancer. Site IPs change, so identity is the boundary: TLS, one credential per gateway, a topic ACL per site, default deny. MQTT broker (own node pool): EMQX runs on a tainted node pool of its own, so ingestion and the application cannot starve each other. Bridge (QoS 1): Holds a durable subscription and acknowledges a message only after the write succeeds, so a crash replays instead of losing data. Device API (Service DNS): In-cluster calls go by Service name; no internal load balancer is needed. Kafka (RF 3): Three brokers with replication factor 3. Rule-engine pods split the partitions, coordinated through ZooKeeper, so each message is processed once. Cloud SQL (idempotent): Time series are keyed by entity, key and timestamp, so a redelivered message overwrites instead of duplicating.

## Pulling from a partner's broker

How private cluster nodes reach a broker outside the network through Cloud NAT. How private cluster nodes reach a broker outside the network through Cloud NAT. Private nodes (no public IP): Cluster nodes have no external addresses at all. Cloud NAT (egress): Outbound connections leave through Cloud NAT on one fixed address the partner can allowlist. Partner broker (MQTT/TLS 8883): Exactly one bridge replica on purpose: it uses a fixed client ID, and two replicas would fight over the session. Same write path (device API): From here it joins the device API, Kafka and Cloud SQL path above.

## A command goes to HVAC plant

How a start or stop command reaches HVAC equipment safely. How a start or stop command reaches HVAC equipment safely. Operator action (server RPC): Starting or stopping plant becomes a server-side RPC inside the platform. Device MQTT transport (ClusterIP only): Never exposed outside the cluster. It can push the RPC because the control bridge holds a persistent MQTT session; request-response HTTP structurally cannot push. Control bridge (Gateway API): Resolves the target through a curated point map instead of guessing names, then validates, rate-limits and logs, behind a dry-run switch and a kill switch. Control broker (internal only): A dedicated broker with no external load balancer, kept separate from telemetry brokers for fault isolation.

## Tunnels into private places

How engineers and devices reach private networks through tunnels: IAP, a WireGuard overlay and reverse SSH. How engineers and devices reach private networks through tunnels: IAP, a WireGuard overlay and reverse SSH. Engineer to a private VM (IAP TCP): No public SSH port. The firewall admits only Google's IAP range to tagged VMs, and every connection is checked against IAM. Cloud VM to an on-site controller (WireGuard): A NetBird overlay: both ends dial out and nothing opens inbound. Explicit access groups replace the default all-to-all rule. Two layers (isolation): That VM sits in its own subnet with egress to the cluster denied. WireGuard is encapsulated UDP, which the VPC firewall cannot inspect, so the overlay enforces its own policy as a second layer. GPU to benchmark VM (reverse SSH): For my FYP, SGLang on an NSCC A100 reached a GCP VM through a reconnecting SSH tunnel and a reverse forward over IAP, exposed locally with socat.

## What I monitor and why

The signals I alert on, chosen by failure mode: Is data still arriving? I watch rate(bridge_writes) ~ 0 for 10 min and missing data counts as breaching. A dead bridge emits nothing at all, so silence itself has to page. Is anything being lost silently? I watch rate(bridge_zero_device_parses) > 0. Failed writes are never acknowledged, so the broker replays them. A message that parses to nothing is acknowledged, so nothing will ever replay it. That one pages. Is the broker healthy? I watch emqx_messages_dropped > 0 and emqx_durable_subscriptions_count < 1 and up == 0, grouped by job. Grouping by job means a new broker inherits every alert without anyone editing them. Is the queue keeping up? I watch sum by (topic) (kafka_consumergroup_lag{topic!~".*notifications.*"}) and under-replicated partitions > 0. Lag with matching throughput is in-flight depth, not a backlog. Check throughput before scaling anything. Are we near a ceiling? I watch HPA headroom = max - current replicas and pods Pending > 15 min and memory working set / request. ScalingLimited also fires at the floor, and the HPA scales on CPU only, so headroom and memory need their own signals. Is memory pressure real? I watch container_memory_rss{container="kafka"}. Working set includes page cache, which Kafka fills by design. The old alert fired on healthy brokers: 84% working set, 58% actually used, no restarts in 27 days. Is the database the wall? I watch Cloud SQL CPU > 80% and connections > 80% of max and disk > 80% and deadlocks > 0. The database is the one tier that cannot scale out, so it is watched closest to its limits. Can users actually reach it? I watch uptime check failing from 2+ regions and rate(http 5xx) on the device API. Pods can be Ready while users get errors, so the check runs from outside the cluster. 28 alert policies and a 40-tile dashboard, all in Terraform with zero drift. Grafana SRE pages: overview, ingestion, dependencies, capacity, changes, observability, SLO. Managed Prometheus via PodMonitoring, plus log-based metrics where there is nothing to scrape.

## Cost review

How I cut cloud spend by about 15%: Measure before cutting (Visibility, done): Managed Prometheus was about US$106 a month, the third-largest line item, and had never been measured. I attributed it per exporter before touching anything.  Reap orphaned Kafka topics (Monitoring, done): 914 of 1,066 topics named pods that no longer existed, each emitting 7 metric families. A scheduled reaper keeps it clean; broker memory fell from 2.4 to 1.8 GiB too. Topics regrew to 191 in five days, which is why it is a CronJob, not a one-off. Drop metrics nobody reads (Monitoring, done): Turned off three GKE metric packages after confirming 0 of the 27 alerts then in place and 0 of 40 dashboard tiles used them: a 40% cut in samples per scrape. Kept cAdvisor, the biggest remaining source, because three memory alerts depend on it. Right-size Cloud SQL (Database, done): 4 vCPU/16 GiB down to 2 vCPU/12 GiB from measured utilization, and regional HA moved to zonal. Trade-off on record: no automatic failover, so recovery is 7 backups plus 7-day point-in-time recovery, and reverting is one line of Terraform. Size pods to need, scale nodes to load (Compute, done): Core API requests cut from 1 CPU/4 GiB to 0.5 CPU/3 GiB so pods pack tighter. The HPA and cluster autoscaler keep 3 nodes at the floor and add up to 9 only under load. Consolidate nodes only on seven days of working set against requests. CPU alone is not evidence. Retire a replaced lane end to end (Decommission, done): After ingestion moved in-cluster, removed the old Pub/Sub topic, Cloud Run importer, Firestore database, bucket and two internal load balancers. Removed only after the new lane carried production traffic. Unused VPN and a smaller SQL tier (Next, in review): An unused Classic VPN gateway (five resources, zero tunnels) and a Cloud SQL memory step down, worth about US$29 a month. Each goes through a reviewed, saved Terraform plan before it is applied.

## Products I work on

The products I run and build: Building IoT platform, Primustech (Kubernetes on GKE): Collects live data from building systems (sensors, HVAC, meters) and serves the dashboards and alarms facility teams use. AI agent platform, Primustech (Agents and RAG on Cloud Run): Two AI agents for facility teams: one answers from the IoT platform's live data, the other from company documents. ARIES agent benchmarks, NTU HyScale Lab (SGLang on an A100, Google Cloud VM): Making the lab's open-source agent-serving framework run long agent tasks, then measuring what serving them costs.

## SLIs, SLOs and error budgets for the IoT platform

How I monitor the IoT platform the SRE way: I measure SLIs from what users feel, set SLOs with an error budget for each, and alert when a budget burns too fast. SLI (Service level indicator): A measure of what users experience, written as good events over valid events. The share of outside uptime probes that get an answer. SLO (Service level objective): The target for an SLI over a window. Whatever it leaves over is the error budget, to spend on change. 99.9% over 30 days: about 43 minutes of budget a month. SLA (Service level agreement): The promise made to customers, with consequences if it is broken. It is set by the business. SLOs sit tighter than any SLA, so the team acts before a promise is at risk. Proposed SLOs: Reachable from outside, 99.9% over 30 days, measured as probes answered without an error over all probes. Pods can be Ready while users get errors, so this SLI is measured from outside the cluster. Telemetry kept, 99.9% over 30 days, measured as messages written to the database over messages the broker received. A message lost silently is worse than an outage: nothing will ever replay it. Telemetry fresh, 99.5% over 30 days, measured as site-minutes with a fresh reading over site-minutes a site is expected to publish. Dashboards and alarms are only as good as the newest reading. Freshness catches a stalled lane that error counts miss. Burn-rate alerts: Fast burn above 14.4x over 1 h and 5 min means 2% of the budget in an hour, so it is a page. Slow burn above 6x over 6 h and 30 min means 5% of the budget in six hours, so it is a page. Steady leak above 1x over 3 days and 6 h means 10% of the budget in three days, so it is a ticket. Error budget policy: Budget left: Ship as normal. The budget is there to be spent on change. Burning fast: Page. Mitigate first (roll back, shift traffic), investigate second. Budget spent: Risky changes wait, reliability work goes first, and the incident gets a blameless postmortem.

## How I investigate an incident

Seven Grafana pages, each starting from a question: Overview: Are users or telemetry affected? Can we still scrape? Narrow using requests, errors and duration, queue trend, node memory. Ingestion: Which lane stopped advancing? Narrow using broker receives and drops, then parser, transport, queue, persistence. Dependencies: A shared dependency, or one service? Narrow using DB pool waiting, heap and RSS, Kafka, SQL CPU, connections and disk. Capacity: Resource pressure, or a scheduling constraint? Narrow using usage against requests, throttling, Pending pods, HPA and node spread. Changes: What happened just before? Narrow using GitHub, Cloud Build and image digest evidence. Observability: Can the signal itself be trusted? Narrow using datasource auth, scrape failures, data age and gaps, ingestion volume. SLO: Are the agreed objectives met? Narrow using objectives first: no invented reliability score. Rules I monitor by: No data is unknown, never zero. A mean latency is not p99. Pod Ready is not telemetry persisted. Standing Kafka lag is not automatically a backlog. A high working set is not necessarily a leak. Runbook for late telemetry: Confirm the time range, and that monitoring itself is fresh. If Grafana and Cloud Monitoring both lack data, check the collector first. Find whether one lane or a shared dependency is affected: compare the publishers' expected cadence with what the broker received. Check parser outcomes apart from write errors: messages that parse to nothing lose data while every pod stays Ready. Check Kafka consumer groups, consumer errors and database waits, and line them up with recent deploys. A correlation is a hypothesis, not a cause. Record the incident: impact, evidence, hypothesis, mitigation or rollback, and recovery confirmed over several publish cycles.

## The AI infrastructure stack I run

The AI agent platform, layer by layer: Models (Gemini on Vertex AI, OpenRouter: Gemini, Grok, Gemini embeddings): Managed models behind two gateways: Vertex AI for the operations agent, OpenRouter for multi-provider access and the document assistant. My part: Service-account access to Vertex AI; provider keys kept in Secret Manager. GPU inference (bge-reranker-v2-m3, Infinity server): A cross-encoder reranker on its own accelerator. It re-scores retrieved chunks against the question, so only the best reach the model. Retrieval (Milvus, MongoDB, LLM chunking, Gemini vision for images): Documents become Markdown, then meaningful chunks, then vectors in Milvus. Metadata and conversation history sit beside them in MongoDB. Orchestration (Agno agent teams, Redis queue, Cloud Run Jobs, Cloud Scheduler): Agents answer on the request path. Heavy document processing goes onto a queue for a scheduled job, so uploads never block an answer. Serving (Cloud Run services, Cloud Run Jobs, dev, staging, production): The agent APIs scale as Cloud Run services; the document worker, a much larger image, runs as a job. My part: I deploy and run both, in three environments. Observability (Langfuse, self-hosted, OpenTelemetry, tokens, cost, tenant per request): Every request is a trace: each model call and tool call, with its latency, tokens and cost. My part: I used these traces to benchmark latency and cost, and to compare models. Delivery and security (Cloud Build, Artifact Registry, Secret Manager, Parameter Manager, shared VPC, Cloud Armor): A pipeline per environment, no keys in images, private networking, and one front door. My part: Mine, end to end.

## The AI agent platform I run

Primustech's AI agent platform gives facility teams two agents: an operations agent that answers from the IoT platform's live readings, and a document assistant that answers from company documents. I deploy and run it on Google Cloud. The operations agent is an Agno team on Cloud Run using Gemini on Vertex AI, with Grok through OpenRouter; it pulls live readings from the IoT platform over GraphQL and aggregates them in DuckDB. The document assistant is a RAG service: a Cloud Run API, a Redis queue and a Cloud Run Job worker that turn documents into Gemini embeddings in Milvus, then a bge-reranker-v2-m3 cross-encoder on a GPU and Gemini answers with citations. Langfuse traces every request with its tokens and cost. My part: deploying and running the agent APIs and the worker across dev, staging and production, the per-environment Cloud Build pipelines, service accounts, Secret Manager and the shared VPC, the IoT platform as the building-data source, and benchmarking latency, token cost and models.

## Ask about the building

The operations agent: a small team of AI agents that answers facility managers from live building data. A question: A facility manager asks something like "which floors ran warm this afternoon?" One way in: Public traffic enters through one shared load balancer with Cloud Armor, routed by path to the right environment. An agent team: A FastAPI service running an Agno team on Cloud Run: a manager agent plans the answer and hands the data work to a data-engineer agent. I deploy and run it in dev, staging and production. Live readings: The data-engineer agent pulls occupancy, temperature, air quality and water readings over GraphQL, from the building IoT platform. The IoT platform is the GKE system I run; it has a chapter of its own. SQL, not a giant prompt: Readings are loaded into in-memory DuckDB and aggregated with SQL, so the model reads a summary rather than thousands of raw points. The answer: Gemini on Vertex AI writes the reply. Grok models are also available through OpenRouter. Memory: The conversation is saved, so a follow-up question keeps its context. A trace per request: Every model call is traced with its tokens and its cost. I used these per-request numbers to compare models on latency and cost.

## Add a document

The document assistant's pipeline: upload a PDF, spreadsheet, deck or image, and it becomes searchable for RAG. An upload: Someone adds a document to a knowledge base: PDF, Word, PowerPoint, Excel, CSV or an image. One way in: Same front door: the shared load balancer and Cloud Armor. Accept, don't wait: The API stores the file and queues a job, then answers at once instead of holding the request open while it processes. I deploy the API and the worker as separate services. The original: The file itself lands in Cloud Storage. A job: A processing job goes onto a Redis queue, with progress the UI can show. Heavy lifting, elsewhere: A much larger image, with the document parsers, runs as a Cloud Run Job on a schedule and drains the queue. Keeping it out of the API keeps the API small and quick to start. Read, chunk, embed: Files become Markdown, images are described by Gemini's vision model, an LLM splits the text into meaningful chunks, and each chunk is embedded. Searchable: The vectors go to Milvus and the document's metadata to MongoDB.

## Ask the documents

The document assistant's RAG, in stages: fetch broadly, rerank precisely, answer with sources. A question: Say, a construction rate from a schedule of rates, or an HR policy. One way in: The shared load balancer and Cloud Armor. The right specialist: An agent is chosen by document category: one for HR documents, one for everything else. Fetch broadly: The question is embedded and the closest chunks come back from Milvus. Rerank precisely: A cross-encoder, bge-reranker-v2-m3 served on a GPU, scores each candidate against the question, so only the best reach the model. Answer with sources: Gemini writes the answer and cites the documents it came from. Memory: The last ten turns are kept for follow-up questions. Accounted for: Tokens, cost and tenant are recorded for every request.

## Ship a release

How a change reaches dev, staging and production. This path is my work. Merge: A change is merged on GitHub. Its own pipeline: The environment's Cloud Build trigger fires. Dev, staging and production each have their own project, GitHub connection and trigger, so a dev build holds no rights in production. I set these up per environment. An image: The image is pushed to that environment's Artifact Registry. Config from outside: Keys come from Secret Manager and settings from Parameter Manager, never baked into the image. Roll out: Both assistants' APIs roll out as new Cloud Run revisions, and the document worker's job is updated.

## Lock it down

Identity, secrets and the network around every environment. This path is my work. One network: A shared VPC host project owns the network for every environment: subnets, firewall rules, Cloud NAT, DNS and peering live in one place. One front door: Users come in through the shared load balancer: Google-managed certificates, path routing to each environment, and Cloud Armor in front. Private by default: Cloud Run services send all outbound traffic into the shared VPC, which is how they reach services on the private network. No keys in code: Each service runs as its own service account and reads its API keys from Secret Manager. Even the tooling: The self-hosted Langfuse VM sits behind its own Cloud Armor policy.

## How I benchmark and watch an LLM service

What I measure on an LLM service, and why: Is it fast enough to talk to? I measure end-to-end latency p50 / p95, time per agent step, model vs tool vs data time. An agent answer is several model calls and tool calls in a row. The trace shows which step is slow, so the fix lands in the right place. What does one answer cost? I measure input / output tokens per request, cost per request, cost per tenant. Cost scales with tokens, not requests. Prompt size and model choice move the bill far more than traffic does. Which model for which job? I measure latency per model, cost per model, same questions, side by side. Run the same questions through each model and compare. The cheapest model that answers well is the right one for that step. Does it hold up under load? I measure latency vs concurrency, error rate, provider 429s, Cloud Run instances. Load tests find the knee before users do. A provider's rate limit fails differently from running out of your own capacity.

## Benchmarking long-horizon LLM agents

My final-year project at NTU HyScale Lab benchmarks long-horizon LLM agent workloads. I contributed a Go benchmark adapter to ARIES, the lab's open-source agent-serving framework, so Toolathlon tasks (108 tasks, 34 MCP servers) run with live applications in isolated containers.Findings: 2.7% of agent time is spent inside MCP tool calls; 23% of wall clock goes to setup, vs 2% for single-service tasks; 96% of input tokens served from the prefix cache; ~7x more A100 time held by serving jobs than the agents used. Running 4 tasks at once gave 41.5 attempts per hour against 13.7 at 1, a 3.02x throughput, using 6.4 GiB of memory. Self-hosted Qwen3-4B was served with SGLang on an NVIDIA A100 at NSCC, reached from the Google Cloud benchmark VM over an SSH tunnel through IAP. I also found and fixed a Docker exec race condition in ARIES that dropped runs under load.

## ARIES on a supercomputer without root

I added an Apptainer backend to ARIES so the whole agent stack runs on an HPC cluster, where users have neither Docker nor root. Merged into ARIES on 10 Nov 2026: 6,678 lines across 46 files. On NSCC's ASPIRE 2A supercomputer, the agent, its sandbox and Qwen3-4B served by SGLang ran in one batch job on the same A100 node. The harness and the sandbox are Apptainer (Singularity) instances with a fake root, and the bridge is a process beside them; the run has its own user and network namespace through slirp4netns, so other users of a shared node cannot reach it; and because the cluster has no cgroup per container, CPU and memory are measured from /proc. One hang traced to a fake-root daemon per command, fixed with one per instance. Portable agent plugins (skills and MCP servers) are in review next.

## What MCP tool servers cost an agent

I profiled MCP tool servers across 48 instrumented agent runs. Model calls are 72.5% of the agent's time and the harness's own tools 18.5%; MCP calls are 8.6% overall, 13% in tasks whose servers reach the internet and 3.4% in the pool tasks, which are mostly backed by applications. Local MCP servers answer in 2 to 15 ms, application servers in 6 to 191 ms, and servers that reach the internet in 0.2 to 1.8 s at the median with long tails. Two ideas would save memory rather than time: starting a server lazily at its first call adds about 1.2 s per server, roughly as much as all the MCP waiting in a run, and restoring an idle application from a snapshot takes about 1.5 s. Neither promises much on this workload. The expensive server is the Playwright browser: 437 MiB and 56 CPU-seconds a run.

## PVChat: Personalized Video Chat with One-Shot Learning

Co-author of PVChat: Personalized Video Chat with One-Shot Learning, published at ICCV 2025. The first video LLM that learns a person from a single reference video, so it can answer identity-aware questions about them.

## Education

Bachelor of Computer Science at Nanyang Technological University, Jul 2023 - Jun 2027. Exchange semesters at Tsinghua University and Luleå University of Technology.

## Skills

Cloud: Google Cloud (GKE, Cloud Run, Cloud Build, Cloud SQL, IAM), AWS (EKS, EC2), Kubernetes, Docker, Terraform, Pulumi, Linux. CI/CD: GitHub Actions, Jenkins, Cloud Build. Observability: Prometheus, Grafana, PromQL, Cloud Monitoring, alerting, load testing. AI infrastructure: LLM serving with SGLang, LLM benchmarking, A100 GPUs, Vertex AI, RAG, MCP, PyTorch. Data: Kafka, ZooKeeper, PostgreSQL, Redis, MQTT. Languages: Python, Java, Bash, Go, C and C++. Industrial: BMS, SCADA, BACnet, OPC, PLCs.

## How this site is built

This site is a Next.js static export served by a small Go server on Cloud Run, which also runs the API behind the Ask console, the contact form and the live status panel. It exposes Prometheus metrics; GitHub Actions builds and deploys it, and its infrastructure is in Terraform. Its own SLO is 99.9% availability over 30 days, measured from outside: Google Cloud uptime checks request /api/health every minute from Singapore, Belgium and Oregon, and the status page shows the real availability and error budget left.

## This site as an MCP server

An AI agent can use this site directly: it is a Model Context Protocol (MCP) server. POST /mcp is an MCP server over Streamable HTTP with five read-only tools: ask, search_facts, list_topics, get_status and get_delivery. Add the URL to Claude Code, Cursor or any MCP client and the agent can answer questions about my work from the site's own facts and read its live SLO and delivery numbers. It needs no key, keeps no session and runs no model: retrieval only, so the calling agent brings its own LLM. It speaks the stateless 2026-07-28 revision of the protocol and still answers the initialize handshake of older clients, written in Go against the standard library. For crawlers there is /llms.txt, /llms-full.txt and a Markdown copy of every page under /md/; from a terminal, curl on the home page returns a text card with the live commit and uptime. The /agents page explains how to connect and has a console that sends real requests.

## How this site releases

Every push to main goes out the same way, and each step can stop it. CI builds the image once, smoke-tests it, holds it to Lighthouse budgets and a critical-CVE gate, then pushes it with an SBOM and signed build provenance. The deploy releases that exact digest without rebuilding: it verifies the attestations, holds the release if the site's error budget is spent, starts the new revision with no traffic and smoke-tests it at its own URL, then moves traffic and checks again, rolling back on its own if that fails. A rollback drill rehearses this in production; the first one had traffic back on the old revision 20 seconds after the switch. Every release becomes a GitHub release, and the changelog page lists each one with the commits it shipped.

## Architecture decisions for this site

Why this site is built the way it is, as architecture decision records (ADRs): Run the site on Cloud Run, not GKE. One Go binary serves the static site and the API. Build once, release that digest, check it before it takes traffic. Measure the site's SLO from outside. Call Google Cloud with plain REST, not client libraries. Answer questions by retrieval first; an LLM is optional. The error budget gates releases, in code. Page with scheduled checks, not Cloud Monitoring alert policies. No long-lived credentials anywhere. An MCP server with no sessions and no model. Chapters, not one long page. Each record gives the context, what the choice buys and what it costs.

## Postmortems from this site

What went wrong while I built and ran this site, written up as blameless postmortems (incident reviews). “Message sent” for messages nobody would read: For the site's first 3 h 32 min, the contact form said a message was sent while keeping it on a disk that Cloud Run throws away. The lesson: Only say “done” after the durable handoff. The health check that only worked from inside: Cloud Run's front end answers some paths ending in “z” itself, so /healthz returned 404 to every outside check while the app was healthy. The lesson: Test through the same front door your users use. A failing check on a healthy site: The smoke test reported /metrics as broken while Cloud Run's logs showed every /metrics request returning 200. The lesson: A flaky check is a broken check. Fix it before you trust it, and before you learn to ignore it. Each one has a timeline, the root cause, how it was detected and action items linked to the commits that fixed it.

## On-call drills

At /oncall you can practise being on call: two simulated pages, one on each platform I run. Building IoT platform, paged at 02:14: Telemetry fresh: fast burn. AI agent platform, paged at 10:42: Document assistant: slow answers, fast burn. Each drill uses the platform's own services and dashboards; the objectives behind the pages are proposals or made up for practice, and so is what goes wrong. Every look and every action costs simulated minutes while the error budget burns, and at the end you get a debrief of your own response: the timeline, time to mitigate, budget spent and what went well. The site's terminal runs the same drills as text: type oncall.

## Writing

Things I have written up at length: "Running an agent benchmark on a supercomputer, without root". "What 320 agent runs taught me about serving LLM agents". "Finding a metrics bill in 914 orphaned Kafka topics". All are in the Writing section, with the measurements behind them.