Building IoT platformPrimustech, 2026 · Case study
Changing a live cluster in steps that can't break it
Security and reliability changes to a running platform, each split so that no single step can take it down, and rehearsed first.
- steps for a breaking change: widen first, tighten second
- 2
- paths meant to close only at cutover that the allow rules alone closed, in rehearsal
- 28 of 46
- without a command route when a transport pod moves, before and after a second replica (live upgrades)
- 88 s → 5-16 s
A small working model of the real system. Press the buttons.
- coreno auth
- rule engineno auth
- http transportno auth
- mqtt transportno auth
Cache
open
No password anywhere
The cache accepts any client inside the cluster. Flip a switch to start the change.
The problem
The platform carried live telemetry and control traffic. Its cache accepted clients without a password, one transport pod held every gateway session, and nobody had written down what would be lost when a node went away. Every fix was a change to a system that could not stop.
What I did
- 1
Gave the cache a password in two steps. First every client sends it, which the server accepts while it still requires none; then the server requires it. Neither step can break the cache, and either can be undone alone.
- 2
Rehearsed a default-deny network policy before any cutover, and found the obvious rollout was not reversible: the allow rules alone closed 28 of the 46 paths that should close only at cutover, and deleting the deny rule afterwards left 43 closed. So the cutover plan gives each namespace a transition policy to fall back to, and its own window.
- 3
Recorded all 16 single points of failure, each with its blast radius, a recovery time marked measured or estimated, and an owner wherever one could be named. A node upgrade had taken lanes out for between 9 seconds and 3 minutes 18 seconds, and losing the one transport pod left 88 seconds with no route for commands.
- 4
Ran that transport at two replicas behind a disruption budget. In the next live node upgrades a moved session was back in 5 to 16 seconds instead of 88; on the test rig it takes 1 to 2.
- 5
Set freshness objectives per data lane from 30 days of measured cadence, separating "the site is late" from "we are behind", with alerts, runbooks and synthetic cases that proved the notifications arrive.
Every change is planned as steps that can be checked and undone on their own, and the platform now has a register of what breaks, for how long, and who decides.
Stack
- GKE
- Calico NetworkPolicy
- Valkey
- Terraform
- Secret Manager
- Cloud Monitoring