Building IoT platformPrimustech, 2026 · Case study
Alerts that mean something
Dashboards and alerts built around how this system actually fails.
- alert policies, all in Terraform
- 28
- tiles on the platform dashboard
- 40
- drift between Terraform and what is live
- 0
A small working model of the real system. Press the buttons.
leader
follower
follower
Quorum held, 3 of 3 up
All quiet.
The problem
The cluster could lose ZooKeeper quorum, stall consumers or silently drop telemetry without anyone being told.
What I did
- 1
Versioned Grafana SRE dashboards and Cloud Monitoring alerts for ZooKeeper quorum loss, stalled consumers and crash loops.
- 2
Chose signals by failure mode: absence alerts for ingestion, silent-loss detection, consumer lag checked against throughput, HPA headroom, and Kafka memory on RSS rather than working set.
- 3
Fixed a ZooKeeper leader-election livelock that appeared after a node was replaced, and kept every alert and dashboard in Terraform with zero drift.
Failures that used to be silent now raise an alert, and the metrics bill went down, not up.
Stack
- Grafana
- Prometheus
- PromQL
- Cloud Monitoring
- ZooKeeper