# Alerts that mean something

Building IoT platform (Primustech), 2026. Case study by Li ZhenXi. Page: https://portfolio-emsghin2iq-as.a.run.app/work/observability

> Dashboards and alerts built around how this system actually fails.

- **28**: alert policies, all in Terraform
- **40**: tiles on the platform dashboard
- **0**: drift between Terraform and what is live

## The problem

The cluster could lose ZooKeeper quorum, stall consumers or silently drop telemetry without anyone being told.

## What I did

1. Versioned Grafana SRE dashboards and Cloud Monitoring alerts for ZooKeeper quorum loss, stalled consumers and crash loops.
2. Chose signals by failure mode: absence alerts for ingestion, silent-loss detection, consumer lag checked against throughput, HPA headroom, and Kafka memory on RSS rather than working set.
3. Fixed a ZooKeeper leader-election livelock that appeared after a node was replaced, and kept every alert and dashboard in Terraform with zero drift.

## Result

Failures that used to be silent now raise an alert, and the metrics bill went down, not up.

Tools: Grafana, Prometheus, PromQL, Cloud Monitoring, ZooKeeper.

## Go further

- [SLOs, error budgets and burn-rate alerts, on a Grafana-style board](https://portfolio-emsghin2iq-as.a.run.app/iot#platform-signals)
- [Postmortems I wrote for this site](https://portfolio-emsghin2iq-as.a.run.app/writing#postmortems)