Building IoT platform19 Oct 2026 · 2 min read

Finding a metrics bill in 914 orphaned Kafka topics

Managed Prometheus was the third-largest line item and nobody had measured why. Attribution first, then a reaper, then dropping what nobody read.

orphaned Kafka topics, 86% of all topics: 914
samples per scrape from dropping unread metrics: -40%
cloud spend cut, with no alert dropped: ~15%

The building IoT platform I run at Primustech is on GKE, monitored with Google Cloud Managed Service for Prometheus. In the bill, Managed Prometheus was about US$106 a month: the third-largest line item, climbing, with no matching growth in traffic. Nobody had measured where it came from.

Measure before cutting

Managed Prometheus bills by samples ingested, so the first job was attribution: which scrape job produces the samples? One query answers it.

sort_desc(sum by (job) (scrape_samples_scraped))

Samples per scrape, by job

The Kafka exporter alone produced 10,386 of 43,580 samples per scrape, about a quarter of everything, while traffic had not grown to match.

914 topics for pods that no longer existed

The exporter emits seven metric families per topic, so the topic count drives the bill. 914 of the 1,066 topics were named after pods that no longer existed. Deleting them took the exporter from 10,386 to 2,891 samples per scrape, and broker memory fell from 2.4 to 1.8 GiB as a side effect.

Five days later the orphaned topics were back to 191. A one-off clean-up would have quietly undone itself, so the reaper runs as a Kubernetes CronJob.

If waste regrows, the fix is a schedule, not a clean-up.

Drop what nobody reads, and check first

Three of GKE's built-in metric packages were on. Before turning anything off I checked every consumer: 0 of the 27 alerts then in place and 0 of 40 dashboard tiles used them. Turning them off cut all samples per scrape by 40%, from 43,580 to 26,150. cAdvisor stayed, even though it is the biggest remaining source, because three memory alerts depend on it.

The rest of the review

Cloud SQL from 4 vCPU/16 GiB to 2 vCPU/12 GiB, sized from measured use, and regional HA to zonal. The trade-off is written down: no automatic failover, so recovery is 7 backups plus 7-day point-in-time recovery, and reverting is one line of Terraform.
Core API pod requests from 1 CPU/4 GiB to 0.5 CPU/3 GiB so pods pack tighter, with the HPA and cluster autoscaler keeping 3 nodes at the floor and up to 9 under load.
An ingestion lane that had been replaced, retired end to end: six resources, removed only after the new lane carried production.

Together that cut cloud spend by about 15%, and no alert was dropped. Another US$29 or so a month (an unused VPN gateway and one more Cloud SQL step) is going through a reviewed Terraform plan.

What I took from it

Attribute the cost before you cut anything; the biggest line was not where I would have guessed.
A signal with no consumer is a cost with no benefit, but check the consumers, because some are not obvious.
Write the trade-off down next to the change, so the next person knows what was given up and how to undo it.