False alarm1 Oct 2026Blameless postmortem
A failing check on a healthy site
The smoke test reported /metrics as broken while Cloud Run's logs showed every /metrics request returning 200.
- requests to /metrics answered 200 during the replay
- 43 / 43
- replays of the check reproduced the failure
- 0 / 20
- causes: one confirmed, one by construction
- 2
Impact
No user impact. The cost was a red check on a healthy revision. Left alone, it would have failed deploys at random, and a gate that fails at random gets ignored or bypassed.
Timeline
Singapore time (UTC+8), 1 Oct 2026 unless marked
Change
A revision serving precompressed pages goes live.
Detected
The smoke test against the live site reports “metrics exposed” as failed. Cloud Run's log for that same request: 200.
Investigating
Replayed for 21 s: 20 runs of the check as written and 20 that read the whole body. None failed, and all 43 requests show 200 in the log.
Fix
The pipe race is real by construction, so the checks stop using grep -q rather than waiting to catch it again.
Investigating
Three back-to-back runs of the fixed script: the third fails the contact check. The log shows 429 from the rate limiter, not the 400 the check expected.
Fix
Both fixes committed. CI and the deploy's verification switch to the same script.
Resolved
The fix's own deploy runs the full smoke test against the new revision and passes.
Root cause
Most likely: each check piped curl into grep -q under pipefail. grep -q exits at the first match; if curl still has body to write, the write fails and pipefail marks the whole check as failed. The metric it looked for sits partway down a 19 KB response, so grep can finish first. The replay never caught it, so this is the cause by construction, not by reproduction.
Confirmed: the contact check sends bad input and expected a 400. The rate limiter runs first, so after 3 requests in 10 minutes it answers 429.
Contributing factors
- CI's smoke step and the deploy's verify step each had their own copy of the same greps. Fixing the script alone would have left both pipelines flaky.
How it was detected
The check itself, contradicted by Cloud Run's request logs. Trusting the logs over the check is what settled it.
What went well
- The request logs were checked before anyone touched the app.
- When the replay came back clean, the race was removed anyway instead of being written off as a one-off.
- Re-running back to back, rather than once, surfaced the second cause.
Where I got lucky
It showed up on day one, while the whole pipeline was fresh in mind.
Action items
3 of 3 done
Read the whole response before matching, so a match can never fail the pipe.
Count 429 as a rejection in the contact check.
One smoke script, run locally, in CI against the container, and after every deploy against the live URL.
A flaky check is a broken check. Fix it before you trust it, and before you learn to ignore it.
Blameless means the write-up is about systems and decisions, not people. Every decision here was mine. Sources: the repository's commits, the GitHub Actions run history, and Cloud Run's revisions and request logs.