# A failing check on a healthy site

Blameless postmortem (false alarm), 1 Oct 2026, by Li ZhenXi. Page: https://portfolio-emsghin2iq-as.a.run.app/postmortems/smoke-test

> The smoke test reported /metrics as broken while Cloud Run's logs showed every /metrics request returning 200.

- **43 / 43**: requests to /metrics answered 200 during the replay
- **0 / 20**: replays of the check reproduced the failure
- **2**: causes: one confirmed, one by construction

## Impact

No user impact. The cost was a red check on a healthy revision. Left alone, it would have failed deploys at random, and a gate that fails at random gets ignored or bypassed.

## Timeline

- **12:41** (change): A revision serving precompressed pages goes live.
- **12:42:30** (detect): The smoke test against the live site reports “metrics exposed” as failed. Cloud Run's log for that same request: 200.
- **12:42:40** (investigate): Replayed for 21 s: 20 runs of the check as written and 20 that read the whole body. None failed, and all 43 requests show 200 in the log.
- **12:43** (fix): The pipe race is real by construction, so the checks stop using grep -q rather than waiting to catch it again.
- **12:44:01** (investigate): Three back-to-back runs of the fixed script: the third fails the contact check. The log shows 429 from the rate limiter, not the 400 the check expected.
- **12:44** (fix): Both fixes committed. CI and the deploy's verification switch to the same script.
- **12:48** (resolve): The fix's own deploy runs the full smoke test against the new revision and passes.

## Root cause

Most likely: each check piped curl into grep -q under pipefail. grep -q exits at the first match; if curl still has body to write, the write fails and pipefail marks the whole check as failed. The metric it looked for sits partway down a 19 KB response, so grep can finish first. The replay never caught it, so this is the cause by construction, not by reproduction.

Confirmed: the contact check sends bad input and expected a 400. The rate limiter runs first, so after 3 requests in 10 minutes it answers 429.

## Contributing factors

- CI's smoke step and the deploy's verify step each had their own copy of the same greps. Fixing the script alone would have left both pipelines flaky.

## Detection

The check itself, contradicted by Cloud Run's request logs. Trusting the logs over the check is what settled it.

## What went well

- The request logs were checked before anyone touched the app.
- When the replay came back clean, the race was removed anyway instead of being written off as a one-off.
- Re-running back to back, rather than once, surfaced the second cause.

## Where we got lucky

It showed up on day one, while the whole pipeline was fresh in mind.

## Action items

- [x] Read the whole response before matching, so a match can never fail the pipe. (commit cfcdeb9)
- [x] Count 429 as a rejection in the contact check. (commit cfcdeb9)
- [x] One smoke script, run locally, in CI against the container, and after every deploy against the live URL. (commit cfcdeb9)

## Lesson

A flaky check is a broken check. Fix it before you trust it, and before you learn to ignore it.