Near miss1 Oct 2026Blameless postmortem
The health check that only worked from inside
Cloud Run's front end answers some paths ending in “z” itself, so /healthz returned 404 to every outside check while the app was healthy.
- visitors affected: caught before launch
- 0
- from first sight to fix
- 6 min
- from the front end; the app never saw the request
- 404
Impact
None on the live site: it was caught on a trial deploy, before this site's own service existed. Had it shipped, every deploy's verification would have failed against a healthy revision, and an outside uptime check would have reported the site down permanently. That is the kind of false alarm that teaches people to ignore alerts.
Timeline
Singapore time (UTC+8), 1 Oct 2026 unless marked
Change
First commit. The API serves /healthz and /readyz; the container health check, the CI smoke step and the deploy's verification all call /healthz. All green, because every test talks to the container directly.
Change
A trial deploy to Cloud Run, before this site had its own project.
Detected
The trial service is not public yet, so an anonymous request should be refused with 403. /healthz gets 404 instead: something answered before access was even checked.
Investigating
Cloud Run's documentation: some paths ending in “z” are reserved, and it recommends avoiding all of them.
Fix
Fix committed: /api/health for every check that goes through the front end. /healthz and /readyz stay for the container probe, which connects directly.
Resolved
This site's first real revision goes live (Terraform's at 12:10 ran Google's placeholder image). The pipeline's verification against /api/health passes.
Root cause
Cloud Run reserves some URL paths ending in “z”. Requests for them through the public front end are answered before they reach the container, so the app returns 200 to its own probe and the world gets 404.
Contributing factors
- Every pre-launch test hit the container directly: on localhost, and in CI's container smoke step. Nothing exercised the real front door.
- /healthz is the Kubernetes convention, where it is fine. It came along without checking the new platform's rules.
How it was detected
A manual check from outside, on a trial deploy. No automated check would have caught it before launch; after launch it would have looked like the deploy gate failing on healthy revisions.
What went well
- Checking from outside before sharing the link.
- Reading the odd status code instead of moving on: a 404 where a 403 was due pointed straight at the front end.
Where I got lucky
It was found by a habit, not by a test designed to find it.
Action items
3 of 3 done
Add /api/health for checks that go through the front end; it also reports not-ready if the knowledge base failed to load.
Run the full smoke test against the live URL after every deploy, so the real front door is tested every time.
Leave a comment in the server code on why both health endpoints exist.
Test through the same front door your users use.
Blameless means the write-up is about systems and decisions, not people. Every decision here was mine. Sources: the repository's commits, the GitHub Actions run history, and Cloud Run's revisions and request logs.