Near miss1 Oct 2026Blameless postmortem
“Message sent” for messages nobody would read
For the site's first 3 h 32 min, the contact form said a message was sent while keeping it on a disk that Cloud Run throws away.
- the form could lose a message
- 3 h 32 min
- requests in that window, all the smoke test's
- 11
- visitor messages lost
- 0
Impact
None in the end. Cloud Run's request logs show 11 POSTs to the contact API while the bug was live, every one from the smoke test sending deliberately bad input: 8 rejected with 400, 3 with 429. No visitor used the form. Had a recruiter written in that window, they would have been told it was sent and never heard back.
Timeline
Singapore time (UTC+8), 1 Oct 2026 unless marked
Change
The site's first real revision goes live (the 12:10 one ran Google's placeholder image). The contact API validates a message, appends it to a file and replies 202. Forwarding to a webhook ran in the background, if one was configured. None was.
Change
11 requests reach the contact API, all from the smoke test: 8 rejected as invalid, 3 by the rate limiter. No visitor writes in.
Detected
Found in review, walking the contact path end to end: on Cloud Run that file lives in per-instance scratch space that nobody reads.
Fix
Fix committed: deliver to the webhook before replying, up to 3 attempts inside 6 s, and a 502 that tells the visitor to email instead if delivery fails.
Fix
Deployed. Until a webhook URL is added, the form can still say “sent”.
Fix
Webhook URL added by hand as a Secret Manager version, so it never appears in the repository or in Terraform state.
Resolved
A new revision picks up the secret. A test message is delivered at 15:46:31 and one from a browser at 15:51.
Resolved
The last gap closed: on Cloud Run, a missing webhook now gets a 503 and the email fallback, never “sent”.
Root cause
The success reply did not depend on delivery. The API answered 202 once the message was on local disk, and that disk is not durable: Cloud Run's writable filesystem is in memory, belongs to one instance and disappears with it.
The background send would not have been reliable either. With request-based billing, an instance gets almost no CPU once it has replied, so work left running after the response can stall.
Contributing factors
- Built and tested locally, where the file persists and a developer can read it. The local setup hid the production storage model.
- The webhook was optional and its absence was silent: no startup warning, and the only metric counted replies, which were all “accepted”.
How it was detected
Code review, not monitoring. Nothing could have fired: every signal the system had said the form was working.
What went well
- Validation and rate limiting held: all 11 requests in the window were rejected.
- The webhook secret never touched the repository or Terraform state.
- The fix went from commit to live in 4 minutes, through the normal pipeline.
Where I got lucky
Nobody used the form during the window. That is luck, not design.
Action items
6 of 6 done
Deliver before replying, with retries inside a 6 s budget and an email fallback on failure.
Keep the webhook URL in Secret Manager; Terraform creates the empty secret and wires it to the service.
Tests for the contact package: from none to 77% coverage.
Fail closed on Cloud Run: no webhook means a 503 and the email fallback, never “sent”.
shipped 16 Oct
Alert when a message goes undelivered: an hourly scheduled check searches Cloud Run's logs for one since its last search, and a hit fails the check, which emails me. A free GitHub Action rather than a Cloud Monitoring alert policy, which is billed per condition.
shipped 19 Oct
A synthetic end-to-end check twice a day: a message marked by a secret token goes through the live form and the real webhook, and is posted silently so it notifies no one.
shipped 19 Oct
Only say “done” after the durable handoff.
Blameless means the write-up is about systems and decisions, not people. Every decision here was mine. Sources: the repository's commits, the GitHub Actions run history, and Cloud Run's revisions and request logs.