Skip to content
Li ZhenXi, home page

Writing and postmortems

Writing

Longer notes on what the measurements showed. The numbers are the same ones the rest of this site uses.

What broke here, written up properly.

Blameless postmortems from building and running this site: timeline, root cause, how it was caught, and what changed. The times and numbers come from the commits, the CI history and Cloud Run's request logs.

Want to be on the other side of one? Go on call for a simulated incident, and get a postmortem of your own response.

  1. Near miss1 Oct 2026

    “Message sent” for messages nobody would read

    For the site's first 3 h 32 min, the contact form said a message was sent while keeping it on a disk that Cloud Run throws away.

    3 h 32 minthe form could lose a message

    6 of 6 actions doneRead it

  2. Near miss1 Oct 2026

    The health check that only worked from inside

    Cloud Run's front end answers some paths ending in “z” itself, so /healthz returned 404 to every outside check while the app was healthy.

    0visitors affected: caught before launch

    3 of 3 actions doneRead it

  3. False alarm1 Oct 2026

    A failing check on a healthy site

    The smoke test reported /metrics as broken while Cloud Run's logs showed every /metrics request returning 200.

    43 / 43requests to /metrics answered 200 during the replay

    3 of 3 actions doneRead it