Fault injection · real requests to the Go server on Cloud Run

Break it yourself.

Choose what goes wrong on the server and how the client copes, then send real requests and watch every attempt come back. The faults land on your requests only: the site's SLO is measured from outside, on another endpoint, so nothing here can hurt it.

1. What goes wrong on the server

Requests answered with a 50330%

Requests held before answering0%

How long they are held0 ms

The client waits 1 s for each attempt, so a hold longer than that becomes a timeout. Limits on the server: 60% errors, 1.5 s holds.

2. How the client copes

Give upOne attempt, a 1 s timeout. Whatever goes wrong, the user sees.RetryUp to 3 more attempts, waiting a random 0 to 100·2ⁿ ms (capped at 1 s) between them.Retry + circuit breakerRetries, but after 4 failures in a row it stops calling for 2 s and fails fast, then lets one call through to test.HedgeIf no answer in 300 ms, sends a second copy; the first success wins and the other is hung up on.

3. What happened

Pick the faults and a client, then send. Every bar below will be a real request to the Go server on Cloud Run, drawn as it comes back.

How it works

The server answers /api/lab/echo, holding or failing each request with the odds you chose. It has limits of its own: at most 60% errors, 1.5 s holds, 24 requests held at once on an instance, and 240 requests a minute from one visitor. Those limits are what keep a page like this from costing money or hurting the site: a held request keeps an instance billed while it waits. Lab requests stay out of the live latency on the status page, and the injected errors out of the error logs. The 503s are real, so your browser's developer console lists them as failed requests. The server code

The client runs in your browser: four requests at a time, a 1 s timeout per attempt. The strategies fit on one screen.

Why these four. Retries hide errors but multiply load, which is why they wait a random, growing time between attempts: Timeouts, retries and backoff with jitter. A circuit breaker stops a struggling service from being hammered by its own clients: CircuitBreaker. Hedging trades a few extra requests for a shorter tail: The Tail at Scale.

For a whole incident rather than a single request, go on call.