You're viewing a live demo of the Engineering theme.See what's included
← Back to Platform Dispatch

incident · Sep 03, 2026

Incident Review: The Retry Storm Nobody Wrote

A slow tax lookup turned into six times normal traffic in three minutes, and no single team had written a retry bug. A blameless review of how two sensible defaults multiplied.

Incident Review: The Retry Storm Nobody Wrote

On a Tuesday afternoon our invoice-render service spent fourteen minutes returning errors. The cause was a dependency that got slow for about two minutes. The remaining twelve were us hurting ourselves.

This is a blameless review, which means nobody gets named for a bad line of code. Mostly because there wasn't one.

What happened

Time

Event

14:02

tax-lookup p99 climbs from 180 ms to 2.4 s

14:03

invoice-render starts timing out; retries begin

14:04

tax-lookup is serving about six times its normal request count

14:05

Oksana pages the on-call; the dependency's own latency is already back to normal

14:16

Retries drain after Bram turns off the gateway retry

The dependency recovered at 14:04. Our traffic didn't, because by then the retries were generating their own load, and a service that is being hammered stays slow no matter how healthy it was a minute ago.

Two sensible defaults

The gateway in front of invoice-render retries a failed upstream call up to three times. That's a good default, set years ago after a real flaky-network week. The mobile client, separately, retries a timed-out request up to three times. Also a good default, also set for a real reason.

Neither team had done anything wrong. Together they meant that one slow request could become up to sixteen attempts: the client's four tries, each of which the gateway could try four times. No test ever composed them, because the two sets of defaults lived in two repositories owned by two teams that had never needed to talk.

Here is the query that showed it. It divides attempts by distinct requests for each minute.

sqlite3 -header -column :memory: "
CREATE TABLE m(minute TEXT, requests INT, attempts INT);
INSERT INTO m VALUES ('14:02',1200,1210),('14:03',1180,3490),('14:04',1150,6870),('14:05',1190,7010);
SELECT minute, round(1.0 * attempts / requests, 1) AS amplification FROM m;"

The rows come back as 1.0, 3.0, 6.0 and 5.9. Real requests stayed flat at around twelve hundred a minute. Attempts went from twelve hundred to seven thousand. The sample table here is trimmed to four minutes, but the shape is what we saw on the real dashboard.

Why it took twelve minutes to stop

The on-call looked at the dependency first, as you do, and found it healthy. That cost about five minutes. The signal that mattered, attempts per request, wasn't on any dashboard because nobody had ever needed it.

What we're changing

  1. Retries get a budget, not a count. A caller may retry only while retries stay under ten percent of its recent requests, and the budget is shared across the gateway and the client.

  2. Every retry adds jitter and doubles its wait. Before, the retries were immediate.

  3. Attempts per request is now a panel on every service dashboard, with an alert at 1.5.

  4. The gateway sends a header marking a request as already retried. The client reads it and doesn't retry again.

The fourth item needed a conversation between two teams, which is the real fix. The code change took an afternoon. Getting two owners to agree what a retry means took a week, and it was the useful week.

incidentretriespostmortem
Share this post
Report