How do you keep retry with backoff from turning into a thundering herd?

programming web dev
1

Yo folks, I’m wiring up a fetch wrapper that retries with exponential backoff + jitter, and I’m trying to keep it fast under load. My failure mode right now is when the API has a brief outage, a bunch of callers all retry around the same time and it spikes even harder.

const sleep = (ms, signal) => new Promise((res, rej) => {
  const id = setTimeout(res, ms);
  signal?.addEventListener("abort", () => {
    clearTimeout(id);
    rej(new DOMException("Aborted", "AbortError"));
  }, { once: true });
});

export async function fetchWithRetry(url, opts = {}) {
  const {
    retries = 5,
    baseDelay = 200,
    maxDelay = 5000,
    signal,
  } = opts;

  let lastErr;
  for (let attempt = 0; attempt <= retries; attempt++) {
    try {
      const res = await fetch(url, { ...opts, signal });
      if (res.ok) return res;
      if (res.status >= 400 && res.status < 500 && res.status !== 429) return res;
      throw new Error(`HTTP ${res.status}`);
    } catch (err) {
      lastErr = err;
      if (attempt === retries) break;

      const exp = Math.min(maxDelay, baseDelay * 2 ** attempt);
      const jitter = exp * (0.5 + Math.random());
      await sleep(jitter, signal);
    }
  }
  throw lastErr;
}

Algorithm-wise, what’s a solid pattern to reduce herd behavior here (like request coalescing, per-host token bucket, or a shared retry scheduler) without making the complexity explode?

2

Your jitter is fine, but the herd usually comes from every caller running the same loop independently. The simplest pattern is “single-flight” per key: keep a Map of in-flight promises keyed by (method + url + body hash), and have latecomers await the same promise instead of starting their own retries. It’s like one person going to check if the store is open and everyone else just waits for the text back, instead of 50 people driving there at once. I’m not sure how gnarly your request shapes are, but even coalescing just GETs (or only idempotent requests) tends to knock the spikes down a lot without needing a full shared scheduler.

3

Coalesce retries per “key” (method + URL + a normalized body) so you only ever have one in-flight attempt and everybody else awaits that same promise, because jitter alone still lets 200 callers dogpile the moment the outage clears. The tradeoff is you have to be careful not to accidentally merge requests that shouldn’t share a response (anything non-idempotent, or auth/user-specific headers). Do you have a clean way to compute a stable dedupe key for the calls you actually want to coalesce?