Retries, backoff and jitter: control repeated work
Design bounded retries with exponential backoff, jitter and deadlines. Calculate attempt amplification and decide which failures are safe to retry.
TL;DR: Retry only failures that are both recoverable and safe to repeat. Limit total attempts, randomize the delay and keep every attempt inside the caller's remaining deadline and the service's retry budget.
Repeating a request consumes someone else's capacity
A timeout leaves the outcome uncertain. The server may have committed the operation before its response was lost. For a payment or job submission, establish an idempotency contract before replaying the request. Backoff changes timing; it cannot prevent duplicate business effects.
Classify failures at the operation boundary. A temporary connection failure may justify another attempt. Invalid input needs correction. Authentication failure normally needs a credential decision rather than the same request repeated immediately. A throttling response requires attention to the server's retry advice and the client's deadline.
| Failure | Candidate action | Evidence required |
|---|---|---|
| Connection unavailable before a read operation | Bounded retry | Operation remains useful within deadline |
| Response lost after a write | Retry with the same deduplication identity | Server enforces that identity across attempts |
| Request rejected as invalid | Return the error | Caller must change the request |
| Dependency overloaded | Reduce offered load and consider delayed retry | Retry allowance and server timing permit it |
The AWS SDK retry guide describes a practical combination of attempt limits, full jitter and retry quotas. Check the actual SDK configuration: “three attempts” includes the first call, whereas “three retries” permits four calls.
Compute the delay, then ask whether another attempt fits
One full-jitter policy chooses a delay uniformly between zero and a capped exponential window. For retry number k, beginning at zero, define window = min(cap, base × 2^k). The following Python example uses a local random generator so the illustration is reproducible; production clients should not seed every process identically.
import random
rng = random.Random(17)
base, cap = 0.1, 0.8
for k in range(5):
window = min(cap, base * 2**k)
delay = rng.uniform(0, window)
assert 0 <= delay <= cap
print(k + 1, round(window, 3), round(delay, 3))
The windows are 0.1, 0.2, 0.4, 0.8 and 0.8 seconds. Individual delays can decrease even while windows grow. Randomization spreads clients across a window; it does not promise evenly spaced traffic or eliminate overload.
Suppose a caller has 700 ms remaining. It reserves 100 ms to finish its own response, needs up to 400 ms for the next dependency attempt and selects a 250 ms delay. That retry does not fit: 750 ms exceeds the remaining budget. Reject it before sleeping. Deadline propagation explains how cancellation should follow the request through deeper services.
One layer should own the retry decision
Consider three nested services, each allowing three total attempts. If every deeper attempt fails, one user request can drive 3 × 3 × 3 = 27 calls to the bottom dependency. The arithmetic assumes retries are independently nested and all attempts are exercised. It is a failure scenario, not the normal request rate.
Choose the layer with enough operation context to decide safely, and account for retries already implemented by SDKs, proxies and queue consumers. Disable or tightly bound redundant policies. Otherwise an application's “one retry” may hide many transport attempts.
A retry budget limits extra work over a population of requests. For an illustrative allowance of ten extra attempts per hundred original requests, 1,000 originals permit 100 additional calls during the accounting interval. Specify how that allowance is replenished and partitioned. A single tenant should not consume the entire retry allowance for unrelated traffic.
Test the outage and the recovery wave
Test a dependency that rejects requests for a while and then recovers. Observe original request volume separately from attempt volume. Record cancellation, retry count and deadline exhaustion without attaching unbounded request identifiers to metric labels. Recovery should restore useful throughput without a synchronized replay of every failed call.
Combine this with circuit breakers when repeated calls to a failing dependency should stop temporarily. Breaker probes and retries need a shared capacity argument; multiplying them blindly recreates the original load problem.
Self-check: after adding jitter, duplicate orders disappear in a small test. Has the duplicate-write problem been solved?
No. Different timing can make the race harder to reproduce. Verify server-side deduplication by losing a successful response and retrying with the same operation identity. The test must assert one durable order, not merely one response received by the client.