Safe retries in distributed systems

August 29, 2026

Intro

In distributed systems, inter-system communication failures are inevitable, like transient network failures, throttling exceptions, or downstream service being unavailable for some time.

The usual mitigation for these failure modes is to just try again, or to retry.

“Just retrying” in distributed systems can cause 2 categories of problems:

The correctness problem

When a request times out, you don’t know what happened on the other side. The downstream service may have processed it successfully, you just never got the confirmation. If you retry that request, the same operation is requested twice (i.e. a user may get charged twice for a single purchase).

The load problem

When a downstream service gets slow, upstream services start timing out and retrying. The already struggling service now receives more traffic than before the incident started. It slows further, more clients start getting timeouts, resulting in more retries, turning the partial outage into a full outage.

Solving the correctness problem

Most operations in distributed systems are not idempotent by default. An operation is idempotent if executing it multiple times produces the same result as executing it once. Charging a user is not idempotent. Sending an email is not idempotent. Creating a record in a database is not idempotent. These operations have side effects, repeating them causes real damage.

Idempotency keys

The standard pattern to address this is idempotency keys. When a client sends a request, it generates a unique key for that operation and attaches it to the request. If the client retries, it sends the same key. The downstream service uses that key to recognize it has already processed this operation and returns the previous result instead of executing it again.

Back to the payment example: the client generates a unique key for the purchase. On the first attempt, the payment service processes the charge and stores the result against that key. If the client retries due to a timeout, the payment service finds the key, sees it was already processed, and returns the original result. The user is charged once.

For this to work, the downstream service needs to store the result of each operation against its key, usually a short-lived cache or a dedicated table in the database. The client is responsible for generating a unique key per logical operation, not per request. The same key must be reused across retries of the same operation.

Solving the load problem

Even if your retries are logically safe to repeat, firing them too aggressively can take down a struggling service.

Exponential backoff

The first mistake most implementations make is retrying immediately after a failure. If a service is overloaded, hitting it with retries every few milliseconds makes the situation worse. Exponential backoff means waiting longer after each failed attempt. First retry after 1 second, second after 2, third after 4, and so on, i.e. this formula: sleep=min(max_wait, base * 2 ^ attempt). This gives the downstream service time to recover instead of making it struggle further.

Exponential backoff alone is not enough. Imagine thousands of clients all start retrying at the same moment, for example, after a brief outage. Even with backoff, if they all follow the same schedule, they all retry at the same intervals. The downstream service receives a spike of traffic at second 1, another at second 2, another at second 4, the load is still synchronized.

Jitter

The usual solution to this problem is jitter - adding randomness to the backoff interval. Instead of every client waiting exactly 2 seconds, each waits a random amount between 1 and 3 seconds, i.e. this formula: sleep=random(0, min(cap, base * 2^attempt)). The retries spread out over time, the synchronized spikes disappear, and the downstream service sees a steady traffic instead of repeated spikes.