How do you review retry logic safely?

A retry can help when a failure is temporary and a later attempt has a reasonable chance of succeeding. Validation failures, permission failures, and malformed requests usually need a changed request or changed authority instead. Retrying them automatically spends time and capacity without changing the condition that failed.

A timeout deserves special care because it describes what the caller observed, not necessarily what the server did. The server might have rejected the operation, still be processing it, or have completed the side effect before the response was lost. Mark that outcome as unknown until the system's contract gives you evidence otherwise.

An operation is idempotent when repeating the same request is intended to have the same effect as making it once. That does not require every response to be byte-for-byte identical. Setting a record to a known state can be repeatable, while incrementing a counter, sending a message, or creating a new charge can produce another effect on every attempt.

For each retryable call, list the durable writes, messages, external requests, and user-visible actions it can trigger. Then find the mechanism that makes a repeated attempt safe: a stable operation key, a uniqueness constraint, a compare-and-set condition, or a reconciliation step that checks whether the first attempt already took effect.

Retries at several layers can multiply calls when a dependency is already struggling. Pick the layer that has enough context to classify the failure and own the policy. Record the maximum attempts or deadline there so callers and inner clients do not each start an independent retry loop.

Use delay and jitter to avoid sending every retry back at the same instant, but do not confuse pacing with safety. Backoff cannot make a permanent error temporary, prevent a duplicate side effect, or decide how long the user can wait. The total deadline still has to fit the caller's latency and cancellation contract.

When the budget ends, the code should preserve enough evidence for the caller and operator to act: the final error, the number of attempts, relevant timing, and whether the outcome is known or uncertain. A generic fallback can hide a partial success and make later reconciliation harder.

Test at least four paths: immediate success, a temporary failure followed by success, a permanent failure with no retry, and an exhausted or ambiguous attempt. Use a controllable clock or scheduler so the test verifies the policy without sleeping in real time.