A second chance consumes real resources
A business application asks an AI model to analyze a document. The response takes too long, the client times out and retries. Meanwhile an intermediate service and the model-calling library also retry. How many executions reach the expensive component? Does a random wait solve the problem? Three small verifiable experiments separate theoretical reliability, call multiplication and timing. We identify each mechanism’s guarantees and what still requires real-service measurement.
Abstract. For independent failure probability q and at most N total attempts, derive expected cost and success probability. Without independence, three layers exhausting three attempts each can produce 27 leaf calls per original request. A 20-retry budget for 100 initial requests caps attempts at 120 without refill. Finally 100 clients, with a declared seed, spread one retry wave across 100 milliseconds: peak count in 10-millisecond bins falls from 100 to 14 while total retries remain 100. These are executed educational counts, not server or EL-AI benchmarks.
Count first: an attempt is not a retry
An attempt is an actual call; a retry is an attempt after the first. N=3 always means one initial attempt and at most two retries. The original request is the user’s goal and may generate many calls. This distinction prevents counting errors across layers. Cost here means attempt count, not money, processed tokens or GPU seconds; provider billing is not assumed.
Marc Brooker’s 2019 AWS technical guide distinguishes timeouts, increasing waits, randomization and retry limits, including layered amplification. It documents operational experience, not an AWS experiment reproduced here. Our models isolate selected effects under explicit assumptions, without treating SDK settings as universal AI-service behavior. Actual behavior depends on error semantics, cancellation and resources.
The favorable case: independent failures
First suppose each call fails with constant probability q independently of other calls. Success stops retries. Reaching attempt 2 requires one failure, probability q; reaching attempt 3 requires two, probability q². Expected attempt count sums the probabilities of reaching each attempt, since each contributes one only if reached. That is the question answered by the formula.
A is a dimensionless count, N the total maximum, q between 0 and 1. With q=.5,N=3, expected attempts are 1+.5+.25=1.75 and success probability.875. A request does not make 1.75 calls: this is a theoretical repeated-process mean. At q=.9, expected attempts 2.71 buy success probability.271. More attempts against near-certain failure buy little success and much load. If retries themselves change q through load, the formulas no longer describe the system; they are not outage predictions.
| q | N | Expected attempts | Success probability |
|---|---|---|---|
| 0.1 | 3 | 1.11 | 0.999 |
| 0.5 | 3 | 1.75 | 0.875 |
| 0.9 | 3 | 2.71 | 0.271 |
Three layers: when three becomes twenty-seven
Now use a deterministic all-fail case. Each layer learns failure after the lower layer exhausts attempts, with no cancellation. The outer layer tries three times; each invokes three middle attempts; each invokes three final calls:3 × 3 × 3=27. The attached recursive function enumerates this tree, without random failures or the previous model’s q.
D counts retrying layers, not every software step. Retrying at only one layer makes this model’s count 3. This does not prove top-level retries are always optimal: rerunning an entire workflow may repeat successful work, while local retries can recover one cheap step. The policy must consider the whole path. Work continuing after caller timeout can create overlapping calls not measured by this sequential tree.
A budget limits count, not success
Now take 100 initial requests, at most 3 attempts each and a shared 20-token counter for retries only. No refill occurs in this window; each retry consumes a token atomically. Initial attempts do not. Maximum total is 100+min(20,100 × 2)=120, versus 300 without budget when all fail. Code runs a simple allocation:100 initial failures, then 20 second calls also fail;80 requests receive no first retry and none receives a third attempt.
The 120 bound is an admission property, not server capacity. Every request can still fail. Ten independent local 20-token counters can admit 200 retries, not 20 globally. Real systems must define scope, refill, concurrency and behavior when the counter is unavailable. This no-refill model is not a complete token-bucket simulation. Its ordering also does not ensure client fairness: load limits and priority policies solve different problems.
Jitter: spreading a crowd without shrinking it
In experiment 3,100 clients each make exactly one retry. Fixed 100-millisecond waits synchronize arrivals. Full jitter draws each delay uniformly between 0 and 100 milliseconds. Python 3.14.0 random.Random uses seed 20261017; every delay is saved. Counts in 10-millisecond bins are 9,7,9,12,14,14,7,11,6,11, peak 14. Fixed waits place 100 arrivals in the bin starting at 100 milliseconds. Both policies still total 100 retries.
The formula extends the idea to waves: k starts at 0 for the first retry; B₀ and cap B_max are milliseconds. Only one wave with B₀=100 was executed, not exponential recovery. Uniform delay has mean 50 milliseconds, not 100, so peak comparison hides a waiting-time difference. We simulated no queue, service time, load-dependent errors or model latency. Fourteen arrivals per bin are not fourteen simultaneous executions or a guaranteed 99 th-percentile improvement.
The deadline belongs to the whole task
Per-call timeout is not task deadline. Three attempts capped at 200 milliseconds with waits 50 and 100 yield worst-case 750 milliseconds, excluding other costs. With a 600-millisecond caller deadline, two failed attempts plus both waits consume 550, leaving 50, not 200. Attempt 3 must fit the remaining time or be denied. This is a model bound, not observed latency. A timer excluding DNS, connection or body reading is not automatically end-to-end time.
Cancellation matters for AI: a client stopping its wait does not prove GPU generation stopped. Retrying may duplicate ongoing work. For mutations or actions, timeout also does not prove no side effect occurred. Correctness needs idempotency semantics or outcome reconciliation, not implemented here. Retry limits protect resources but do not make duplicated mutations safe. Waiting alone also does not repair credentials or schema errors.
One seed is not a guarantee: peak probability
The observed 14 is a saved random-sequence result, not a proven maximum. In the ideal model, how likely is at least 25 arrivals in any of ten bins? Each client independently selects one bin with probability.1. A single-bin count X is binomial with 100 trials and probability.1. The ten bin counts are not assumed independent: their sum is 100. This prevents unjustified multiplication of probabilities.
The binomial coefficient counts which j clients select the bin; the next factors give entry and non-entry probabilities. Summing j=25…100 gives about.0000130728 for one bin. The union bound sums ten risks, giving upper bound.000130728, about.0131%, for any bin reaching 25. This dimensionless number is not outage probability: server capacity is unknown. It requires independent uniform draws and the stated fixed bins; shared random streams, coarse timers or extra retries change the model.
There are three evidence levels. Peak 14 is a reproducible numerical observation of our sequence. Mean 10 and the probability bound follow from distribution assumptions. Avoiding actual overload remains empirical because service times and resources are unmodeled. Distinguishing them prevents turning an orderly simulation into an operational promise. Bin width also changes measured peak; comparisons must preserve metric definition.
What remains to verify on a real service
The experiments remain separate: independent q was not linked to a queue, nor peak 14 converted into success probability. Operational comparison needs original arrivals, call durations and costs, cancellation propagation, errors and actual layer policies. A future test should separate completed user requests from attempts, useful from duplicate work, mean latency from percentiles and recovery from overload. These are proposed measurement requirements, not EL-AI or provider results.
Retries help when recovery is plausible and added cost fits resources and deadline. Jitter spreads arrivals, budgets limit count and coordinated policies prevent unnoticed layered multiplication; none replaces the others. Controlled counts establish this separation. A service with smaller peaks can still perform too much total work; a budgeted service can still complete no requests.
from experiment import expected, nested
print(expected(.5,3), nested(3,3))Code, data, and instructions · JSON. Educational calculations executed with Python 3.14.0; figures with Matplotlib 3.11.2. AI-assisted analysis, without claiming peer review or human review. Original illustrative ImageGen cover: it does not document EL-AI people, premises, or installations. Sources accessed 17 October 2026.