← All insights

Research benchmark · Agentic AI

Agent Retry Amplification: when a failed tool response repeats a successful side effect

Four retry policies tested across 6,000 synthetic scenarios to measure reliability, retry amplification and duplicate side effects.

By Muzammal Hussain · 2 November 2026 · 6 min read

A tool call can succeed at the remote system while its response is lost or reported as a transient error. An agent that interprets the response as proof of failure may retry an operation that has already taken effect. For operations such as creating a ticket, sending a notification or submitting a payment, that retry can produce a duplicate. The Agent Retry Amplification Benchmark (ARAM) examines this failure pattern and compares four retry strategies in a deterministic synthetic simulator.

What was tested?

ARAM v0.1.0 generates 6,000 synthetic scenarios using five fixed random seeds. Each scenario represents a tool operation with a number of transient failures, a possibility that an operation commits before its error response, and a base latency. Four policies are evaluated on every scenario, producing 24,000 policy-episode records.

The modeled operations are sending a notification, creating a ticket, submitting a payment and updating a record. These are labels in the simulator; no real external services or transactions are executed.

Policies compared

• No retry: makes one attempt and does not repeat an operation after an error.
• Naive retry: retries transient failures up to the configured five-attempt limit, without a backoff delay beyond the simple modeled retry cost.
• Bounded backoff: caps attempts at three and applies an increasing modeled backoff delay.
• Idempotency key: uses the same retry limit as naive retry, but assumes the server deduplicates repeated committed operations under a stable idempotency key.

How outcomes were measured

• Task success rate: proportion of episodes in which the policy receives a successful response within its attempt limit.
• Mean attempts: average number of tool attempts per episode.
• Retry amplification: mean additional attempts beyond the first attempt.
• Duplicate-side-effect episode rate: proportion of episodes that produce more than one modeled side effect.
• Mean duplicate side effects: average number of side effects beyond the first per episode.
• Mean latency: modeled latency steps, including attempt costs and policy backoff.

Results

All values below are generated by the published v0.1.0 simulator and aggregated over 6,000 episodes per policy. Lower duplicate-effect rates, retry amplification and latency are generally preferable; task success is preferable when higher.

Interpretation

The central failure mode is ambiguity between operation outcome and response outcome. A transient error response does not necessarily mean that the remote operation failed to commit. When the client retries without deduplication, a previously committed operation can be repeated. In this simulator, naive retry improves task success compared with no retry, but introduces duplicate effects in a material fraction of episodes.

Bounded backoff reduces the number of attempts and the mean number of duplicate effects relative to naive retry. However, its duplicate-side-effect episode rate is unchanged in this run. Backoff changes when retries occur; by itself, it does not establish whether an earlier attempt committed.

The idempotency-key policy demonstrates the value of server-side deduplication in the modeled setting. It produces the same task-success rate as naive retry and zero modeled duplicates, but has higher modeled latency because it uses the same retry count with exponential backoff. The zero-duplicate result depends on an idealised deduplication contract; it is not evidence that all real idempotency implementations provide exactly-once execution.

Limitations

• This is a deterministic synthetic simulator, not a measurement of deployed AI agents, payment systems or production tool APIs.
• The model does not implement real network partitions, concurrent requests, distributed transaction protocols, idempotency-key expiry, key-scope errors or database consistency anomalies.
• The idempotency policy assumes idealised server-side deduplication. Real systems must define key scope, retention, payload matching and behaviour for concurrent duplicate requests.
• Latency values are abstract simulation steps, not milliseconds or observed service response times.
• Policy behaviour is encoded in the benchmark. The findings describe these modeled rules and should not be generalised to every retry implementation.
• Further work should test real tool adapters with controlled response loss, commit-before-error fault injection and trace-based duplicate detection.

Reproduce the benchmark

The repository contains the scenario dataset, benchmark implementation, runner, episode-level results, policy summary, dataset card and automated tests. Run with Python 3:

python3 run_benchmark.py

python3 -m unittest discover -s tests -v

The published v0.1.0 run generated 6,000 scenarios, 24,000 policy evaluations and passed all eight automated tests.

Research note: ARAM v0.1.0 is a reproducible synthetic baseline. It does not establish real-world retry reliability, exactly-once execution or production payment safety. All reported values are generated by the published simulator.