← All insights

Research benchmark · Agentic AI

Agent Cancellation Propagation: what happens after an AI agent is told to stop?

Four cancellation policies tested across 6,000 synthetic workflows, measuring work that continues after a stop signal.

By Muzammal Hussain · 1 November 2026 · 6 min read

An agent workflow may involve several steps and tool calls. A user can cancel the task, a parent workflow can time out, or a budget can run out. But receiving a cancellation signal is not necessarily the same as stopping every downstream action. This benchmark asks a narrow question: under different cancellation policies, how much work continues after cancellation, and what side effects can still occur?

What was tested?

Agent Cancellation Propagation Benchmark (ACPB) v0.1.0 is a deterministic synthetic simulator. Its default run creates 6,000 scenarios using five fixed random seeds. Each scenario varies workflow depth, the step at which cancellation arrives, the number of in-flight calls, a possible side-effect location and the cancellation source. The cancellation sources are user cancellation, parent timeout and budget exhaustion.

Each scenario is evaluated under four policies, producing 24,000 policy-episode records:

• Cooperative: stops new tool dispatches immediately when cancellation is observed. Already in-flight work may still produce a modeled side effect.
• Polling: checks for cancellation periodically, allowing a limited amount of residual work before observing the signal.
• Cancellation-blind: continues the remaining workflow without responding to cancellation.
• Shielded section: completes a modeled protected section before observing cancellation.

How were outcomes measured?

• Post-cancellation dispatches: new tool dispatches after cancellation.
• Orphaned calls: calls still represented as orphaned under the simulator's policy rules.
• Cancellation latency: modeled steps between cancellation and the policy's stop point.
• Post-cancellation side-effect rate: proportion of episodes in which a side effect occurs after cancellation.
• Strict cancellation compliance: an episode counts as compliant only when it has both zero post-cancellation dispatches and zero post-cancellation side effects.

Results

All figures below are averages or rates across 6,000 synthetic episodes per policy. Lower values are preferable for dispatches, orphaned calls, latency and side effects; higher is preferable for strict compliance.

What do the results mean?

The cooperative policy illustrates why stopping new work and stopping all effects are different engineering requirements. It dispatched no new tools after cancellation in the model, but an already in-flight call could still reach a side-effect point. That is why its strict compliance rate was 81%, rather than 100%.

Polling had the lowest modeled side-effect rate, but it did not achieve strict compliance: its periodic check allowed residual dispatches. This is a useful distinction. A policy can reduce one undesirable outcome without satisfying a stricter definition of cancellation.

Cancellation-blind execution performed worst on dispatches, orphaned calls and latency. The shielded-section policy reduced new dispatches substantially compared with cancellation-blind execution, but completing a protected section did not guarantee that no side effect would occur after cancellation.

One caution applies to every number above. Each policy is a fixed rule in the simulator, so several results follow from the rule itself rather than from anything the benchmark discovered. Polling always lets one more call out when steps remain, so its strict compliance is 0% by construction. Cooperative latency is set to one step. The useful part is that the rules are written down and compared on the same scenarios; the next step is to test the same four policies on real tool-calling agents and see whether the measured behaviour matches the rules.

Limitations

• This is a deterministic, stylized synthetic simulator, not a measurement of deployed AI agents or production workflows.
• The model does not implement real concurrency, cancellation races, network timing, distributed task queues, rollback or framework-specific cancellation semantics.
• Policy behaviour is explicitly encoded in the simulator. The results therefore describe those rules, not every implementation of cooperative cancellation, polling or protected sections.
• Side effects are modeled events; they are not actual external transactions, emails, payments or tool executions.
• Runtime adapters, trace-based experiments and fault-injection tests are needed before drawing conclusions about real agent frameworks.

Reproduce the benchmark

The repository includes the scenario dataset, benchmark implementation, runner, episode-level outputs, policy summary, dataset card and automated tests. On macOS with Python 3, run:

python3 run_benchmark.py
python3 -m unittest discover -s tests -v

The default experiment generates 6,000 scenarios and evaluates all four policies, producing 24,000 policy-episode records. The published v0.1.1 code passed all six automated tests.

Research note: ACPB v0.1.1 is a reproducible synthetic baseline. It does not establish real-world cancellation reliability, production safety or a guarantee that a particular agent framework will stop work. All reported values are generated by the published simulator.