[wilke]

Failure

This was the third live refiner build on Argo (2026-10-08, build_request_e25c9514…). The Materialization Implementer's streamed call failed after 164 s:

APIError … provider_error.type = RemoteProtocolError, http_status = None, retryable = True
"peer closed connection without sending complete message body (incomplete chunked read)"

The route's frozen recovery policy was retry_on_healthy_probe with max_replacements: 1. However, ADR 0008 replacement only covers attempts that go silent. An attempt that raises went straight to physical_attempt_failed → ModelCallFailed ("no alternate route was selected"), and the build ended interrupted 3 minutes after starting.

The same Argo drop also hit the chat coding adapter earlier today (fixed there in #75). Argo drops long streamed responses intermittently, so one replacement turns a lost build into a short delay.

Change

• agent/model_call_recovery.py: when the primary attempt raises, the supervisor calls a new replace_failed_primary(). It replaces the attempt only if all of the following hold:

  • the policy mode is retry_on_healthy_probe;
  • the failure is a dropped connection with no provider verdict: retryable, http_status is None, and the transport error type is in {RemoteProtocolError, ReadError, WriteError, ConnectError, ReadTimeout, APIConnectionError};
  • the same max_replacements budget used for silent calls isn't exhausted;
  • the call isn't cancelled.

  A replacement closes the failed attempt, backs off (5 s × attempt, capped by recovery_grace_seconds), and starts a new physical attempt on the same route. It's recorded as recovery_replacement_proposed with basis retryable_transport_failure and duplicate_server_work_possible: true. Only one result is ever admitted, as before.

• Unchanged:

  • provider error responses (an HTTP status such as 503 "overloaded", or an error event in the stream) still fail with no retry, which test_episode_model_failure_reporting requires ("Reporting must not introduce a retry");
  • preserve/disabled modes;
  • routes without a recovery record (legacy).

• ADR 0008: adds a dated amendment describing this case.

Tests

tests/agent/test_model_call_failed_attempt_replacement.py, 7 tests:

• a dropped stream is replaced and the result admitted;
• the budget is respected;
• preserve and disabled keep failing;
• non-retryable errors aren't replaced;
• a 503 provider verdict isn't retried;
• legacy routes keep their behaviour.

Unchanged and passing: test_episode_model_failure_reporting.py (5) and test_launch_transport_streams.py (4).

Note

The chat coding adapter (#75) has its own small retry loop, because Implementer coding sessions don't go through this supervisor. ADR 0008 prefers a single shared policy, so a follow-up could route coding-model calls through supervise_model_call.

🤖 Generated with Claude Code