[wilke]

Fixes #77.

What happened (2026-10-08, refiner build on Argo)

• 14:45:05: the Designer's second call was dispatched. Shortly after, the host left the ANL VPN. The socket (140.221.15.46 → Argo:443) became half-open, and the client was httpx.Client(timeout=None), so the read blocked forever.
• ADR 0008 health probes ran (15:55, 16:01, 16:12, 16:33, 17:14). The first came 70 min in instead of 15, consistent with the Mac sleeping, since the monotonic clock pauses during sleep. Each probe failed with "Request timed out." before any response. The policy treats a failed probe as "preserve the original", so the call waited indefinitely.
• /build status showed state: refining, model_wait: null throughout. model_wait only tracks the Builder's in-process calls, and since Construct workflows through scoped Episodes and update coding/viewer interfaces #66 the work happens in refiner Run calls recorded under the refiner's Duet.

Changes

1. Dead connections fail. New agent/provider_http.py: provider clients use TCP keepalive (idle 60 s, interval 15 s, 4 probes, about 2 min to declare a vanished peer) and a 30 s connect timeout. Reads stay unbounded, because a live gateway may send nothing until a long completion finishes, and keepalive ACKs from a live peer keep the socket open. A dead peer surfaces as a read error, which Replace a model attempt whose connection dropped mid-response (ADR 0008 amendment) #76 replaces once. Used by the launch transport and the chat coding adapter.
2. Waiting heartbeat. A silent call records call_waiting every 120 s, with idle seconds, phase, whether a probe is active, and the time to the next probe. A stall is now visible in the ledger, and "supervisor stopped" can be told apart from "still waiting".
3. Unreachable provider ends the call. 3 consecutive health probes that get no response at all end the logical call with provider_unreachable and ModelCallRecoveryExhausted("Model provider unreachable …") instead of preserving it forever. "No response at all" means a connect failure, or a timeout while the probe was still in dispatching/request_dispatched, i.e. before response headers. A probe that received headers, or succeeded, resets the count, so a slow but live provider is unaffected.
4. /build status while refining. model_wait now describes the build's newest model call from the ledger: episode, state, phase, idle seconds (also as elapsed_seconds for the status line), physical attempt, endpoint, and response_seen. It uses a new DuetStore.events_of_type() cross-Duet read, filtered by owner_duet_id. On the real stalled ledger it reports:

   {"active": true, "episode": "launch", "state": "original_preserved", "phase": "request_dispatched",
    "idle_seconds": 10433.6, "response_seen": false, "physical_attempt": 1,
    "endpoint": "https://apps.inside.anl.gov/argoapi/v1"}

Tests

tests/agent/test_model_call_liveness.py, 8 tests:

• keepalive options;
• connect bounded and read unbounded;
• unanswered probes end the call as unreachable;
• a probe that got headers doesn't count;
• waiting heartbeats are recorded;
• status for a long-silent call;
• status follows the newest call;
• events_of_type across Duets.

Unchanged and passing: test_model_call_failed_attempt_replacement (7), test_episode_model_failure_reporting (5), test_launch_transport_streams (4), test_refinement_chat_coding (14).

test_build_refinement_job.py::test_one_build_enters_refinement_and_cancel_stops_same_job[materialized-build-linked-refiner] fails identically on clean main; it is pre-existing and unrelated.

Not addressed

On macOS the monotonic clock stops during sleep, so probe scheduling ignores sleep time. Keepalive (1) covers the network-loss case regardless.

🤖 Generated with Claude Code