Dialog retries exhaust before the configured fallback chain is reached

A dialog request failed after repeated attempts. The available evidence shows overload responses from primary routes and response-format failures from another route. The observed attempts did not reach the fallback tail expected by the configured routing design.

Observed behavior

Eleven failed attempts were recorded across repeated runs over approximately 75 seconds.
The attempted routes returned overload or response-format errors.
Each observed run reached at most three started attempts; the documented routing design allows five to reach the complete fallback chain.
Some attempt records did not resolve the selected model, limiting reliable attribution.
The sampled runtime and database metrics did not show a restart, memory kill, or lock/pool exhaustion. This does not by itself rule out every resource-related cause.

Questions to resolve

Was the fallback tail absent from the loaded configuration, filtered by capabilities, or skipped by routing logic?
Are response-format errors caused by an external endpoint or by our decoder?
Is incomplete model attribution a telemetry defect or inconsistent routing data?

External failures and a possible code/configuration defect must be assessed independently. A code change is not yet justified solely by these observations.

Relevant code and prior work

Review the route-selection policy, retry walker, capability filtering, loaded hop/time budgets, and attempt attribution. PR #98 describes the intended fallback reachability; PR #30 is relevant to response parsing.

Acceptance

Establish which candidates were eligible and why the remaining fallback routes were not attempted.
If a code defect is confirmed, reproduce it with synthetic route fixtures and verify fallback reachability within the configured budgets.
Verify attempt attribution with synthetic provider/model records.
Preserve existing retry, task persistence, and user delivery behavior.

Exact deployment, routing, provider, model, and infrastructure details are retained in the private incident context supplied to the assigned agent. They are intentionally absent from this public issue, including its hidden markup.

Problem

A dialog task on the dialog queue failed with terminal all_attempts_exhausted whose reason is provider_protocol_error, the 49th occurrence of this signature since it first appeared three weeks ago. The newest occurrence is later than the merge that split validator rejections from endpoint protocol faults, and the routing-classification files are unchanged from that merge to the current deployed head, so the recurrence ran under the split classifier and shows the split did not eliminate the signature. Verified in the deployed tree: two session-engine sites still hard-code the protocol-error class for outcomes where the provider answered and our own validation rejected or emptied the answer (sanitized-to-empty final answer, and outbound deliverability validation), and that misclassified reason is what the job-retry path writes into the terminal event. A second, external funnel remains possible: the self-hosted hop genuinely returning empty or non-JSON bodies, which the same classifier legitimately types as protocol errors. No retained attempt rows, error strings, or raw bodies exist for the newest occurrence, so which funnel produced it is unresolved.

External cause: possible. Code defect: possible.

Observations

The assigned dialog task on the dialog queue reached state failed; the incident snapshot records terminal event all_attempts_exhausted with reason provider_protocol_error and an empty route object, matching the session-engine job-retry terminal record, which carries no provider or model ids and the last retryable reason of the final attempt.
The signature has 49 occurrences from first seen on 2026-09-09 to last seen on 2026-09-30 16:12 UTC; the prior retained occurrence is 2026-09-17, so at least one new occurrence exists on 2026-09-30, twelve days after the reclassification merge (2026-09-17 20:51 UTC).
No commit between that merge and the current deployed head touches the routed-attempt walker or the retry classifier, so the deployed classification code is identical to what the newest occurrence ran, assuming the container was running a build from that interval.
Retained prior-occurrence evidence shows five attempt_failed routing rows about ten seconds apart over roughly forty-six seconds, all on the single self-hosted dialog hop and all classified provider_protocol_error, with no fallback-hop rows in the retained scope; the rows are signature-scoped, not strictly task-scoped, since some timestamps fall outside the task window.
Verified in the deployed tree (session engine): when the final answer is empty or becomes empty after sanitization, and when the outbound deliverability validator rejects the answer, the turn reports FailureReason::ProviderProtocolError to the job-retry handler, although in both cases the provider answered and our own validation rejected what it wrote.
Verified in the deployed tree (job-retry handler): every retryable reason requeues the whole turn on the same queue up to the configured job-attempt cap; on exhaustion it emits the all_attempts_exhausted routing event whose detail repeats that reason, so both validator rejections and endpoint decode failures surface as the identical terminal signature.
Verified in the deployed tree (walker): a protocol-class failure charges the provider circuit breaker and walks to the next candidate, and the walker's own terminal all_attempts_exhausted event carries the last attempt's reason; only model-output rejections get same-model re-sampling, and only at the walker layer.
The reclassification merge introduced ModelOutputRejected with same-model re-sampling for the adapter-level final-answer rejection and message-rule mappings for decode failures; the two session-engine validator sites were not migrated to the new reason.
Host, runtime and database evidence is healthy: zero restarts, no OOM kill, ample memory and disk, zero waiting locks, normal pool and connection counts, and the deployed revision equals the repository head.
Queue counters show roughly ten thousand completed versus roughly twenty failed dialog tasks, so dialog routing succeeds overwhelmingly and this is a residual failure mode, not an outage.

Hypotheses

The newest occurrence is our code misclassifying validator rejections: the model's answer was emptied by our sanitizer or rejected by our outbound validation, the session engine typed it as a protocol error, the turn requeued to the attempt cap, and the terminal event repeated the protocol-class reason; the earlier reclassification fixed only the adapter site, leaving this funnel producing the same signature.
The self-hosted endpoint genuinely returned empty or non-JSON bodies for these requests; the classifier then correctly types protocol errors, attempts hop and charge the breaker, and the chain or the job budget exhausts with the same terminal signature.
Both funnels coexist and alternate across the 49 occurrences; the prior occurrence's rows and the current healthy host state do not distinguish them.

Supporting

The exact terminal signature (event type, workflow, empty route object, protocol-class reason) is reproducible end-to-end in the deployed tree from the two session-engine validator sites through the job-retry handler, so a code path to the observed event exists without any provider fault.
The two validator sites contradict the documented split introduced by the reclassification merge: the provider answered and our validators rejected the output, which by that policy is a model-output rejection deserving same-model re-sampling, not a protocol-class provider failure.
The recurrence after the reclassification merge, with classifier files unchanged since, is consistent with a remaining misclassification funnel rather than the fixed one.
The prior occurrence's five protocol-class rows confined to one self-hosted hop, with no decode detail retained, fit both funnels; nothing in the retained scope evidences an endpoint fault specifically.
Healthy host, runtime and database evidence rules out local resource exhaustion, crash loops, and lock or pool starvation as the cause.

Contradicting

A pure our-code story is not established: no error strings, raw bodies, or retained attempt rows exist for the newest occurrence, so it may be a genuine endpoint protocol fault that the classifier correctly reports.
A pure external-endpoint story is not established: the verified validator-rejection funnel can produce the identical signature without any provider misbehavior, and the pre-fix analysis already proved this conflation produced earlier occurrences.
The reclassification merge postdates every occurrence known to the earlier analysis, but the newest occurrence postdates the merge, so the earlier conclusion that the deployed fix was merely unverified is now superseded by an observed recurrence under the fixed classifier.

Related changes

The reclassification merge (merged 2026-09-17, present in the deployed head) split model-output rejections from protocol errors and added walker-level same-model re-sampling; it did not migrate the two session-engine validator sites, which still emit the protocol class.
The cascade merge that gave the dialog workflow its multi-hop fallback chain and wall-clock budget, whose reachability the two prior issues questioned.
The parser-parity audit of the response-decoding surface and the duplicated-response-block rejection fix, both touching how protocol-class failures are produced from provider payloads.
The provider-attribution fix for routed events, relevant because the prior occurrence's self-hosted rows lacked resolved model attribution.
The incident-feedback and notification coordination merge, which owns how these terminal events surface to operators.
The admin workflow-update path that silently defaults the hop budget and wall clock when a payload omits them, a previously verified truncation mechanism still capable of shortening chains.

Missing

The per-attempt error strings or job events for the 2026-09-30 occurrence: without them the protocol-class reason cannot be attributed to a decode failure versus a sanitizer or deliverability rejection; the deep stage should pull the job's retry-stage events, whose error field distinguishes the three sites.
Retained routing attempt rows for the newest occurrence (the scoped attempt query now returns no matching evidence), so the set of providers and models actually tried is unknown for it.
Raw provider response bodies behind the prior occurrence's five protocol-class rows; the retention feature exists but no bodies are in scope.
The exact build running at the 2026-09-30 failure time: the currently deployed image is a same-day merge made hours after the failure, and the failure predates it; the container is not observed to have restarted since.
Live database values of the dialog workflow's hop budget, wall clock, job-attempt cap, and the route's enabled, role, ordering and weight fields, needed to say whether the fallback tail was reachable at failure time.
Whether the session engine should map the two validator sites to the existing model-output reason or a new validator-specific reason, a design decision for the deep stage before patching.

Acceptance

A regression test where a step provider answers with text that sanitizes to empty must produce the model-output or validator-specific failure reason, not provider_protocol_error, in the job-retry terminal event's last_retryable_reason.
A regression test where the outbound deliverability validator rejects the answer must produce the same reclassification.
A regression test where the endpoint returns an empty or non-decodable body must still classify provider_protocol_error, preserving genuine protocol-fault semantics.
Deep-stage verification pulls the newest occurrence's job retry-stage events or error strings to attribute the observed reason to one of the two funnels before any patch is declared causal.

Impact

49 captured terminal events.

Exact infrastructure identities and operational evidence remain in the private incident context available to the repair worker.

Problem

A recurring dialog-workflow exhaustion: three retry rounds each start four distinct targets spanning the whole chain; the weighted primary tier rejects with overload while the remaining primary target and the self-hosted fallback tier fail at the protocol level. Host/runtime health and majority dialog success point to external capacity stress, and the fallback tail is demonstrably reachable, contradicting the unreachable-tail premise of the closed prior issues. Unresolved code-side question for the deep stage: the deployed design declares five hops, but chain selection deduplicates by provider+model identity and the terminal fallback never appears in any round, so the four-of-five hop pattern could be a genuine chain-construction defect, a legitimately smaller eligible set, or stored-budget drift; live routing-table data is required to decide.

External cause: confirmed. Code defect: possible.

Observations

A dialog workflow task on the dialog queue terminated with the terminal all_attempts_exhausted event carrying last retryable reason capacity_unavailable; the signature has recurred 179 times.
The scoped capture holds 12 attempt_failed rows forming three retry rounds spaced seconds apart within a 25-second window; each round starts the same four distinct targets.
Per-round pattern: two weighted primary-tier models reject with provider_overloaded while the remaining primary target and the self-hosted fallback tier fail with provider_protocol_error; every visible row is a per-attempt provider failure, not a selection-side stop event.
The fallback tier is reached and attempted in every round, so chain-truncation and unreachable-tail explanations do not fit this capture.
Rows for the self-hosted fallback hop carry provider attribution but no model attribution; external-tier rows carry both.
Host and runtime samples are healthy: zero restarts, no OOM kill, ample memory and disk, moderate load; database locks show zero waiters and the pool is not exhausted.
Queue aggregates show roughly 10k completed versus 212 failed dialog tasks in the same window, while an unrelated background workflow queue shows ~14k failed concurrently.
Only four of the designed five hops start per round: migrations/179_memory_routing_cascades.up.sql sets the dialog retry_max_hops to 5, stating three weighted primaries plus two fallback tiers require five started attempts.
Repository verification at base HEAD: chain selection in crates/openplotva-llm/src/router/policy.rs deduplicates targets by the (ProviderId, ModelId) pair, so an identical provider+model appearing both as a primary and as the terminal fallback is removed from the chain.
Repository verification at base HEAD: the capacity-aware walker in crates/openplotva-app/src/routed_attempts.rs bounds started attempts by route.retry.max_hops, applies a required-capabilities filter to the selected chain, and charges the retry wall clock only for attempt execution (pool slot-wait excluded); no capability-filter, pool-wait, breaker-open, deadline or no-candidate events appear in the scoped rows.
Repository verification at base HEAD: the admin workflow-update path (crates/openplotva-app/src/lib.rs around lines 8487-8488) silently defaults retry_max_hops to 3 and retry_wall_ms to 60000 when the payload omits the retry fields; observed rounds start four hops so this vector is not the current cause but remains latent.
The working tree is clean; no existing patch is present. Base HEAD is the merge of the change making a busy primary pool be waited for instead of falling back instantly.

Hypotheses

Primary driver is external: simultaneous capacity rejection at the weighted primary tier plus protocol-level failures at the remaining tiers during provider-side stress bursts; co-occurring mass failures on the unrelated background workflow queue support broad provider-side stress.
Shared client-side fault: all tiers failing concurrently with fallback failures classified as protocol errors (empty or undecodable responses) could be one request-side payload or response-decoding defect rather than independent provider incidents; unproven because the dialog queue keeps completing most tasks.
Only four of the designed five hops start per round because the terminal fallback target is deduplicated out of the chain by the (provider, model) identity rule when it equals a primary target, or because the live eligible set is four (disabled or capability-filtered terminal fallback), or because the stored hop budget drifted to four; unproven without live routing-table data.

Supporting

Every visible attempt row is a per-attempt provider failure (overload on both weighted primaries, protocol errors on the remaining tiers); the terminal exhaustion event only aggregates them.
Chain reachability is directly observed: fallback-tier candidates start after the primary failures in every round.
Aggregate dialog success plus burst-correlated failures argue for transient external stress rather than a deterministic client-side defect.
The deployed migration defines the designed retry breadth as five hops for the dialog workflow, and its comment states five started attempts are required for full chain reachability.
The (provider, model) dedup rule and the admin omit-default of three hops are both verifiably present in the base code.

Contradicting

The earlier explanation that the fallback tail is unreachable or the loaded chain is truncated is contradicted for this capture: fallback-tier candidates start in every round.
The admin omit-default reset to three hops is contradicted as the cause of this capture: each round starts four distinct targets, so the effective budget is at least four.
No pool-wait, breaker-open, deadline or no-candidate events appear in the scoped subset, so selection-side stop mechanisms are not directly observed here.
If the failures were one deterministic client-side defect, the dialog queue's large completed count in the same period would be hard to explain.

Related changes

Merged cascade change setting the dialog workflow to five hops with an ordered fallback tail (PR Route memory workflows through independent cascades #98) defines the behavior this run executes.
Later GPU-host cutover migrations re-identified the model serving the dialog fallback tier, so the live assignment rows may no longer match the cascade migration's original identities.
Merged at base HEAD: a busy primary pool is now waited for instead of falling back instantly, changing fallback timing in this exact path (fix/dialog-primary-slot-wait merge).
Admin workflow-update path still silently defaults the hop budget to three and the wall clock to 60 seconds when the payload omits the retry fields, unmitigated at the base.
Prior incident-feedback and quota-recovery work for this same signature delivered no routing, walker or chain-loading change (PR Coordinate incident feedback, quota recovery, and actionable notifications #141).
Merged attribution fix for recorded routing events (PR Fix inconsistent LLM provider attribution (vram.cloud tagged as aifarm) #11); attribution is complete for external tiers in this capture, though the self-hosted hop still lacks model attribution.

Missing

Live database values of the dialog workflow's effective retry_max_hops and retry_wall_ms, plus the enabled, role, fallback_order and capability flags of all dialog assignment rows, to decide between dedup, disabled or filtered terminal fallback, and budget drift.
The full routing event stream for the failing runs beyond the capped 12-row subset, including terminal detail fields such as failed_attempts and last_retryable_reason per run.
Raw provider response bodies or error text behind the provider_overloaded and provider_protocol_error classifications, to test a shared decoding defect.
Admin API request history around the failure window, to test the omitted-field silent budget-reset vector.
Whether the concurrent mass failures on the background workflow queue share the same time window and provider tiers.
Model attribution for the self-hosted fallback hop rows, to complete per-attempt attribution and confirm which model serves the ordered fallback.

Acceptance

Deep stage: with live routing-table data and the full event stream, enumerate the candidates eligible for each failing run and confirm whether four started hops matches the eligible set after dedup and capability filtering, or reflects stored-budget drift.
Deep stage: reproduce the round pattern with a synthetic five-candidate chain where the weighted primaries return overload and the remaining tiers return undecodable responses, and assert the walker starts every distinct eligible target before terminal exhaustion, including the case where the terminal fallback shares a provider+model identity with a primary.
Deep stage: verify the admin workflow-update path preserves an existing five-hop budget when the payload omits retry fields.

Impact

179 captured terminal events.

Exact infrastructure identities and operational evidence remain in the private incident context available to the repair worker.

Problem

A dialog task on the dialog queue failed terminally with all-attempts-exhausted surfaced under the overload reason class. The scoped capture shows twelve attempt-failure rows forming four retry rounds seconds apart; every round starts exactly three attempts (both weighted models of the primary external route fail with the overload class, then the first self-hosted fallback hop fails with a protocol-class error) and stops, although the converged configuration documents a five-hop dialog fallback chain. The rounds match the job-level requeue layer, so the truncation sits inside each routed walk. Primary-route overload is directly observed and independently sufficient for terminal failure of those attempts, so the external cause is confirmed. However, verified code mechanisms can silently truncate a walk to three hops (an admin partial update that defaults omitted retry fields to three hops / sixty seconds, a pre-walk capability filter that silently drops chain candidates on image-bearing turns, and a live routing table that may lack the tail assignments), so a code defect remains possible pending live configuration evidence.

External cause: confirmed. Code defect: possible.

Observations

Scoped capture: one dialog task on the dialog queue ended terminal failed with the all-attempts-exhausted event class surfaced under the overload reason; the failure signature has recurred with a large count over an extended window.
Twelve attempt_failed rows form four retry rounds ten to fifteen seconds apart; every round starts exactly three attempts: both weighted models on the primary external route fail with the overload class, then the first self-hosted fallback hop fails with a protocol-class error; no fourth or fifth hop is ever started.
The rounds are spaced like the job-level requeue layer (a dialog job requeues on retryable provider errors with a documented default attempt limit of five, covered by dialog_jobs tests), and each round's three attempts share one timestamp, so the three-attempt truncation sits inside each routed walk rather than in the job retry counter.
Attempt rows for the first self-hosted fallback hop carry no model identifier while primary-route rows do; class attribution (overload vs protocol) is consistent across all rounds.
Task runtime for the captured run is a few seconds and the four rounds span well under a minute, so the wall-clock retry budget cannot explain the three-attempt truncation even at the smallest documented setting.
Host and runtime are healthy: the deployed revision equals the repository base, running, zero restarts, no OOM kills, ample memory and disk; database pool and lock counters show no starvation.
Verified at the pinned revision in the routed run loop: the hop budget is read from the live workflow route row, the selection pass silently filters chain candidates by required capabilities with no event for dropped hops, already-attempted provider/model pairs are skipped silently within a run, and the pass loop breaks once started attempts reach the live hop budget - so a live budget of three, a three-candidate chain, or a capability-filtered chain each produce exactly the observed three-attempt rounds.
Verified at the pinned revision (admin routing handler): update_workflow defaults omitted retry_max_hops to 3 and retry_wall_ms to 60000, so a partial admin update that omits these fields silently rewrites a converged five-hop budget back to three.
Verified at the pinned revision (routing cascade migration): the converged dialog workflow is set to five hops / 180000 ms with tail fallback assignments defined there, so the expected database state does not truncate at three.
Verified at the pinned revision (dialog runtime): only dialog turns carrying images require the vision capability, so the silent capability filter engages only for multimodal turns.
The repository working tree at the pinned base revision is clean; there is no in-progress patch to continue, and the deployed revision matches the base.

Hypotheses

External cause (confirmed): the primary external route is genuinely overloaded - both of its weighted models fail with the overload class on every round, which alone justifies terminal failure of those attempts.
Truncation A (mechanism verified at the pinned revision, live trigger unconfirmed): a partial admin workflow update that omitted the retry fields silently rewrote the live hop budget to the default three.
Truncation B (mechanism verified, trigger unconfirmed): the captured turns carried images, so the pre-walk capability filter silently dropped the tail chain candidates that lack the vision capability.
Truncation C (open): the live routing table no longer contains the documented tail fallback assignments (disabled, deleted, or reordered), leaving only three viable candidates.
Attribution gap (open): the protocol-class failure on the first self-hosted hop may be an endpoint-side protocol break rather than a request-encoding defect in our code.

Supporting

Every observed round starts exactly three attempts and stops, matching a three-hop budget, a three-candidate chain, or a capability-filtered chain equally well; the attempt pattern alone does not discriminate.
The run loop's hop-budget break is a verified code mechanism that produces exactly the observed truncation when the live budget row reads three.
The admin update path's omitted-field defaults of three hops / sixty seconds are a verified silent-rewrite mechanism that would set that live value.
The converged migration and prior reports document walks over more than three distinct candidates for this workflow, so three is not the expected budget.
Primary-route overload is directly observed in all twelve attempt rows and independently justifies terminal failure of those attempts.
The overload class is classified from provider response status/text and the protocol class from decode/empty-body conditions in the response decoder path, so both classes originate in provider responses.
Host and database evidence rules out local resource exhaustion, restarts, and lock or pool starvation as the cause of the truncation.

Contradicting

The converged migration sets this workflow to five hops / 180000 ms, so the default database state should not truncate at three unless something rewrote it.
Thousands of dialog tasks complete versus a low hundreds of failed count, so the fallback chain usually completes; a systematic code truncation would be expected to fail far more often.
Overload on the primary route plus a protocol failure on the first self-hosted hop could exhaust a walk without any truncation defect if the remaining tail candidates were legitimately unavailable.
No empty-selection or circuit-cooldown event appears in the capture, so an explicit selection-empty explanation is not evidenced.

Related changes

pr#98 established the converged five-hop / 180000 ms dialog retry budget and the tail fallback ordering whose migration defines the expected state the observed rounds violate.
pr#30 fixed an earlier dialog-exhaustion root cause by restoring tool-parser parity that had produced protocol-class failures; the same failure class now appears on the first self-hosted hop.
pr#11 fixed earlier provider-attribution inconsistencies between routes; the current self-hosted-hop attempt rows lack a model identifier, so attribution hygiene is again in question.
issue#131 was a synthetic canary exercising the same terminal all-attempts-exhausted class in an injected fixture, confirming the exhaustion path is reachable in tests.
The base/deployed revision head merges the dialog primary-slot-wait fix (visible in git log at the pinned base); that wait path interacts with the same run loop, but the observed rounds show no slot-wait or busy-skip events.

Missing

The live workflow row at incident time (effective hop budget and wall clock) to distinguish a truncated budget from a truncated candidate chain.
A live routing-table snapshot (assignments, roles, fallback ordering, enabled state, model capabilities, duplicate provider entries) to confirm or refute the missing-tail hypothesis.
The stored payloads of the routing events (selected-chain length, hop budget, breaker state) rather than the scoped per-attempt summaries, which would directly show whether selection returned three or five candidates.
Admin workflow-update request bodies or audit records around the incident window to confirm or refute a default-rewrite of the retry fields.
Whether the failing dialog turns carried images or other multimodal input that would engage the vision capability filter.
Server-side error detail and model attribution for the protocol-class failures on the first self-hosted hop, to separate an endpoint-side break from a request-encoding defect.

Acceptance

Deep stage: reproduce a five-candidate chain in a synthetic fixture and show the walk starts at most budget-many hops; a truncated budget or filtered chain must be observable in the recorded events.
Deep stage: an admin workflow update that omits retry_max_hops/retry_wall_ms must not silently change the persisted budget, or the truncation mechanism is otherwise refuted by live configuration evidence.

Impact

246 captured terminal events.

Exact infrastructure identities and operational evidence remain in the private incident context available to the repair worker.

Problem

A dialog task on the dialog worker queue terminated with the terminal all-attempts-exhausted event after retry rounds that each started only the same three routes of the designed five-hop chain; the fallback-tail routes were never started. External failures are directly observed on three independent provider infrastructures, but no captured attempt record carries a failure reason, so external contribution is possible yet unattributed. A concrete code-level truncation vector is verified at the pinned revision (also the deployed and running revision): the admin update_workflow handler substitutes hardcoded defaults of 3 hops and a 60000 ms wall whenever those optional fields are absent from the request body, the storage-layer UPDATE overwrites both columns unconditionally, and the admin UI coerces empty inputs to the same defaults, so any partial admin update would silently revert the migrated 5-hop / 180000 ms dialog budget. Whether such an update occurred in the incident window is not in evidence.

External cause: possible. Code defect: possible.

Observations

Terminal all_attempts_exhausted event for a dialog task on the dialog worker queue; the sampled task is failed after job-level retry rounds that each start only the same three routes of the designed chain.
Scoped attempt capture shows attempt_failed records in a fixed pattern on the same three routes across rounds; no fourth or fifth chain candidate is ever started.
No captured attempt record carries a failure reason, and some engaged routes lack model attribution, so the failure class (endpoint vs decoder) is unattributed.
Host and runtime are healthy: service running, zero restarts, no OOM kill, ample memory and disk; no waiting database locks or pool saturation in evidence.
Deployed image revision, running revision and repository HEAD are all the pinned base revision, so deployment drift is ruled out.
Dialog queue counters show a large completed majority versus a small failed minority, so the routing path works for most tasks and the trigger is conditional.
Verified at the pinned revision: the routed run loop reads the hop budget from the live workflow row (floored at one hop) and its wall budget from the same row (crates/openplotva-app/src/model_routing.rs, RetryBudget construction).
Verified at the pinned revision: the admin update_workflow handler substitutes hardcoded defaults of 3 hops and a 60000 ms wall whenever those optional fields are absent (crates/openplotva-app/src/lib.rs, update_workflow arm using unwrap_or(3) and unwrap_or(60_000)).
Verified at the pinned revision: the storage-layer SQL_UPDATE_WORKFLOW unconditionally overwrites full_routing, retry_max_hops, retry_wall_ms and enabled, so a defaulted call reverts stored budgets (crates/openplotva-storage/src/llm_routing.rs).
Verified at the pinned revision: the admin UI save-budget control sends both budget fields but coerces an empty input to the same 3-hop / 60000 ms defaults (web/admin/index.html, Save budget button).
Verified at the pinned revision: the cascades migration sets the dialog workflow to a 5-hop budget and a 180000 ms retry wall, and a code path re-asserts that same dialog budget, confirming the designed chain length of five.
Verified at the pinned revision: the startup backfill that re-asserts the dialog 5-hop / 180000 ms budget (backfill_memory_routing_cascades in the startup sequence) is guarded to run once, so a mid-lifetime budget truncation would not be auto-repaired while the service keeps running.

Hypotheses

Config truncation: a live update_workflow request omitted the hop or wall field (or the admin UI sent an empty field), so the verified default-substitution path reverted the stored dialog budget to 3 hops / 60000 ms, cutting the chain short before terminal exhaustion.
External: the three engaged routes genuinely failed in every round and the pass ended because eligible candidates were exhausted at the stored three-hop budget rather than the designed five.
Chain eligibility: capability filtering, disabled assignments or circuit-breaker state removed the fallback tail from the eligible candidate set, bounding each round to three attempts.
Client-side shared defect: a request-shape or response-decoding defect fails protocol-class on multiple independent providers, which the retry path counts as hop-consuming failures.

Supporting

attempt_failed events on three distinct independent routes in every round are direct evidence that the external endpoints of those routes were engaged and failing.
Exactly three started attempts per round against a designed five-candidate chain matches what a three-hop stored budget would produce.
The admin default-substitution path is verified at the pinned revision (handler defaults, unconditional storage UPDATE, UI empty-input coercion) and is a concrete code-level route back to a three-hop budget.
A round spans far less time than either the designed or default wall budget, so the wall clock would not truncate a round at three attempts.
Sibling issue threads record the identical multi-round three-route pattern across earlier occurrences, so the truncation is recurring rather than a one-off.

Contradicting

A pure provider-outage explanation does not account for the absence of any fourth or fifth route attempt unless the eligible set or budget was itself reduced.
Simultaneous failures across independent provider infrastructures in the same narrow window are unusual as coincidence, weakening an external-only explanation.
No captured attempt record for this occurrence carries a failure reason or full model attribution, so provider-side fault is asserted but unproven.
The live workflows row values during the incident window are not in evidence, so the stored-budget truncation hypothesis is not yet confirmed for this occurrence.
A sibling issue comment records a different occurrence where four distinct targets spanning the whole chain were started, which a fixed three-hop stored budget would not allow.

Related changes

PR Route memory workflows through independent cascades #98 (merged) established the five-hop dialog cascade and 180000 ms retry wall via the cascades migration; its stated motivation was the three-hop cutoff this incident reproduces, and the admin default-substitution path was not adjusted with it.
PR Add OpenRouter free routing pool #24 (merged) introduced the external [configured resource]-pool fallback route engaged in every round, including quota/cooldown gating that could bound the eligible chain.
PR Replace GPU2 reasoner with Ternary Bonsai #93 (merged) replaced a fallback route's model identity, plausibly explaining missing model attribution on some attempt records.
PR Re-sample a rejected model before falling back to one without tools #151 (merged) reclassifies validator rejections as a distinct event rather than attempt_failed, so the observed records are not self-inflicted rejections.
PR fix(dialog): reject duplicated response blocks #106 (merged) routes malformed final dialog answers through the retry/failover path, the mechanism by which response-format failures consume hop budget.
PR Backend lanes and hourly memory runs #251 (merged) reworked backend lanes so a request moves to a fallback only when the primary fails or waiting would exceed its reserve; it touches when tail hops engage but not the hop budget.

Missing

The live workflows row for dialog during the incident window (effective hop and wall budgets) and workflow assignment state for the unattempted tail routes.
Per-attempt failure reasons and full model attribution; the hop, wall-clock and deadline values actually in effect for the terminal run.
The complete routing event stream for the failing turn (re-sample, capacity-pool wait, no-candidates, circuit-open events) to distinguish a budget cut from an eligibility cut.
Whether any admin update_workflow request omitted the hop or wall fields in the window; admin request logs were not available.
Circuit-breaker and capacity-pool state for the engaged routes and the unattempted tail during the window.
Whether the failing turn carried media requiring capabilities that filter candidates from the chain.
Whether the observed failures originate at the external endpoints or in our response decoder; raw provider payloads are out of scope.
The stored dialog workflow row values as of now (hop budget and retry wall), which would directly confirm or refute the truncation hypothesis without raw payloads.

Acceptance

Read the effective dialog workflow row (hop budget, retry wall) and tail assignment state during an occurrence window to separate budget truncation from eligibility cuts from genuine external failure.
If the default-substitution defect is confirmed causal, add a synthetic regression proving a partial admin update_workflow request preserves stored hop and wall budgets instead of reverting them to the hardcoded defaults.
Add a synthetic five-candidate regression proving eligible fallback routes are attempted before terminal exhaustion under the configured budgets.
Preserve existing retry limits, task persistence and user delivery behavior in any fix.

Impact

42 captured terminal events.

Exact infrastructure identities and operational evidence remain in the private incident context available to the repair worker.

Dialog retries exhaust before the configured fallback chain is reached #137

Description

Observed behavior

Questions to resolve

Relevant code and prior work

Acceptance

Activity

iamwavecut commented on Sep 9, 2026

Problem

Observations

Hypotheses

Supporting

Contradicting

Related changes

Missing

Acceptance

Impact

iamwavecut commented on Sep 9, 2026

Problem

Observations

Hypotheses

Supporting

Contradicting

Related changes

Missing

Acceptance

Impact

iamwavecut commented on Sep 13, 2026

Problem

Observations

Hypotheses

Supporting

Contradicting

Related changes

Missing

Acceptance

Impact

iamwavecut commented on Sep 17, 2026

Problem

Observations

Hypotheses

Supporting

Contradicting

Related changes

Missing

Acceptance

Impact

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions