Dialog turns terminate as provider_protocol_error after our own validators reject the answer: two session-engine exits bypass the model-output split and requeue to exhaustion #277

Activity

1. iamwavecut

   added

   agent:createdCreated by incident automationCreated by incident automation

   agent:queuedQueued for an automated incident investigationQueued for an automated incident investigation

   needs-triageIncident cause still requires investigationIncident cause still requires investigation

   needs-humanAutomated investigation requires the owner decisionAutomated investigation requires the owner decision

   on Oct 23, 2026
2. iamwavecut commented on Oct 25, 2026

   

   iamwavecut

   on Oct 25, 2026

   Last edited by iamwavecut

   OwnerAuthor

   More actions

   Problem

   A dialog task on the dialog worker queue reached terminal state failed after about two minutes of processing (about five seconds of queue wait), emitting the routing event all_attempts_exhausted for the dialog workflow with an empty route object; the signature has 46 recorded occurrences since mid-September and the sampled task is the newest. For this occurrence the scoped attempt capture retains two attempt_failed rows, both on the primary self-hosted dialog hop, with no failure reason on either row; a prior scoped capture of the same signature showed three rows covering every chain route exactly once. Both readings stay open and are code-verified at the pinned (deployed = HEAD) revision: (a) our own session-engine validator exits (empty-after-sanitization / raw-empty / skipped image tool, and outbound deliverability rejection) type the failure as a provider protocol error, which charges the provider circuit breaker, walks to the next candidate, and requeues the whole turn to the attempt cap, ending in the observed terminal event with provider and model unset — even though the split's own policy says a rejected model output leaves the breaker alone and is re-sampled on the same model; (b) a genuine chain-wide endpoint fault in the window produces the identical terminal signature. No retained reason string discriminates the two for this occurrence, so the deep stage must pull retry-stage job events or error strings before any patch is declared causal.

   External cause: possible. Code defect: possible.

   Observations

   • Terminal routing event all_attempts_exhausted for a dialog task on the dialog worker queue; the sampled task is failed, created-to-started about 5 seconds, started-to-completed about 121 seconds, and the snapshot's route object is empty.
   • The signature has 46 recorded occurrences since first seen mid-September; the sampled task is the newest occurrence and provably ran the current build (deployed revision equals repository HEAD equals the pinned base revision, restart count 0, no OOM kill).
   • Scoped attempt capture for this occurrence returns two attempt_failed rows, both on the primary self-hosted dialog hop, about 102 seconds apart inside the processing window; neither row retains a failure reason.
   • A prior scoped capture of the same signature returned three attempt_failed rows, one per chain route (two self-hosted lanes plus the external [configured resource]-pool tail route), in one round; none retained a failure reason and only one carried model attribution.
   • Host and database evidence is healthy (ample memory and disk, zero waiting locks, normal pool and connections), ruling out local resource starvation.
   • Queue counters show about 33590 completed versus 178 failed dialog tasks (about 0.5 percent) with a few processing and pending, so the routing path works for most tasks and the trigger is conditional.
   • Verified at the pinned revision (dialog session engine): when the final answer is empty, becomes empty after sanitization, or skips an explicitly requested image tool, and when the outbound deliverability validator rejects the answer, the turn reports a provider-protocol-class failure reason to the job-retry handler; in all these cases the provider answered and our own validation rejected or emptied what it wrote.
   • Verified at the pinned revision (dialog job-retry handler): every retryable reason requeues the whole turn up to the configured attempt cap, and on exhaustion emits all_attempts_exhausted with provider_id and model_id unset and last_retryable_reason repeating the class, which matches the incident snapshot's empty route object exactly.
   • Verified at the pinned revision (routed-attempt walker): a model-output rejection gets same-model re-sampling and never charges the circuit breaker (documented 'a rejected output leaves the breaker alone', with tests asserting the circuit stays closed), while protocol-class failures charge the breaker and walk to the next candidate, so the two validator exits take the wrong branch of the split's own policy.
   • Existing tests assert the terminal ledger reasons (sanitized_empty_exhausted, undeliverable_answer_exhausted) for these funnels, but no test asserts the terminal routing event's last_retryable_reason for either funnel, so the incident-facing classification is unguarded.

   Hypotheses

   • Misclassification funnel (our code): the model's answer was emptied by our sanitizer or rejected by our outbound validation on each hop, the session engine typed it as a protocol error, each hop charged its breaker and walked, and the job-retry handler requeued to the cap and emitted the observed terminal event; the complete path is verified in the deployed tree with no provider fault required, and the retained per-hop attempt rows on the primary hop fit one failing round of exactly this funnel.
   • Genuine chain-wide endpoint fault (external): the engaged routes really failed in the window (connect or decode failures), the classifier typed protocol errors correctly, attempts hopped and charged breakers, and the round or job budget exhausted with the identical terminal signature.
   • Config truncation (our code): an admin workflow-update request omitting the hop or wall field could revert the stored dialog budget to defaults via a default-substitution and unconditional-update path, bounding each round; the observed timings fit either wall value.
   • Both funnels coexist and alternate across the 46 occurrences; the about-two-minute window fits both an instant validator rejection after a long answer and a long endpoint failure.

   Supporting

   • The exact terminal signature (event type, workflow, empty route, protocol-class reason with ids unset) is reproducible in the deployed tree from the two session-engine validator exits through the job-retry handler, so a complete code path to the observed event exists without any provider misbehavior.
   • The two exits contradict the split's own documented policy in the walker: when the provider answered and our validators rejected the output, the intended class is the model-output one with same-model re-sampling, not a protocol-class failure that charges the breaker and walks to a weaker fallback.
   • A merged fix (Stop tool loops and slow thread history reads, keep memory alive through embedder bursts, route summaries to GPU2 #247) documents live empty-answer replay-to-exhaustion runs on the primary self-hosted dialog hop, confirming the funnel is exercised in production, not merely theoretical.
   • This occurrence's retained attempt rows sit on the primary self-hosted hop, the same hop Stop tool loops and slow thread history reads, keep memory alive through embedder bursts, route summaries to GPU2 #247 documents for the empty-answer replay storm.
   • Healthy host, runtime and database evidence plus the overwhelming dialog success rate rule out local resource exhaustion, crash loops and lock or pool starvation.

   Contradicting

   • No captured attempt record carries a failure reason, so neither genuine endpoint fault nor our validator rejection is proven as the terminal cause for this occurrence.
   • A pure our-code story is not established: an earlier occurrence of the same signature engaged all three chain routes with attempt_failed rows at every position, and without per-attempt reasons genuine endpoint faults producing the identical terminal event cannot be excluded.
   • A pure provider-outage story does not by itself explain the empty route object in the terminal event, which the verified job-retry path produces by repeating the reason with provider and model ids unset after our own classification chose the class.

   Related changes

   • PR Re-sample a rejected model before falling back to one without tools #151 (merged, in the deployed head) split model-output rejections from protocol errors and added walker-level same-model re-sampling; it migrated the adapter sites but not the two session-engine validator exits, which still emit the protocol class.
   • PR Stop tool loops and slow thread history reads, keep memory alive through embedder bursts, route summaries to GPU2 #247 (merged) documents and caps the live empty-answer replay storm on the primary self-hosted dialog hop but leaves the failure class and requeue-to-exhaustion behavior for empty answers unchanged.
   • PR Sample Gemma dialog turns as the model card recommends #262 (merged) changed dialog route sampling parameters; it is the newest dialog-behavior change before this occurrence and has no causal evidence either way.
   • PR Route memory workflows through independent cascades #98 introduced the multi-hop dialog cascade and wall-clock budget whose tail reachability these terminal events question.
   • PR Audit LLM tool parser parity #30 audited the response-decoding surface that produces genuine protocol-class failures, sharpening the split the two exits bypass.
   • PR Add OpenRouter free routing pool #24 added the external [configured resource] routing pool that forms the tail route engaged in an earlier occurrence of this signature.

   Missing

   • Per-attempt failure reasons or retry-stage job events for this occurrence: the error strings would discriminate the session-engine sites ('dialog answer became empty after sanitization', 'dialog answer rejected by outbound validation: ...', 'dialog provider returned no answer...') from a decode-failure string and attribute the observed class to one funnel.
   • Raw provider response bodies behind the occurrences; retention exists per the backend merge but no bodies are in scope.
   • Live database values of the dialog workflow's hop budget, wall clock, job-attempt cap and route role/ordering/weight fields at failure time, needed to judge whether the fallback tail was reachable and whether the stored budget was reverted to defaults.
   • Admin workflow-update request logs for the window, to confirm or exclude the default-substitution budget-revert vector.
   • A design decision for the deep stage: whether the raw-empty branch belongs to the model-output class (as the shared message rules map 'empty final text') or the protocol class (as one adapter family types it), and whether the two validator exits should reuse the model-output reason or a validator-specific reason.
   • Whether the operator-facing admin report should distinguish validator-exhaustion terminal events from endpoint protocol faults, so healthy providers stop appearing faulty.

   Acceptance

   • A regression test where a step provider answers with text that sanitizes to empty must end with a model-output-class (or validator-specific) last_retryable_reason in the job-retry terminal routing event, not the protocol class.
   • A regression test where the outbound deliverability validator rejects the answer must produce the same reclassification.
   • A regression test where the endpoint returns an empty or non-decodable body must still classify as the protocol class, preserving genuine protocol-fault semantics.
   • Deep-stage verification must pull this occurrence's retry-stage job events or error strings and attribute the observed reason to one of the two funnels before any patch is declared causal for the observed occurrences.

   Impact

   46 captured terminal events.

   Exact infrastructure identities and operational evidence remain in the private incident context available to the repair worker.
3. iamwavecut commented on Oct 28, 2026

   

   iamwavecut

   on Oct 28, 2026

   OwnerAuthor

   More actions

   Problem

   A dialog task on the dialog queue failed terminally with event all_attempts_exhausted, reason provider_protocol_error and an empty route object; the incident signature has 127 recorded occurrences. Independently re-verified in the repository at the deployed head (identical to base revision): two session-engine exits hard-code the protocol-error class for outcomes where the provider answered and our own validation rejected or emptied the answer (sanitizer collapse to empty at the raw-empty branch, outbound deliverability rejection), the job-retry handler requeues the whole turn per attempt and on exhaustion emits the observed terminal event with provider and model unset while repeating the protocol-class reason, and the shared message-rule classifier maps the same validator conditions to the model-output class, so the two exits bypass the split. A genuine endpoint protocol fault is not excluded: per-attempt error strings and routing attempt rows for the newest occurrence are not retained in scope, and the classifier's decode-failure path produces the identical terminal signature. Deep stage must attribute the observed reason to one funnel before any patch is declared causal.

   External cause: possible. Code defect: possible.

   Observations

   • Incident snapshot: terminal event all_attempts_exhausted with reason provider_protocol_error, workflow dialog, empty route object; 127 recorded occurrences of the signature per the incident record, and the dialog queue's failed counter is roughly 150 against about 32.3k completed (under 0.5 percent).
   • Scoped task evidence for the newest occurrence: dialog task on the dialog queue in state failed, about 19 seconds from creation to start and about 3 seconds from start to completion, a fast terminal window consistent with either an immediate validator rejection after a fast answer or a fast endpoint failure.
   • Scoped route-attempt evidence for the newest occurrence returns no_matching_evidence, and no per-attempt error strings or raw provider bodies are retained in scope, so the providers/models actually tried and the failing stage remain unknown.
   • Host and runtime evidence is healthy: deployed revision equals repository head and base revision, restart_count 0, no OOM kill, ample memory and disk, moderate load; the database reports zero waiting locks with an active pool, ruling out local resource starvation or a crash loop.
   • Verified in the deployed head (session engine, dialog_turn/session.rs): when the final answer is empty, becomes empty after response sanitization, or an explicitly requested image tool was skipped, the turn reports FailureReason::ProviderProtocolError to the job-retry handler; the outbound deliverability validator rejection does the same. In all these cases the provider answered and our own validation rejected what it wrote.
   • Verified in the deployed head (dialog_jobs/retry.rs): every retryable reason requeues the whole turn on the same queue up to the attempt cap, and on exhaustion the terminal routing event all_attempts_exhausted is emitted with provider_id and model_id None and detail.last_retryable_reason repeating the failure reason, matching the incident snapshot's empty route object and protocol-class reason exactly.
   • Verified in the deployed head (openplotva-llm retry.rs message rules): validator rejections such as 'empty final text', 'only protocol artifacts' and pathology verdicts map to the model-output class, while decode failures such as 'decode chat completion payload' and 'response body is empty' map to the protocol class; the two session-engine exits bypass this split by hard-coding the protocol class.
   • Verified in the deployed head: one adapter family types its own 'empty final text' rejection as a protocol-class error while the shared message rules map the identical phrase to the model-output class, so the layers disagree on the same condition and a reason string alone does not identify the funnel.
   • Existing tests assert the terminal ledger reason codes for these funnels (sanitized_empty_exhausted, undeliverable_answer_exhausted) and one routing-event test asserts last_retryable_reason only for a provider-unavailable funnel; no test asserts the terminal routing event's last_retryable_reason for the sanitizer-empty or undeliverable funnels, so the incident-facing classification is unguarded.

   Hypotheses

   • Misclassification funnel (our code): the model's answer was emptied by our sanitizer or rejected by our outbound validation, the session engine typed it as a protocol error, the turn requeued to the attempt cap, and the terminal event repeated the protocol-class reason; the code path to the observed signature is verified end-to-end in the deployed tree without any provider fault.
   • Genuine endpoint protocol fault (external): the endpoint returned empty or non-decodable bodies for these requests, the classifier correctly typed protocol errors, attempts hopped and charged breakers, and the chain or job budget exhausted with the identical terminal signature.
   • Both funnels coexist and alternate across the occurrences; the few-second final processing window fits both an instant validator rejection after a fast answer and a fast endpoint failure.

   Supporting

   • The exact terminal signature (event type, workflow, empty route, protocol-class reason) is reproducible in the deployed tree from the two session-engine exits through the job-retry handler, so a complete code path to the observed event exists without any provider misbehavior.
   • The two exits contradict the split's own policy established by the reclassification merge: when the provider answered and our validators rejected the output, the intended class is the model-output one with same-model re-sampling, not a protocol-class provider failure that charges the breaker and walks to a weaker fallback.
   • The recurrence continued under a deployed head in which the classifier and these two exits are unchanged since the reclassification merge, and the newest occurrence provably ran the current build (deployed revision equals head).
   • Healthy host, runtime and database evidence plus the overwhelming dialog success rate rule out local resource exhaustion, crash loops and lock starvation.

   Contradicting

   • A pure our-code story is not established: without the newest occurrence's per-attempt error strings or retry-stage job events, a genuine endpoint protocol fault that the classifier correctly reports cannot be excluded.
   • A pure external-endpoint story is not established: the verified validator-rejection funnel produces the identical signature without any provider fault, and nothing in the retained scope evidences an endpoint fault specifically.
   • The classifier inconsistency on the 'empty final text' condition between one adapter family and the shared rules means even a correct-looking reason string does not by itself identify the funnel.

   Related changes

   • Merge Re-sample a rejected model before falling back to one without tools #151 (merged 2026-09-17, in the deployed head) split model-output rejections from protocol errors and added walker-level same-model re-sampling for validator rejections; it migrated the adapter sites but not the two session-engine validator exits, which still emit the protocol class.
   • Merge Sample Gemma dialog turns as the model card recommends #262 (dialog route sampling parameters) is the newest dialog-behavior change before the continued recurrence; no causal evidence either way.
   • Merge Route memory workflows through independent cascades #98 set dialog to weighted-primary multi-hop routing with a workflow budget; these protocol-class terminal events question whether the fallback tail was reachable under that budget.
   • Merge Audit LLM tool parser parity #30 followed up a post-deploy dialog exhaustion audit by tightening the tool-call parsing surface, adjacent to the exhaustion surface but not to the failure-class split.
   • Merge Coordinate incident feedback, quota recovery, and actionable notifications #141 owns the incident-feedback and notification surface these terminal events flow into.
   • The prior analysis of this signature (issue Dialog turns terminate as provider_protocol_error after our own validators reject the answer: two session-engine exits bypass the model-output split and requeue to exhaustion #277) references merges extending the model-output repair verdict carry-back and capping a live empty-answer replay storm; those merge records are not in the provided history and could not be re-verified in this pass.

   Missing

   • The per-attempt error strings or retry-stage job events for the newest occurrence: the event's error field distinguishes the session-engine sites ('dialog answer became empty after sanitization', 'dialog answer rejected by outbound validation: ...', 'dialog provider returned no answer, response, or queued tool material') from a decode-failure string, which would attribute the observed reason to one funnel.
   • Retained routing attempt rows for the newest occurrence (the scoped attempt query returns no matching evidence), so the set of providers and models actually tried is unknown.
   • Raw provider response bodies behind the occurrences; retention exists but no bodies are in scope.
   • A design decision for the deep stage: whether the raw-empty branch belongs to the model-output class (the shared rules map 'empty final text' there) or the protocol class (one adapter family types it there), and whether the two validator exits should reuse the model-output reason or a validator-specific reason; this also decides whether the inconsistent adapter typing must be aligned.
   • Live database values of the dialog workflow's hop budget, wall clock, job-attempt cap and route role/ordering/weight fields at failure time, needed to judge whether the fallback tail was reachable.
   • Whether the operator-facing admin report should distinguish validator-exhaustion terminal events from endpoint protocol faults, so operators stop seeing healthy providers as faulty.
   • Content of the merges referenced by issue Dialog turns terminate as provider_protocol_error after our own validators reject the answer: two session-engine exits bypass the model-output split and requeue to exhaustion #277's body but absent from the provided history (repair-verdict carry-back, empty-answer replay cap), needed to confirm they leave the failure class unchanged.

   Acceptance

   • Deep-stage verification must pull the newest occurrence's retry-stage job events or error strings and attribute the observed reason to one of the two funnels before any patch is declared causal for the observed occurrences.
   • A regression test where a step provider answers with text that sanitizes to empty must end with a model-output-class (or validator-specific) last_retryable_reason in the job-retry terminal routing event, not provider_protocol_error.
   • A regression test where the outbound deliverability validator rejects the answer must produce the same reclassification.
   • A regression test where the endpoint returns an empty or non-decodable body must still classify provider_protocol_error, preserving genuine protocol-fault semantics.

   Impact

   127 captured terminal events.

   Exact infrastructure identities and operational evidence remain in the private incident context available to the repair worker.