Skip to content

fix(sdk_dispatch): refuse scalar silence-threshold on all phases - #306

Open
oprince wants to merge 1 commit into
AI-native-Systems-Research:reflectivefrom
oprince:refuse-scalar-timeout
Open

fix(sdk_dispatch): refuse scalar silence-threshold on all phases#306
oprince wants to merge 1 commit into
AI-native-Systems-Research:reflectivefrom
oprince:refuse-scalar-timeout

Conversation

@oprince

@oprince oprince commented Aug 25, 2026

Copy link
Copy Markdown

Summary

  • Broaden the postmortem fix Agentic Systems Experimentation and Discovery (ASED) #2 (which refused bundle scalar recommended_turn_silence_threshold_seconds on design only) to refuse it on all three phases: design, execute_analyze, report.
  • Only the per-phase map form is honored now. An explicit {design, execute_analyze, report} entry means the author intentionally tuned that phase.
  • Motivated by an IOCR follow-up run (iocr-hrl-ocr-models-detection-word-unet-v2-cand-0001) that reproduced the same class of failure on execute_analyze instead of design: iter-2 rehearsal measured per-image OCR latency (~3.5s) and emitted a scalar 15s threshold, but iter-3 execute_analyze wraps a 10+ minute subprocess (6 policies × 25 repeats × 8MP-image benchmark). The SDK-event gap was 621.4s, so the 15s watchdog fired on every attempt and burned 71 minutes silently.
  • The scalar's assumption "portable across phases" doesn't hold: every dispatched phase wraps LLM-driven tool calls around a benchmark, and subprocess wall-time (not per-op timing) is what the watchdog actually sees. The bundle writer emitting the map form remains the durable upstream fix (postmortem fix Agentic Strategy Evolution: a three-loop methodology for optimizing multi-layer policy spaces #1).

Test plan

  • `pytest tests/test_experiment_spec.py::TestWatchdogReadsBundleOverride -v` (13 tests pin the resolver contract in both directions: scalar refused for every phase; map honored for every phase)
  • The two existing override-and-restore tests were converted to the map form — confirm their invariant is still covered
  • Full test suite: `pytest tests/test_experiment_spec.py`

Closes #305

…n all phases

SDK_SILENCE_FAILURE_POSTMORTEM described a 71-minute silent grind where
iter-2's bundle emitted a scalar
`recommended_turn_silence_threshold_seconds: 15` (measured on
execute-analyze's numpy benchmark ~3-4 ms/call) and iter-3's design
phase inherited that 15s watchdog — killing every LLM think turn on
the very first ThinkingBlock gap. The postmortem's fix AI-native-Systems-Research#2 refused
scalar overrides on design specifically.

A follow-up IOCR run
(iocr-hrl-ocr-models-detection-word-unet-v2-cand-0001) reproduced
the same class of failure on execute-analyze instead. Its iter-2
rehearsal measured per-image OCR latency (~3.5s) and emitted
`recommended_turn_silence_threshold_seconds: 15`, but iter-3
execute-analyze wraps a 10+ minute subprocess (6 policies × 25
repeats × 8MP-image benchmark) — the SDK-event gap between
UserMessage and the pre-tool HookEventMessage was 621.4s, so the
15s watchdog fired on every attempt and burned another 71 minutes.

The postmortem author's assumption "scalar is portable back to
execute_analyze" doesn't hold: every dispatched phase wraps
LLM-driven tool calls around a benchmark, and the subprocess wall-
time (not per-op timing) is what the watchdog actually sees.

Broaden the refusal: `_resolve_turn_silence_threshold` no longer
consults the bundle-side scalar override for any phase. Only the
per-phase map form is honored (priority 1) — an explicit
{design,execute_analyze,report} entry means the author intentionally
tuned that phase. The bundle writer emitting the map form remains
the durable upstream fix (postmortem fix AI-native-Systems-Research#1).

Tests: 13 tests in TestWatchdogReadsBundleOverride pin the resolver
contract in both directions (scalar refused for every phase; map
honored for every phase). The two existing override-and-restore
tests were converted to the map form so their invariant stays
covered.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant