Repository navigation
Project sandbox failure diagnostics into durable D1 observations - #369
Merged
Merged
Conversation
A failed publisher/execution sandbox process was undiagnosable: the container is reaped on terminal observation, so /logs returns an empty fallback, and the Worker's computed diagnostic was logged only to short-retention Workers Logs, never to the durable executor_observations.facts_json. The result: an exit-1 publisher failure left no recoverable cause (blocker-8/10 observability gap). - sandboxProcessDiagnostic now also classifies publisher failures (the plain `delivery publisher failed: <code>:<step>` line, step validated against the publisher runner's error codes) and execution-runner failures (the execution_attempt_result/failed structured log, reusing the analysis failure vocabulary), in addition to the existing analysis + bootstrap cases. - CloudflareSandboxProviderFact carries two optional flat strings, diagnosticKind and diagnosticDetail (bounded), added to both strict provider-fact schemas and threaded through the runtime client's parse+reconstruct. - On a failed observation the Worker sets these from the diagnostic (compact JSON detail), or, when unclassified, keeps a bounded raw stderr tail so an exit-1 still leaves a clue. The plugin copies them into ExecutionFact.facts, which persists to executor_observations.facts_json — queryable without the container and without a live tail. Scope: observability only. No change to publisher logic, the watchdog, or retry behavior. The container is still reaped on terminal (no sandbox leak); the cause now survives in D1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When a publisher or execution sandbox process fails, its container is reaped on the terminal observation, so
/v1/executions/{id}/logsreturns an empty fallback. The Worker already computed a diagnostic from stderr, but only logged it to short-retention Workers Logs — never to the durable, queryableexecutor_observations.facts_json. Net effect: an exit-1 failure left no recoverable cause (the blocker-8/10 observability gap).This projects the failure cause into D1 so it survives container reaping.
Changes
sandboxProcessDiagnosticnow also classifies:delivery publisher failed: <code>:<step>line (step validated against the publisher runner's error codes)execution_attempt_result/failedstructured log (reusing the analysis failureKind/failureStage vocabulary)CloudflareSandboxProviderFactcarries two optional flat stringsdiagnosticKind+diagnosticDetail(bounded), added to both.strict()provider-fact schemas (worker-emit + control-plane-parse) and threaded through the runtime client's parse+reconstruct.ExecutionFact.facts, which persists toexecutor_observations.facts_json.Tested
tsc --noEmit+eslint .clean.facts.executor_observations.facts_jsonnow records{"exitCode":"1","diagnosticKind":"execution_failure","diagnosticDetail":"{\"kind\":\"execution_failure\"}"}— recovered after the container was reaped (raw/logswas empty, as expected).Scope
Observability only. No change to publisher logic, the watchdog, or retry behavior. The container is still reaped on terminal (no sandbox leak); the cause now survives in D1.
Known follow-up (not in this PR)
The prod validation surfaced a deeper gap: the
execution_failurediagnostic had nofailureKind/failureStage— the execution runner emitted the failed result without a classifiedExecutionAttemptError.kind. So we now know which lane failed, not yet why. That's a runner-side classification gap, addressed separately.🤖 Generated with Claude Code