Skip to content

Project sandbox failure diagnostics into durable D1 observations - #369

Merged
evilstar9527 merged 1 commit into
mainfrom
fix/publisher-failure-observability
Aug 28, 2026
Merged

evilstar9527 merged 1 commit into
mainfrom
fix/publisher-failure-observability

Conversation

@evilstar9527

Copy link
Copy Markdown
Owner

Summary

When a publisher or execution sandbox process fails, its container is reaped on the terminal observation, so /v1/executions/{id}/logs returns an empty fallback. The Worker already computed a diagnostic from stderr, but only logged it to short-retention Workers Logs — never to the durable, queryable executor_observations.facts_json. Net effect: an exit-1 failure left no recoverable cause (the blocker-8/10 observability gap).

This projects the failure cause into D1 so it survives container reaping.

Changes

  • sandboxProcessDiagnostic now also classifies:
    • publisher failures — the plain delivery publisher failed: <code>:<step> line (step validated against the publisher runner's error codes)
    • execution failures — the execution_attempt_result / failed structured log (reusing the analysis failureKind/failureStage vocabulary)
    • (existing analysis + bootstrap cases unchanged)
  • CloudflareSandboxProviderFact carries two optional flat strings diagnosticKind + diagnosticDetail (bounded), added to both .strict() provider-fact schemas (worker-emit + control-plane-parse) and threaded through the runtime client's parse+reconstruct.
  • On a failed observation the Worker sets these from the diagnostic (compact JSON detail), or — when unclassified — keeps a bounded raw stderr tail so an exit-1 still leaves a clue. The plugin copies them into ExecutionFact.facts, which persists to executor_observations.facts_json.

Tested

  • 963 unit + 550 workflow pass; tsc --noEmit + eslint . clean.
  • New parser cases (publisher with/without step, unknown step dropped, execution failure) in the worker test; runtime-client test asserts the fields survive the strict parse; plugin test asserts a failed observation surfaces the diagnostic into facts.
  • Validated in prod (v22): a real tipsy-backend work execution failed (exit 1); executor_observations.facts_json now records {"exitCode":"1","diagnosticKind":"execution_failure","diagnosticDetail":"{\"kind\":\"execution_failure\"}"} — recovered after the container was reaped (raw /logs was empty, as expected).

Scope

Observability only. No change to publisher logic, the watchdog, or retry behavior. The container is still reaped on terminal (no sandbox leak); the cause now survives in D1.

Known follow-up (not in this PR)

The prod validation surfaced a deeper gap: the execution_failure diagnostic had no failureKind/failureStage — the execution runner emitted the failed result without a classified ExecutionAttemptError.kind. So we now know which lane failed, not yet why. That's a runner-side classification gap, addressed separately.

🤖 Generated with Claude Code

A failed publisher/execution sandbox process was undiagnosable: the container
is reaped on terminal observation, so /logs returns an empty fallback, and the
Worker's computed diagnostic was logged only to short-retention Workers Logs,
never to the durable executor_observations.facts_json. The result: an exit-1
publisher failure left no recoverable cause (blocker-8/10 observability gap).

- sandboxProcessDiagnostic now also classifies publisher failures (the plain
  `delivery publisher failed: <code>:<step>` line, step validated against the
  publisher runner's error codes) and execution-runner failures (the
  execution_attempt_result/failed structured log, reusing the analysis failure
  vocabulary), in addition to the existing analysis + bootstrap cases.
- CloudflareSandboxProviderFact carries two optional flat strings, diagnosticKind
  and diagnosticDetail (bounded), added to both strict provider-fact schemas and
  threaded through the runtime client's parse+reconstruct.
- On a failed observation the Worker sets these from the diagnostic (compact JSON
  detail), or, when unclassified, keeps a bounded raw stderr tail so an exit-1
  still leaves a clue. The plugin copies them into ExecutionFact.facts, which
  persists to executor_observations.facts_json — queryable without the container
  and without a live tail.

Scope: observability only. No change to publisher logic, the watchdog, or retry
behavior. The container is still reaped on terminal (no sandbox leak); the cause
now survives in D1.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@evilstar9527
evilstar9527 merged commit 49d5a16 into main Aug 28, 2026
6 checks passed
@evilstar9527
evilstar9527 deleted the fix/publisher-failure-observability branch August 28, 2026 02:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant