⏱️ fix: Recycle Sandboxes Before Clock Drift

[danny-avila]

Summary

I added an actual-skew readiness and liveness guard that removes long-lived libkrun sandbox guests from service and recycles them before wall-clock drift crosses the 30-second execution-manifest tolerance.

• Measure guest wall-clock skew from the sandbox API HTTP Date header against the enclosing host request interval.
• Remove a skewed Kubernetes runner from service on the first failed readiness probe, then restart it after sustained liveness failure.
• Recycle guests at a configurable 10-second maximum, with a stable per-pod 0–2-second offset to reduce synchronized restarts.
• Reserve the five-second request timeout below the manifest tolerance and reject non-integer or unsafe Helm configuration.
• Validate IMF-fixdate syntax before parsing the guest timestamp so permissive GNU date inputs cannot bypass the guard.
• Keep the clock-skew guard disabled by default for Docker and Compose callers, where health status alone does not restart a container, and opt in explicitly through the production Helm deployment.
• Add focused shell and Helm assertions for skew boundaries, request latency, readiness behavior, configuration validation, disabled checks, invalid timestamps, and image wiring.

This contains the immediate failure mode without widening the signed execution-manifest replay window. A true guest synchronization path can be pursued separately when libkrun/libkrunfw supports an appropriate host-to-guest time source.

Change Type

• Bug fix (non-breaking change which fixes an issue)

Testing

• Ran bash -n docker/sandbox-runner-healthcheck.sh tests/sandbox_runner_healthcheck.sh tests/block_root_package_delivery.sh.
• Ran tests/sandbox_runner_healthcheck.sh.
• Ran tests/block_root_package_delivery.sh.
• Ran docker buildx build --check -f api/Dockerfile ..
• Ran docker buildx build --check -f docker/Dockerfile.worker-sandbox ..
• Ran docker compose -f docker-compose.scalable.yml config --quiet.
• Ran git diff --check.

Test Configuration:

• Local macOS host with Docker Desktop's desktop-linux builder.
• Helm rendering exercised through the repository's block-root package delivery test.

Checklist

• My code adheres to this project's style guidelines
• I have performed a self-review of my own code
• I have commented in complex areas of my code
• My changes do not introduce new warnings
• I have written tests demonstrating that my changes are effective
• Local tests pass with my changes

[danny-avila]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 77fd5adbf1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[danny-avila]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e213accc26

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[danny-avila]

@codex review

[danny-avila]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: ef1055b0de

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4241b96965

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[danny-avila]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. What shall we delve into next?

Reviewed commit: 783573def0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[evellasques]

Great work, should indeed contain the problem. Maybe based on this:

A true guest synchronization path can be pursued separately when libkrun/libkrunfw supports an appropriate host-to-guest time source.

I'd advocate for creating a Linear issue to track that.