Skip to content

test(harness): make the suite pass on stock macOS without weakening coverage - #7005

Open
chanolan20 wants to merge 1 commit into
odysseus-dev:devfrom
chanolan20:fix/macos-test-runtime
Open

chanolan20 wants to merge 1 commit into
odysseus-dev:devfrom
chanolan20:fix/macos-test-runtime

Conversation

@chanolan20

@chanolan20 chanolan20 commented Oct 31, 2026 •

Copy link
Copy Markdown

Summary

The full pytest suite is green on Linux CI but failed 82 tests on a stock macOS host (81 failed, 14116 passed before; 14185 passed, 96 skipped, 0 failed after this change). Tracked as #7006. All failures were harness-side; four root causes:

  1. basetemp outside the confinement allowlist — tests/helpers/worker_runtime.py set pytest's basetemp to <runtime>/pytest while the private TMPDIR was <runtime>/tmp. On Linux this passes only because the runtime root itself lives under /tmp (an allowlisted root); on macOS the runtime root is /var/folders/..., so every tmp_path path was rejected by _resolve_tool_path (poisoning tool-path-confinement, write_file race, and edit_file tests). The basetemp now lives inside the private TMPDIR.
  2. AF_UNIX overflow — macOS's 49-char /var/folders TMPDIR left too few of the 104 sun_path bytes for the tmux socket and agent runtime socket bound under tmp_path. The runtime root now sits at the short real temp root (/private/tmp, 0700 subdir), matching the doctrine already documented in tests/helpers/unix_sockets.py.
  3. Enforcing-containment tests on hosts that refuse uncontained runs — 16 tests drive the bwrap filesystem+process_tree enforcing path, which by design refuses to run on hosts without bubblewrap (related to Native macOS (Apple Silicon): bash/python tools refuse to run — "containment unavailable (filesystem, process_tree)" #6688). They are gated by tests/helpers/containment_gate.py, a cached probe of the same mechanism table src/containment.py uses — so they still execute on Linux CI and skip (not fail) elsewhere.
  4. Resolved-vs-unresolved temp path comparisons — the exact family documented in tests/KNOWN_FAILURES.md (/tmp → /private/tmp): fixed in test_workspace_confine, test_write_file_empty_body (fixture now yields the realpath), test_review_regressions. The pinned-transport SSRF fallback test gained a probe gate for hosts (stock macOS) where 127.0.0.2 drops instead of refusing.

Target branch

  • This PR targets the correct integration branch: dev in the public repository.

Linked Issue

Fixes #7006 (related: #6688)

Type of Change

  • Bug fix (non-breaking — fixes a confirmed issue)
  • New feature (non-breaking — adds new behaviour)
  • Breaking change (changes or removes existing behaviour)
  • Refactor / cleanup (behaviour unchanged)
  • Documentation only
  • CI / tooling / configuration

Checklist

  • I searched open issues and open PRs — this is not a duplicate.
  • This PR targets the correct integration branch (lab in maintainer-preview; dev in the public repository)
  • My changes are limited to the scope described above — no unrelated refactors or whitespace changes mixed in.
  • I actually ran the app (docker compose up or uvicorn app:app) and verified the change works end-to-end. Type-checks and unit tests are not enough.
  • I did not run the app/runtime validation and stated that gap in How to Test. Leave this unchecked when the app-run box above is checked.

How to Test

The change is test-harness-only; there is no app behavior to view in a browser, so the app-run box is intentionally unchecked. To verify:

  1. On any Linux machine (or in CI — the gated tests run unchanged where bubblewrap is available): ./venv/bin/python -m pytest -q tests/test_tool_path_confinement.py tests/test_workspace_confine.py tests/test_worker_runtime.py tests/test_containment_enforcement.py — all gated tests still run (no new skips) and pass, because enforcing_containment_available() probes the real mechanism table.
  2. On a stock macOS host with the default /var/folders/wk/88p0jv_51m7d3x224pz1rsfh0000gn/T/: the same command — the previously-failing tests now pass or skip with the containment reason instead of erroring.
  3. Full suite on macOS before/after: 81 failed, 14116 passed → 14185 passed, 96 skipped, 0 failed (CHROMADB_PORT=<closed-port> ./venv/bin/python -m pytest -q, ~25 min, with npm ci + npx playwright install chromium).
  4. Confirm no production code changed: git diff dev...fix/macos-test-runtime --stat touches only tests/.

…overage

The full suite is green on Linux CI but 82 tests failed on a stock macOS
host. Four root causes, all test-harness-side:

- worker_runtime set the pytest basetemp beside the private TMPDIR, so
  tmp_path paths sat outside the tool-path confinement allowlist (which
  only contains <runtime>/tmp) and path-confinement, write_file race,
  and edit_file tests failed. The basetemp now lives inside the private
  TMPDIR.
- The runtime root lived under macOS's 49-char /var/folders TMPDIR, so
  AF_UNIX sockets (tmux, agent runtime) overflowed the 104-byte
  sun_path. The root now sits at the short real temp root, matching the
  doctrine already in tests/helpers/unix_sockets.py.
- Tests that drive the enforcing containment path (bwrap
  filesystem+process_tree) cannot observe anything on hosts that refuse
  to run uncontained. They are now gated by a cached probe of the same
  mechanism table containment itself uses, so they still run on CI.
- Resolved-vs-unresolved temp path comparisons (the family documented in
  tests/KNOWN_FAILURES.md) in test_workspace_confine,
  test_write_file_empty_body, test_review_regressions, and a
  127.0.0.2-refusal probe gate for the pinned-transport fallback test.

No production code changed.
@github-actions github-actions Bot added the needs work PR description incomplete — please update before review label Oct 31, 2026
@github-actions github-actions Bot added ready for review Description complete — ready for maintainer review and removed needs work PR description incomplete — please update before review labels Oct 31, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready for review Description complete — ready for maintainer review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Full pytest suite fails on stock macOS (82 failures; all test-harness-side)

2 participants