Repository navigation
test(harness): make the suite pass on stock macOS without weakening coverage - #7005
Open
chanolan20 wants to merge 1 commit into
Open
chanolan20 wants to merge 1 commit into
chanolan20 wants to merge 1 commit into
Conversation
…overage The full suite is green on Linux CI but 82 tests failed on a stock macOS host. Four root causes, all test-harness-side: - worker_runtime set the pytest basetemp beside the private TMPDIR, so tmp_path paths sat outside the tool-path confinement allowlist (which only contains <runtime>/tmp) and path-confinement, write_file race, and edit_file tests failed. The basetemp now lives inside the private TMPDIR. - The runtime root lived under macOS's 49-char /var/folders TMPDIR, so AF_UNIX sockets (tmux, agent runtime) overflowed the 104-byte sun_path. The root now sits at the short real temp root, matching the doctrine already in tests/helpers/unix_sockets.py. - Tests that drive the enforcing containment path (bwrap filesystem+process_tree) cannot observe anything on hosts that refuse to run uncontained. They are now gated by a cached probe of the same mechanism table containment itself uses, so they still run on CI. - Resolved-vs-unresolved temp path comparisons (the family documented in tests/KNOWN_FAILURES.md) in test_workspace_confine, test_write_file_empty_body, test_review_regressions, and a 127.0.0.2-refusal probe gate for the pinned-transport fallback test. No production code changed.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The full pytest suite is green on Linux CI but failed 82 tests on a stock macOS host (
81 failed, 14116 passedbefore;14185 passed, 96 skipped, 0 failedafter this change). Tracked as #7006. All failures were harness-side; four root causes:tests/helpers/worker_runtime.pyset pytest's basetemp to<runtime>/pytestwhile the private TMPDIR was<runtime>/tmp. On Linux this passes only because the runtime root itself lives under/tmp(an allowlisted root); on macOS the runtime root is/var/folders/..., so everytmp_pathpath was rejected by_resolve_tool_path(poisoning tool-path-confinement, write_file race, and edit_file tests). The basetemp now lives inside the private TMPDIR./var/foldersTMPDIR left too few of the 104sun_pathbytes for the tmux socket and agent runtime socket bound undertmp_path. The runtime root now sits at the short real temp root (/private/tmp, 0700 subdir), matching the doctrine already documented intests/helpers/unix_sockets.py.filesystem+process_treeenforcing path, which by design refuses to run on hosts without bubblewrap (related to Native macOS (Apple Silicon): bash/python tools refuse to run — "containment unavailable (filesystem, process_tree)" #6688). They are gated bytests/helpers/containment_gate.py, a cached probe of the same mechanism tablesrc/containment.pyuses — so they still execute on Linux CI and skip (not fail) elsewhere.tests/KNOWN_FAILURES.md(/tmp→/private/tmp): fixed intest_workspace_confine,test_write_file_empty_body(fixture now yields the realpath),test_review_regressions. The pinned-transport SSRF fallback test gained a probe gate for hosts (stock macOS) where127.0.0.2drops instead of refusing.Target branch
devin the public repository.Linked Issue
Fixes #7006 (related: #6688)
Type of Change
Checklist
labin maintainer-preview;devin the public repository)docker compose uporuvicorn app:app) and verified the change works end-to-end. Type-checks and unit tests are not enough.How to Test
The change is test-harness-only; there is no app behavior to view in a browser, so the app-run box is intentionally unchecked. To verify:
./venv/bin/python -m pytest -q tests/test_tool_path_confinement.py tests/test_workspace_confine.py tests/test_worker_runtime.py tests/test_containment_enforcement.py— all gated tests still run (no new skips) and pass, becauseenforcing_containment_available()probes the real mechanism table./var/folders/wk/88p0jv_51m7d3x224pz1rsfh0000gn/T/: the same command — the previously-failing tests now pass or skip with the containment reason instead of erroring.81 failed, 14116 passed→14185 passed, 96 skipped, 0 failed(CHROMADB_PORT=<closed-port> ./venv/bin/python -m pytest -q, ~25 min, withnpm ci+npx playwright install chromium).git diff dev...fix/macos-test-runtime --stattouches onlytests/.