os: add support for windows - #709
Open
AmyJeanes wants to merge 5 commits into
Open
Conversation
Queues and the fake-event state are named shared-memory sections rather than files: Windows has no /dev/shm, a section needs no cleanup (the kernel drops it with its last handle or view, where a Linux queue file stays until reboot) and queue files could not be swept safely, since an NTFS delete succeeds on a file that is only mapped and leaves the mapping attached to an orphan. A section keeps its name only while a handle to it is open, so every mapping holds one for its lifetime: the queue struct carries it and the event state keeps one per view. msgq_shm_dir() picks the queue directory on Linux and macOS (/dev/shm, /tmp) instead of three ifdef ladders; on Windows it locates the visionipc sockets under %TEMP%, "/tmp" being the current drive's root there. Readers block in msgq_poll() on a per-thread named event which publishers signal by thread id, replacing the SIGUSR2/tkill wakeup; the wait loops until the steady clock passes the deadline, since WaitForSingleObject's timeout runs on the interrupt clock and can expire a few microseconds early. Fake events use named Win32 events instead of FIFOs. visionipc uses AF_UNIX stream sockets with length-prefixed messages. The buffers are anonymous sections whose handles the server duplicates into the client process, the Windows form of SCM_RIGHTS; the socket reports the peer's pid (SIO_AF_UNIX_GETPEERPID). The C++ tests now release their queues: a section is created afresh once nobody maps it, where the tests used to rely on deleting the queue file. test_receive_timeout measures with perf_counter: on Windows Python 3.12 time.monotonic() is the 15.6 ms tick clock and read a 5 ms wait as 0 ms about once in a hundred runs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016AgqYZYWLpwE2T5vS3nVEt
toggle_fake_events() and set_fake_prefix() set CEREAL_FAKE* through the C runtime's putenv. Python's os.environ is a snapshot taken at startup, and on Windows multiprocessing spawns children from that snapshot (in a venv it hands os.environ.copy() to CreateProcess), so the replayed process in process_replay never entered fake mode and the test waited for it forever. Keep os.environ in sync from the Python wrappers; a forked child on Linux inherits both anyway. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016AgqYZYWLpwE2T5vS3nVEt
Python imports extension modules as .pyd on Windows. The cython tool renames the .so targets the SConscripts declare, so neither they nor a SConstruct that loads the tool has to know. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016AgqYZYWLpwE2T5vS3nVEt
Windows checkouts default to CRLF, which breaks the shell scripts and makes every file look modified from an MSYS2 shell. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016AgqYZYWLpwE2T5vS3nVEt
The standalone build picks the mingw tool with clang (the default tool picks MSVC), links libc++ statically so the binaries run outside the MSYS2 shell, and links mman-win32 and winsock for mmap and visionipc's sockets; the extension modules link against the interpreter's import library. setup.py copies the .pyd the cython tool names. setup.sh pins uv to a native CPython (the MSYS2 toolchain's own python comes first on PATH and its wheels are incompatible) and activates the venv from Scripts/; test.sh's install test uses the same layout. tests.yml gains a windows-latest entry running in the CLANG64 shell from setup-msys2, with a five-minute budget for ./test.sh on the 4-core runner. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016AgqYZYWLpwE2T5vS3nVEt
This was referenced Sep 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Native Windows support for development builds of openpilot (MSYS2 CLANG64, clang/libc++). Linux and macOS code paths are untouched apart from a few shared wrappers (
event_open()/event_close()/event_state_shm_munmap()/ipc_close()) that call what they called before, andmsgq_shm_dir(), which now picks the queue directory on Linux and macOS instead of three ifdef ladders.Local\msgq_<name>, per logon session) rather than files. Windows has no/dev/shm, and a section needs no cleanup: the kernel drops it with its last user, where a Linux queue file stays until reboot. Queue files could not have been swept safely, since an NTFS delete succeeds on a file that is only mapped and leaves the mapping attached to an orphan. A section keeps its name only while a handle to it is open, so every mapping holds one for its lifetime. Readers block inmsgq_poll()on a per-thread named event which publishers signal by thread id, the low 32 bits of a reader uid on every platform, replacing the SIGUSR2/tkill wakeup. The wait loops until the steady clock passes the deadline:WaitForSingleObject's timeout runs on the interrupt clock and can expire a few microseconds early.CEREAL_FAKE*intoos.environ, which is what a spawned child inherits on Windows (a forked one inherits both).SOCK_SEQPACKET). The buffers are anonymous sections whose handles the server duplicates into the client process, the Windows form ofSCM_RIGHTS; the socket reports the peer's pid (SIO_AF_UNIX_GETPEERPID). The listener usesWSAPolland binds under%TEMP%(/tmpis the current drive's root on Windows), the one thingmsgq_shm_dir()locates there.cythonSCons tool names the extension modules.pydon Windows, so the SConscripts keep their.sonames, andsetup.pycopies that.pyd. msgq's own SConstruct uses the mingw tool with clang (the default tool picks MSVC), links libc++ statically so the test runner and the modules run outside the MSYS2 shell, links winsock for visionipc's sockets (thecommonlist now chooses per platform,rton Linux as before) and the Cython modules against the interpreter's import library.setup.shpins uv to a native CPython in an MSYS2 shell (the toolchain's own python comes first on PATH, and its wheels are incompatible) and activates the venv fromScripts/;test.sh's install test uses the same layout. A.gitattributeskeeps every checkout LF: Git for Windows defaults to autocrlf, and bash cannot run a CRLF script.test_receive_timeoutmeasures withperf_counter; on Windows Python 3.12time.monotonic()is the 15.6 ms tick clock and read a 5 ms wait as 0 ms about once in a hundred runs.windows-latestentry intests.ymlrunning the same./test.shfrom an MSYS2 CLANG64 shell (only clang is installed on top of the runner's MSYS2), with a five-minute budget where the others keep one. Only that entry names its shell; the others keepbash -e {0}, GitHub's implicit default, through the job's run defaults.Tested on Windows: the C++ test runner (14/14); the msgq, messaging and visionipc Python tests (321) through openpilot's test runner; a 30 s stress run with queues created from six processes at a time, a visionipc client in another process receiving 12k frames, and a queue's section checked to exist exactly as long as its owner; and openpilot's CI on commaai/openpilot#38810 (full unit test suite on Windows, process replay on Linux).
./test.shon Windows from an MSYS2 CLANG64 shell runs the build, ruff, ty, codespell, cppcheck, cpplint, the C++ runner (14/14), the unit tests (29) and the install test (3) in 13 s on a workstation. This repo's CI on this PR on all four platforms, the Windows entry included; the same tests on Linux.This PR is part of a larger series of work bringing support for Windows to openpilot:
Each PR in the series merges independently; only the openpilot one depends on the others.
The changes in this PR were generated by Claude Fable 5.1 but were human reviewed and fully tested end to end both locally and in CI.