Skip to content

os: add support for windows - #709

Open
AmyJeanes wants to merge 5 commits into
commaai:masterfrom
AmyJeanes:windows
Open

os: add support for windows#709
AmyJeanes wants to merge 5 commits into
commaai:masterfrom
AmyJeanes:windows

Conversation

@AmyJeanes

@AmyJeanes AmyJeanes commented Sep 7, 2026

Copy link
Copy Markdown

Native Windows support for development builds of openpilot (MSYS2 CLANG64, clang/libc++). Linux and macOS code paths are untouched apart from a few shared wrappers (event_open()/event_close()/event_state_shm_munmap()/ ipc_close()) that call what they called before, and msgq_shm_dir(), which now picks the queue directory on Linux and macOS instead of three ifdef ladders.

  • msgq: queues are named shared-memory sections (Local\msgq_<name>, per logon session) rather than files. Windows has no /dev/shm, and a section needs no cleanup: the kernel drops it with its last user, where a Linux queue file stays until reboot. Queue files could not have been swept safely, since an NTFS delete succeeds on a file that is only mapped and leaves the mapping attached to an orphan. A section keeps its name only while a handle to it is open, so every mapping holds one for its lifetime. Readers block in msgq_poll() on a per-thread named event which publishers signal by thread id, the low 32 bits of a reader uid on every platform, replacing the SIGUSR2/tkill wakeup. The wait loops until the steady clock passes the deadline: WaitForSingleObject's timeout runs on the interrupt clock and can expire a few microseconds early.
  • events: fake events are named Win32 events instead of FIFOs (manual reset, so they stay signalled until cleared like unread bytes in a FIFO), and their shared state is a named section too. The Python wrappers mirror CEREAL_FAKE* into os.environ, which is what a spawned child inherits on Windows (a forked one inherits both).
  • visionipc: AF_UNIX stream sockets with length-prefixed messages (no SOCK_SEQPACKET). The buffers are anonymous sections whose handles the server duplicates into the client process, the Windows form of SCM_RIGHTS; the socket reports the peer's pid (SIO_AF_UNIX_GETPEERPID). The listener uses WSAPoll and binds under %TEMP% (/tmp is the current drive's root on Windows), the one thing msgq_shm_dir() locates there.
  • build: the cython SCons tool names the extension modules .pyd on Windows, so the SConscripts keep their .so names, and setup.py copies that .pyd. msgq's own SConstruct uses the mingw tool with clang (the default tool picks MSVC), links libc++ statically so the test runner and the modules run outside the MSYS2 shell, links winsock for visionipc's sockets (the common list now chooses per platform, rt on Linux as before) and the Cython modules against the interpreter's import library. setup.sh pins uv to a native CPython in an MSYS2 shell (the toolchain's own python comes first on PATH, and its wheels are incompatible) and activates the venv from Scripts/; test.sh's install test uses the same layout. A .gitattributes keeps every checkout LF: Git for Windows defaults to autocrlf, and bash cannot run a CRLF script.
  • tests: the C++ tests release their queues, since a section is created afresh once nobody maps it where the tests used to rely on deleting the queue file. test_receive_timeout measures with perf_counter; on Windows Python 3.12 time.monotonic() is the 15.6 ms tick clock and read a 5 ms wait as 0 ms about once in a hundred runs.
  • CI: a windows-latest entry in tests.yml running the same ./test.sh from an MSYS2 CLANG64 shell (only clang is installed on top of the runner's MSYS2), with a five-minute budget where the others keep one. Only that entry names its shell; the others keep bash -e {0}, GitHub's implicit default, through the job's run defaults.

Tested on Windows: the C++ test runner (14/14); the msgq, messaging and visionipc Python tests (321) through openpilot's test runner; a 30 s stress run with queues created from six processes at a time, a visionipc client in another process receiving 12k frames, and a queue's section checked to exist exactly as long as its owner; and openpilot's CI on commaai/openpilot#38810 (full unit test suite on Windows, process replay on Linux). ./test.sh on Windows from an MSYS2 CLANG64 shell runs the build, ruff, ty, codespell, cppcheck, cpplint, the C++ runner (14/14), the unit tests (29) and the install test (3) in 13 s on a workstation. This repo's CI on this PR on all four platforms, the Windows entry included; the same tests on Linux.

This PR is part of a larger series of work bringing support for Windows to openpilot:

Each PR in the series merges independently; only the openpilot one depends on the others.

The changes in this PR were generated by Claude Fable 5.1 but were human reviewed and fully tested end to end both locally and in CI.

AmyJeanes and others added 5 commits September 7, 2026 21:06
Queues and the fake-event state are named shared-memory sections rather
than files: Windows has no /dev/shm, a section needs no cleanup (the
kernel drops it with its last handle or view, where a Linux queue file
stays until reboot) and queue files could not be swept safely, since an
NTFS delete succeeds on a file that is only mapped and leaves the mapping
attached to an orphan. A section keeps its name only while a handle to
it is open, so every mapping holds one for its lifetime: the queue struct
carries it and the event state keeps one per view. msgq_shm_dir() picks
the queue directory on Linux and macOS (/dev/shm, /tmp) instead of three
ifdef ladders; on Windows it locates the visionipc sockets under %TEMP%,
"/tmp" being the current drive's root there.

Readers block in msgq_poll() on a per-thread named event which publishers
signal by thread id, replacing the SIGUSR2/tkill wakeup; the wait loops
until the steady clock passes the deadline, since WaitForSingleObject's
timeout runs on the interrupt clock and can expire a few microseconds
early. Fake events use named Win32 events instead of FIFOs.

visionipc uses AF_UNIX stream sockets with length-prefixed messages. The
buffers are anonymous sections whose handles the server duplicates into
the client process, the Windows form of SCM_RIGHTS; the socket reports
the peer's pid (SIO_AF_UNIX_GETPEERPID).

The C++ tests now release their queues: a section is created afresh once
nobody maps it, where the tests used to rely on deleting the queue file.

test_receive_timeout measures with perf_counter: on Windows Python 3.12
time.monotonic() is the 15.6 ms tick clock and read a 5 ms wait as 0 ms
about once in a hundred runs.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016AgqYZYWLpwE2T5vS3nVEt
toggle_fake_events() and set_fake_prefix() set CEREAL_FAKE* through the
C runtime's putenv. Python's os.environ is a snapshot taken at startup,
and on Windows multiprocessing spawns children from that snapshot (in a
venv it hands os.environ.copy() to CreateProcess), so the replayed
process in process_replay never entered fake mode and the test waited
for it forever. Keep os.environ in sync from the Python wrappers; a
forked child on Linux inherits both anyway.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016AgqYZYWLpwE2T5vS3nVEt
Python imports extension modules as .pyd on Windows. The cython tool
renames the .so targets the SConscripts declare, so neither they nor a
SConstruct that loads the tool has to know.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016AgqYZYWLpwE2T5vS3nVEt
Windows checkouts default to CRLF, which breaks the shell scripts and
makes every file look modified from an MSYS2 shell.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016AgqYZYWLpwE2T5vS3nVEt
The standalone build picks the mingw tool with clang (the default tool
picks MSVC), links libc++ statically so the binaries run outside the
MSYS2 shell, and links mman-win32 and winsock for mmap and visionipc's
sockets; the extension modules link against the interpreter's import
library. setup.py copies the .pyd the cython tool names. setup.sh pins
uv to a native CPython (the MSYS2 toolchain's own python comes first on
PATH and its wheels are incompatible) and activates the venv from
Scripts/; test.sh's install test uses the same layout. tests.yml gains a
windows-latest entry running in the CLANG64 shell from setup-msys2, with
a five-minute budget for ./test.sh on the 4-core runner.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016AgqYZYWLpwE2T5vS3nVEt
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant