Skip to content

rig: the ROS distribution is one switch — jazzy/lyrical features in rig/pixi.toml (rigs' environments unchanged) - #294

Open
mbreiser wants to merge 10 commits into
mainfrom
rig/ros-distro-switch
Open

mbreiser wants to merge 10 commits into
mainfrom
rig/ros-distro-switch

Conversation

@mbreiser

@mbreiser mbreiser commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Tooling + docs only (no Pages tree change; rig/ is the rig PCs' workspace). Companion of reiserlab/rig-nodes#8, which made the ROS 2 distribution one switch there; this PR mirrors it in rig/pixi.toml. Parent record: LAB-338 (rig-nodes extraction). The rigs' environments do not change.

What changes

  • rig/pixi.toml: [feature.jazzy] (the rigs: robostack-jazzy, mutex 0.19.*, Python 3.12.*, ros-jazzy-* ros-base / rmw_zenoh / rosbag2 / demo-nodes, the two Zenoh config URIs under python3.12) and [feature.lyrical] (robostack-lyrical, mutex 0.22.*, Python 3.14.*, ros-lyrical-*, URIs under python3.14, and rig_nodes + rig_interfaces built from the rig-nodes git branch feat/ros-distro-switch, because the published channel is Jazzy-only). Workspace channels shrink to conda-forge; each feature appends its RoboStack channel with priority = -1 and the channel feature carries the rig-nodes release URL with priority = -2, so every existing environment keeps its channel order. Environments: default / typed / viz = jazzy + channel (+ typed / viz), dev = jazzy + dev, new lyrical = lyrical (own solve group).

  • dev loses the ros-jazzy-* toolchain + cmake/ninja/cxx-compiler lines: pixi-build resolves each source package's build/host environment itself; the lyrical environment's build from git on this Mac lists none of them and compiles rig_interfaces fine (he8ea13f_0, python_abi 3.14.* *_cp314, mutex >=0.22.0,<0.23.0a0). The arena harness then passed 63/63 on that environment (pixi run -e lyrical test: whole graph, Studio emulator, model fly, crash phase — on Python 3.14 and rosbag2 0.33); pyzmq resolves as the abi3 build (py312… build string, imports under 3.14).

  • [workspace.build-variants] ros2-distro-mutex = ["0.19.*|0.22.*"], the same line as rig-nodes' own workspace: once rig-nodes main has no manifest pin, a consumer that builds the source packages needs it or the Jazzy host solve takes mutex 0.20 (the Python 3.14 rebuild of Jazzy).

  • Docs: rig/README.md (the -e lyrical line + a paragraph), a dated note in docs/development/rig-software-repos-and-runners.md (before §8), one clause in CLAUDE.md (never spell ros-jazzy-* outside a distro feature). The record itself lives in rig-nodes docs/ros-distro-switch.md.

  • New CI workflow rig-graph.yml (path-filtered to rig/**): until now nothing in this repository's CI ran rig/. jazzy (blocking; ubuntu-24.04 + windows-2022) installs the committed lock's default environment and runs test-gateway, test-stimulus, test-viz and the whole-graph harness; lyrical (allowed to fail) does the same on the Lyrical environment built from git; watch (Mondays 06:17 UTC and on demand) re-solves lyrical against the live RoboStack channel in the runner only, prints which ROS packages moved, and runs the harness on it — the "keep an eye on the channel" job. The committed lock is never touched by it.

Lyrical on this Mac, beyond the harness

check-typed (8 types serialize), test-gateway 5/5, test-stimulus and test-viz pass, and the rig's own composition pixi run -e lyrical rig -- --sim --bias --bias-sim driven by the Studio emulator with --connector-cmds: every check PASS, 400 frames applied over 2 epochs, BIAS connect/startRecording/getTimestamp/stopRecording ok, fictrac p50 latency 0.52 ms (p99 2.03), clean stop on SIGINT.

Regression found on ww18 and fixed (dc2d9b2)

Moving the two ZENOH_*_CONFIG_URI lines into the distro features left the workspace-level [target.win-64.activation.env] override dead: a feature's activation env takes precedence over it, so on Windows every environment (the rigs' default included) got the literal $CONDA_PREFIX/lib/python3.x/… and rmw_zenohd could not open its config. That was the real cause of both Windows CI failures (not the 20 s router wait, which stays at 60 s anyway). Each feature now carries its own [feature.<distro>.target.win-64.activation.env]; no lock change. Found by the ww18 bench session before any merge; its Lyrical install on the rig PC took 43 s and compiled rig_interfaces with VS 2022 19.44; check-typed, test-gateway 5/5, test-stimulus 51/51, test-viz 12/12 passed there.

CI after the fix (run at dc2d9b2): jazzy windows-2022 ✓, lyrical ubuntu ✓; jazzy ubuntu 59/60 and lyrical windows 62/63, each one timing-sensitive check on a shared 2-vCPU runner (an epoch's start-frame tare one frame late; conversion fidelity |Δms| max 107 ms against a 50 ms bound). Both pass on the Mac and on ww18. The harness steps retry once; a deterministic break still fails twice. Follow-up: a --ci tolerance scale in test_ros2_poc.py / studio_emulator.py instead of the retry.

ww18, the first rig PC on Lyrical (2026-11-05, run by the ww18 session at dc2d9b2)

Install 43 s (rig_interfaces compiled by VS 2022 19.44); runtime lyrical / rmw 0.10.6 / rclpy 10.0.11 / rosbag2 0.33.4 / mutex 0.22.0 lyrical_30 / Python 3.14.10; check-typed ✓, test-gateway 5/5, test-stimulus 51/51, test-viz 12/12; arena harness 63/63 in 80 s (Jazzy baseline on the same PC 60/60 in 56 s); one real closed-loop run: FicTrac 2.1.2 on the FLIR camera at ~96 Hz (typed adapter), Production Studio, G6 on COM4 — bag 4,182 records → 6,637 converted lines identical to the live file, runlog-check 4/4, controller 0x70 = host 0x70 = 1,543, shutdown parsed 29,472 / skipped 0 / bad 0, p99 0.77 ms, Console clean; soak smokes 6/6 (simulator and real FicTrac, 35,044 frames, 0 bad, p99 0.61 ms, clean CTRL_BREAK stop); 8 h real-FicTrac soak: 48/48 emulator runs ok, graph clean exit 0 (13:04–21:04; 2,776,157 frames parsed, 0 skipped, 0 bad, fictrac p99 0.84 ms, BIAS simulator 144,261 frames 0 bad; rig.log without errors or tracebacks; adapter at 96.2–96.4 Hz to the end). Record: rigs/ww18/lyrical-bench-2026-11-05.md in arena-experiments; rig-nodes docs PR #9.

Findings adopted here: lyrical-viz environment (ros-lyrical-foxglove-bridge, same solve group as lyrical) and rig.py --foxglove choosing the bridge's environment from PIXI_ENVIRONMENT_NAME — a Jazzy foxglove_bridge cannot join a Lyrical router. Benign: Lyrical's bridge logs one Failed to retrieve parameters from node '/rig_host_monitor' on connect. Not Lyrical: 4 of 1,548 long arena_command lines are kept verbatim because the Studio truncates their head (same on Jazzy; Studio-side item).

Soak driver + lab-PC handover

  • pixi run soak (rig/tests/soak_graph.py): one rig.py graph kept running for --hours (simulated FicTrac at 100 Hz or --fictrac for the real adapter, BIAS simulator + node, host monitor, gateway), a recorded run driven through the gateway by the Studio emulator every --every minutes, a rig-style stop through the process group (CTRL_BREAK on Windows), runlogs/soak-<stamp>/soak.json with every run and the gateway's accounting; refuses to start when 7447 / 8765 / 60000 are taken. Smoke on the Mac's Lyrical environment: 3/3 runs ok, 11,129 frames, 0 bad, p99 1.4 ms, clean stop.
  • rig/LYRICAL-BENCH.md: the procedure for the first Lyrical session on a Windows rig PC (prerequisites incl. MSVC v143 for the source build and the C:\px build root, install, the no-hardware suites, one real Studio run, the overnight soak, where results go, cleanup).
  • Bags across distros: a bag written on Lyrical (rosbag2 0.33, metadata version 9) converts with the Jazzy environment's mcap_convert (2,031 records, 1,121 frames, closed: true) and opens with Jazzy's ros2 bag info exactly as with Lyrical's. rosbag2 0.33 writes the rosbag2 MCAP metadata record (with our custom_data runtime dict) at open as well as at close, so even a crashed Lyrical bag says what it ran on. rosbag2's Reindexer keeps only ros_distro in the regenerated metadata.yaml; the MCAP record is the authority.

Lock

Diffed environment by environment against main's lock (package URLs per platform):

environment linux-64 osx-arm64 win-64 channels
default 438 → 438, 0 removed / 0 added 399 → 399, 0 / 0 398 → 398, 0 / 0 same order
typed 440 → 440, 0 / 0 401 → 401, 0 / 0 400 → 400, 0 / 0 same order
viz 499 → 499, 0 / 0 444 → 444, 0 / 0 430 → 430, 0 / 0 same order
dev 456 → 439 433 → 400 407 → 399 rig-nodes channel no longer listed (it builds from git)
lyrical 436 (new) 399 (new) 398 (new) conda-forge, robostack-lyrical

dev's change is the dropped toolchain shims plus conda-forge churn on a re-solve (flake8, python-dateutil, libcxx, BLAS builds); its rig_interfaces record is still cp312 + mutex >=0.19.0,<0.20.0a0 from rig-nodes main a32ee8e. pixi install --locked -e default and -e lyrical both installed from this lock on this Mac.

After rig-nodes#8 merges

Point [feature.lyrical]'s two git sources at main (and drop the note), re-lock. The dev feature already tracks main.

Tier

Tooling-only: no web tool, no Pages tree. Not a release candidate; merges on its own like the other rig/ tooling PRs (ask if it should ride the next candidate instead).

Release notes

None (nothing user-visible in the web tools).

🤖 Generated with Claude Code

…ig/pixi.toml (rigs' environments unchanged)

Mirrors reiserlab/rig-nodes#8. `[feature.jazzy]` (robostack-jazzy, mutex 0.19.*, Python 3.12.*,
ros-jazzy-* ros-base/rmw_zenoh/rosbag2/demo-nodes, Zenoh config URIs under python3.12) and
`[feature.lyrical]` (robostack-lyrical, mutex 0.22.*, Python 3.14.*, ros-lyrical-*, URIs under
python3.14, rig_nodes + rig_interfaces from the rig-nodes git branch since the channel is
Jazzy-only). Workspace channels = conda-forge; each feature appends its RoboStack channel with
priority -1 and the `channel` feature the release URL with priority -2, so default / typed / viz
keep their channel order and resolve to the SAME packages (0 records added or removed on all three
platforms against main's lock). `dev` drops the ros-jazzy toolchain lines (pixi-build resolves the
build env itself; the lyrical build from git proved it) and gains the jazzy feature. The mutex
build-variant line from rig-nodes' workspace is mirrored for the source-building environments.

Verified on this Mac: `pixi install --locked -e default` and `-e lyrical`; the arena harness on
the lyrical environment (Python 3.14, rosbag2 0.33) 63/63.

Docs: rig/README.md, a dated note in docs/development/rig-software-repos-and-runners.md, one
CLAUDE.md clause. The record is rig-nodes docs/ros-distro-switch.md. LAB-338 follow-up.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
mbreiser and others added 8 commits November 5, 2026 12:11
…rical (allowed to fail), plus a weekly channel watch

`.github/workflows/rig-graph.yml`, path-filtered to rig/**: `jazzy` installs the committed lock's
default environment on ubuntu-24.04 + windows-2022 and runs test-gateway / test-stimulus / test-viz
and the whole-graph harness `rig/tests/test_ros2_poc.py`; `lyrical` does the same on the lyrical
environment (rig-nodes built from git on RoboStack Lyrical, Python 3.14), allowed to fail; `watch`
(Mondays + on demand) re-solves `lyrical` against the live RoboStack channel in the runner only,
prints what moved, and runs the harness on it. Until now nothing in this repository's CI ran rig/.

Mac, lyrical environment: check-typed 8 types, test-gateway 5/5, test-stimulus + test-viz pass,
`rig.py --sim --bias --bias-sim` + Studio emulator with connector commands all PASS (400 frames
applied, fictrac p50 0.52 ms, clean stop).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…as-videos/

`rig.py --bias-sim` writes the fake camera's videos into bias-videos/ under the working directory;
a Mac demo run on the lyrical environment left two files that the previous commit swept up.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… 0.4.0 predates it

First run of rig-graph.yml: `pixi install --locked -e default` succeeded on ubuntu-24.04 and the step
died on `python -m rig_nodes.runtime`, a module that exists only on the rig-nodes branch the
lyrical environment builds from git. The jazzy job prints the package version instead.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…few minutes) and the Lyrical bench handover for a lab PC

`tests/soak_graph.py`: one `rig.py` graph (simulated FicTrac at 100 Hz or the real adapter, BIAS
simulator + node, host monitor, gateway) kept running for --hours, a recorded run driven through the
gateway by `studio_emulator.py` every --every minutes, a rig-style stop through the process group
(rig_nodes.supervise, so CTRL_BREAK on Windows), `runlogs/soak-<stamp>/soak.json` with every run and
the gateway's own accounting. Refuses to start when 7447 / 8765 / 60000 are taken. Written for the
failure modes a new rmw brings: a run that times out, applies fewer frames, or a graph that exits.

`LYRICAL-BENCH.md`: the procedure for the first Lyrical session on a Windows rig PC — prerequisites
(pixi ≥ 0.83, MSVC v143 for the source build, the C:\px build root, the branch), install, the
no-hardware suites, one real Studio run, the overnight soak, where results go, cleanup.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ss waits 60 s for the router and shows its log

The `lyrical` feature's git sources move from the merged branch to `main` (pinned to f7bb3fd in
pixi.lock, the merge commit of rig-nodes#8); `dev` moves from a32ee8e to the same commit, so it now
builds the distro-switch manifests (which is what the mutex build-variant is for). default / typed /
viz are untouched (0 records added or removed against main's lock).

The first Windows run of rig-graph.yml failed before anything started: the harness's 20 s wait for
`ros2 run rmw_zenoh_cpp rmw_zenohd` ran out on a cold windows-2022 runner (ros2cli entry-point
discovery + pyc compilation on first use), and every later check failed on the missing router. The
wait is 60 s now and the check carries the router's log tail (`Proc.text()`), so a real failure is
visible next time. Rig PCs start the router in 2–3 s.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ow; a six-minute soak smoke before the overnight run

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ation env beats the workspace win-64 target

Found on ww18 (Lyrical bench, 2026-11-05) and the reason both Windows CI jobs failed: moving the two
ZENOH_*_CONFIG_URI lines into [feature.jazzy|lyrical.activation.env] left the workspace-level
[target.win-64.activation.env] override dead, because a feature's activation env takes precedence
over it. On Windows every environment, the rigs' default included, got the literal
`$CONDA_PREFIX/lib/python3.x/…` and `rmw_zenohd` could not open its config ("The system cannot find
the path specified"). rig_nodes.zenoh's defaults do not replace a variable that is already set.

Each distro feature now carries its own [feature.<distro>.target.win-64.activation.env] with the
%CONDA_PREFIX%\Lib\site-packages\… spelling (no Python version in that path, so both are alike);
the workspace block is gone. No lock change. Verified by the ww18 session: `pixi run -e lyrical`
resolves both URIs to existing files.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…tuned for rig PCs, not 2-vCPU runners

Run 42076968526 at dc2d9b2 (the Windows activation fix): jazzy windows-2022 ✓ and lyrical ubuntu ✓ for
the first time, and two single-check misses elsewhere — jazzy ubuntu 59/60 (`epoch 2: opened on
start_frame 0 — first index 74`, the tare raced one frame under load) and lyrical windows 62/63
(`conversion fidelity … |Δms| worst ≤ 50 ms`: max 107 ms). Both are timing on a shared runner, both
pass on the Mac and on ww18. One retry; a deterministic break still fails twice. Follow-up noted in
the workflow: a `--ci` tolerance scale in test_ros2_poc.py / studio_emulator.py.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ge from the graph's own RoboStack set (ww18 finding L2)

A Jazzy foxglove_bridge cannot join a Lyrical router (different rmw_zenoh), and `rig.py --foxglove`
hard-coded the `viz` environment. New feature + environment `lyrical-viz` (ros-lyrical-foxglove-bridge,
same solve group as `lyrical`, the win-64 Library\lib PATH override like `viz`); rig.py chooses
`lyrical-viz` when PIXI_ENVIRONMENT_NAME starts with `lyrical`, else `viz`, in the path check, the
message and the `pixi run -e` call. Lock: the new environment only (two packages on top of lyrical);
default / typed / viz unchanged. LYRICAL-BENCH.md step 3 names it. Verified by the ww18 session with
an equivalent local edit: foxglove_bridge 3.6.0 on 8766 with viz_node, Foxglove desktop connected.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant