rig: the ROS distribution is one switch — jazzy/lyrical features in rig/pixi.toml (rigs' environments unchanged)

.github/workflows/rig-graph.yml

@@ -0,0 +1,174 @@
+# The rig graph (rig/) in CI — the arena gateway harness `rig/tests/test_ros2_poc.py` (whole graph:
+# router, FicTrac + Kinefly + BIAS simulators and adapters, bridge node, Studio emulator, model fly,
+# crash phase) on the two ROS distributions the workspace knows (rig-nodes docs/ros-distro-switch.md):
+#
+#   jazzy    what the rigs run — the `default` environment from the committed lock, channel packages,
+#            nothing compiled. Blocking: a red run here means the rig workspace is broken.
+#   lyrical  the migration rehearsal — the `lyrical` environment builds rig-nodes from git on
+#            RoboStack Lyrical (Python 3.14). Allowed to fail: no rig runs it.
+#   watch    weekly (and on demand): re-solve `lyrical` against the LIVE RoboStack channel, print what
+#            moved (mutex set, rmw_zenoh, rosbag2, rclpy, Python) and run the harness on it. The lock in
+#            the repository is not touched; this is the "keep an eye on the channel" job.
+#
+# Windows is in the matrix because that is what the rig PCs run; ubuntu because it is cheap. macOS is
+# the developers' platform and is covered by hand (rig/SETUP.md host table). Path-filtered: only
+# changes under rig/ (or to this file) run it; the root `CI` workflow does not touch rig/.
+# One ROS graph per runner: the harness starts its own Zenoh router on 7447.
+name: rig graph
+
+on:
+    pull_request:
+        paths: ['rig/**', '.github/workflows/rig-graph.yml']
+    push:
+        branches: [main]
+        paths: ['rig/**', '.github/workflows/rig-graph.yml']
+    schedule:
+        - cron: '17 6 * * 1'   # Mondays 06:17 UTC: the channel watch
+    workflow_dispatch:
+
+permissions:
+    contents: read
+
+concurrency:
+    group: rig-graph-${{ github.event.pull_request.number || github.ref }}
+    cancel-in-progress: true
+
+env:
+    # The rigs' pixi floor (rig/SETUP.md) and the pixi that writes rig/pixi.lock.
+    PIXI_VERSION: v0.83.0
+
+jobs:
+    jazzy:
+        name: jazzy (${{ matrix.os }})
+        if: github.event_name != 'schedule'
+        strategy:
+            fail-fast: false
+            matrix:
+                os: [ubuntu-24.04, windows-2022]
+        runs-on: ${{ matrix.os }}
+        defaults:
+            run:
+                shell: bash
+                working-directory: rig
+        steps:
+            - uses: actions/checkout@v7
+              with:
+                  persist-credentials: false
+            - uses: prefix-dev/setup-pixi@v0.10.2
+              with:
+                  pixi-version: ${{ env.PIXI_VERSION }}
+                  run-install: false
+            - name: pixi install --locked (default = jazzy + the rig-nodes channel)
+              run: |
+                  pixi install --locked -e default
+                  pixi list | grep -E "^(rig_|ros2-distro-mutex|python |ros2-rclpy|ros2-rmw-zenoh-cpp)"
+                  # the released rig_nodes (channel 0.4.0) predates rig_nodes.runtime; the lyrical job prints that
+                  pixi run python -c "import rig_nodes, sys; print('rig_nodes', rig_nodes.__version__, 'python', sys.version.split()[0])"
+            - name: gateway registration + stimulus + viz (no graph)
+              run: |
+                  pixi run test-gateway
+                  pixi run test-stimulus
+                  pixi run test-viz
+            # The harness's timing tolerances are tuned for rig PCs (a 100 Hz graph + 8 processes on a 2-vCPU
+            # shared runner has missed `|Δms| worst ≤ 50 ms` and an epoch's start-frame tare by a frame). One
+            # retry on failure; a deterministic break fails twice. Follow-up: a `--ci` tolerance scale in the harness.
+            - name: the arena harness (whole graph, simulators, Studio emulator, crash phase)
+              run: pixi run test || { echo "::warning::harness failed once on this runner (timing-sensitive checks); retrying"; pixi run test; }
+
+    lyrical:
+        name: lyrical rehearsal (${{ matrix.os }}, allowed to fail)
+        if: github.event_name != 'schedule'
+        continue-on-error: true
+        timeout-minutes: 90
+        strategy:
+            fail-fast: false
+            matrix:
+                os: [ubuntu-24.04, windows-2022]
+        runs-on: ${{ matrix.os }}
+        defaults:
+            run:
+                shell: bash
+                working-directory: rig
+        steps:
+            - uses: actions/checkout@v7
+              with:
+                  persist-credentials: false
+            - uses: prefix-dev/setup-pixi@v0.10.2
+              with:
+                  pixi-version: ${{ env.PIXI_VERSION }}
+                  run-install: false
+            # MAX_PATH: the rosidl build of rig_interfaces adds ~211 chars to the checkout path.
+            - name: Windows build root
+              if: runner.os == 'Windows'
+              run: pixi config set --local detached-environments 'C:\px'
+            - name: pixi install --locked -e lyrical (builds rig_nodes + rig_interfaces from git)
+              run: |
+                  pixi install --locked -e lyrical
+                  pixi list -e lyrical | grep -E "^(rig_|ros2-distro-mutex|python |ros2-rclpy|ros2-rmw-zenoh-cpp)"
+                  pixi run -e lyrical python -m rig_nodes.runtime
+            - name: typed package + gateway registration + stimulus + viz
+              run: |
+                  pixi run -e lyrical check-typed
+                  pixi run -e lyrical test-gateway
+                  pixi run -e lyrical test-stimulus
+                  pixi run -e lyrical test-viz
+            - name: the arena harness on Lyrical
+              run: pixi run -e lyrical test || { echo "::warning::harness failed once on this runner (timing-sensitive checks); retrying"; pixi run -e lyrical test; }
+
+    watch:
+        # Re-solve `lyrical` against today's RoboStack channel (the repository lock is NOT updated — the
+        # runner's copy only), print what moved, run the harness. A failure here is information about
+        # the channel, not about the code; the job is allowed to fail and the summary is in the log.
+        name: channel watch (lyrical re-solved against the live channel)
+        if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
+        continue-on-error: true
+        timeout-minutes: 90
+        runs-on: ubuntu-24.04
+        defaults:
+            run:
+                shell: bash
+                working-directory: rig
+        steps:
+            - uses: actions/checkout@v7
+              with:
+                  persist-credentials: false
+            - uses: prefix-dev/setup-pixi@v0.10.2
+              with:
+                  pixi-version: ${{ env.PIXI_VERSION }}
+                  run-install: false
+            - name: what the committed lock pins
+              run: pixi list -e lyrical --no-install | grep -E "^(rig_|ros2-distro-mutex|python |ros2-rclpy|ros2-rmw-zenoh-cpp|ros2-rosbag2-py|ros2-rosbag2-storage-mcap)" || true
+            - name: re-solve lyrical against the live channel
+              run: |
+                  pixi update -e lyrical --json > update.json || true
+                  head -c 4000 update.json; echo
+                  python3 - <<'EOF'
+                  import json
+                  try:
+                      d = json.load(open("update.json"))
+                  except Exception as exc:
+                      print("no JSON report (nothing moved, or the report is not JSON):", exc); raise SystemExit(0)
+                  # Tolerant of the report's shape: every dict that names a package with a before/after is a change.
+                  moved = []
+                  def walk(x, path=()):
+                      if isinstance(x, dict):
+                          if "name" in x and ("before" in x or "after" in x):
+                              b, a = x.get("before") or {}, x.get("after") or {}
+                              moved.append((x["name"], b.get("version") if isinstance(b, dict) else b, a.get("version") if isinstance(a, dict) else a, "/".join(map(str, path))))
+                          for k, v in x.items():
+                              walk(v, path + (k,))
+                      elif isinstance(x, list):
+                          for v in x:
+                              walk(v, path)
+                  walk(d)
+                  print(f"{len(moved)} package changes in the re-solved lyrical environment")
+                  for name, before, after, where in sorted(moved):
+                      if name.startswith(("ros2-", "ros-lyrical", "python", "rig_")) or "zenoh" in name or "rosbag2" in name:
+                          print(f"   {name}: {before} -> {after}  ({where})")
+                  EOF
+                  pixi list -e lyrical | grep -E "^(rig_|ros2-distro-mutex|python |ros2-rclpy|ros2-rmw-zenoh-cpp|ros2-rosbag2-py|ros2-rosbag2-storage-mcap)"
+            - name: the arena harness on the re-solved environment
+              run: |
+                  pixi run -e lyrical python -m rig_nodes.runtime
+                  pixi run -e lyrical check-typed
+                  pixi run -e lyrical test || { echo "::warning::harness failed once on this runner (timing-sensitive checks); retrying"; pixi run -e lyrical test; }

CLAUDE.md

@@ -114,7 +114,10 @@ arena visualization stay here; the Python arena library + headless runner go to
 `LED-Display_pyDisplayTools` (LAB-337); shared ROS 2 infrastructure lives in **`reiserlab/rig-nodes`**
 (`rig_interfaces`, the device adapters + simulators, `rig_nodes.recorder` / `mcap_index` / `cdr` /
 `foxglove_json`, `rig_nodes.supervise`, the Zenoh configs; contract: its `CONTRACT.md`) and is INSTALLED
-into `rig/` as the pinned `rig_nodes` + `rig_interfaces` packages from that repo's channel. A node fix
+into `rig/` as the pinned `rig_nodes` + `rig_interfaces` packages from that repo's channel. The ROS distribution is one
+switch (2026-11-05): `[feature.jazzy]` / `[feature.lyrical]` in `rig/pixi.toml` and in rig-nodes carry the
+RoboStack channel + mutex + Python pins; never spell `ros-jazzy-*` outside a distro feature (rig-nodes
+`docs/ros-distro-switch.md`). A node fix
 is tested here with `pixi install -e dev` (rig-nodes `main` built from source — git source dependencies,
 NEVER path dependencies on a sibling checkout: pixi validates every environment's source paths before
 installing any environment, which broke `pixi install` on rig PCs on 2026-10-27); a release of the set

docs/development/rig-software-repos-and-runners.md

@@ -347,6 +347,22 @@ These items are the body of the rig-nodes ticket and apply to pyDisplayTools as
 13. **Maintainer cost, named.** Three repositories, two CI pipelines with three-platform builds, a
     channel, a compatibility table, and one maintainer today.
 
+### The ROS distribution is one switch (2026-11-05)
+
+Asked how hard a move from Jazzy to a newer ROS 2 would be, the answer was: the code's ROS surface is
+tiny and the record is self-describing MCAP, so the cost is the distro string spread across pixi
+manifests and CI, Python 3.14, and the Zenoh configs. Done before any migration: in rig-nodes the
+distro is a pixi feature (`jazzy` default, `lyrical` opt-in: channel, mutex pin, Python pin), the two
+package manifests name no distro (pixi-build-ros reads the `robostack-<distro>` channel; `rig_nodes`
+depends on RoboStack's distro-neutral `ros2-*` names), every bag records what it ran on (rosbag2
+`custom_data` from `rig_nodes.runtime`), and an allowed-to-fail CI job builds and tests the package
+set on RoboStack Lyrical for the three platforms. `rig/pixi.toml` mirrors the features (the rigs'
+environments unchanged; `lyrical` builds rig-nodes from git since the channel is Jazzy-only). The
+record — what differs between sets, Lyrical availability, rehearsal results, what blocks a real move
+(Python 3.14 for the lab's own scripts, channel maturity, rmw_zenoh 0.10.8's config schema, one distro
+per host, nothing run on hardware, the Jazzy-only channel, binding drift such as `rosbag2_py`'s new
+`sequence_number` argument) — is rig-nodes `docs/ros-distro-switch.md`.
+
 ## 8. Known debts that hold regardless of layout
 
 - Closed-loop math runs inside the gateway's `Pipeline`, and it is stateful: heading unwrapping,

rig/.gitignore

@@ -5,6 +5,8 @@ __pycache__/
 *.pyc
 # bridge-node run logs land here by default
 runlogs/
+# what the BIAS simulator writes when the graph runs from here (rig.py --bias-sim)
+bias-videos/
 *.jsonl
 *.jsonl.gz
 # ...except the recorded test fixtures (okey_smoke_violet_cw.jsonl.gz was silently ignored before 2026-10-22)

rig/LYRICAL-BENCH.md

@@ -0,0 +1,109 @@
+# Lyrical bench session on a lab PC — handover
+
+**Goal.** Run the rig graph on ROS 2 Lyrical Luth (RoboStack, Python 3.14) on a Windows rig PC
+for the first time: install, the no-hardware suites, one Studio run against real FicTrac if a
+camera is there, then an overnight soak. Nothing here touches what the rig runs: the `default`
+environment and its lock are unchanged, and the Lyrical environment is a separate directory you
+can delete afterwards. Record: rig-nodes `docs/ros-distro-switch.md` (what differs, what blocks a
+migration, Mac results); this page is the PC procedure. Not during an experiment, and never with
+another graph or bridge 3.4 on the same PC (one Zenoh router per host).
+
+## 0. Before you start (5 min)
+
+- [ ] pixi ≥ 0.83: `pixi --version`; otherwise `pixi self-update` (replaces one binary, touches no environment).
+- [ ] Visual Studio 2022 Build Tools with **MSVC v143 x64/x86** (Desktop C++): the Lyrical
+      environment compiles `rig_interfaces` from source because the published channel is Jazzy-only.
+      ww18 has it (`rig/WINDOWS-TEST.md`, 2026-10-15). Check: `vswhere -products * -requires Microsoft.VisualStudio.Component.VC.Tools.x86.x64 -property installationPath`.
+- [ ] Short build root, once per checkout (MAX_PATH): in `rig\`, `pixi config set --local detached-environments C:\px`.
+- [ ] The branch: `git fetch origin && git checkout rig/ros-distro-switch` in the webDisplayTools checkout
+      (PR [#294](https://github.com/reiserlab/webDisplayTools/pull/294); its `lyrical` environment builds
+      rig-nodes `main` from git, pinned in the lock to the merge commit of
+      [rig-nodes#8](https://github.com/reiserlab/rig-nodes/pull/8) — network needed; no rig-nodes checkout needed).
+- [ ] Nothing on ports 7447 / 8765 / 60000: `netstat -ano | findstr ":7447 :8765 :60000"` prints nothing.
+
+## 1. Install (expect 2–5 min; write the time down)
+
+```bash
+cd rig
+pixi install --locked -e lyrical
+pixi run -e lyrical python -m rig_nodes.runtime
+```
+
+The second command prints what the environment is (`ros_distro: lyrical`, `rmw 0.10.6`,
+`ros2_distro_mutex: 0.22.0 lyrical_30`, `python 3.14.x`, the package versions); paste it into the
+results file. If the install fails on the `rig_interfaces` build, the error text verbatim into the
+results file is the result — do not fix the PC's compiler setup mid-session.
+
+## 2. No hardware (≈ 5 min)
+
+```bash
+pixi run -e lyrical check-typed        # 8 typed messages import + serialize
+pixi run -e lyrical test-gateway       # 5 checks
+pixi run -e lyrical test               # the arena harness: whole graph, simulators, emulator, crash phase (≈ 60–90 s; 63 checks on the Mac)
+```
+
+Record the counts. A failure in `test` is worth a second run before calling it (the harness has
+timing-sensitive phases); two failures are a finding.
+
+## 3. One real run (≈ 15 min, only if FicTrac and a controller are at hand)
+
+```bash
+pixi run -e lyrical rig                # router + FicTrac adapter (UDP 60000) + gateway; add -- --bias for the camera node
+```
+
+Foxglove during the run: `pixi install --locked -e lyrical-viz` once, then `pixi run -e lyrical rig -- --foxglove`
+(the bridge must come from the Lyrical set; `rig.py` picks `lyrical-viz` when it runs in `lyrical`).
+Start FicTrac as usual (the sample video is fine), open **Production** Studio
+(`https://reiserlab.github.io/webDisplayTools/arena_studio.html?mode=run`), Connect, run any short
+closed-loop protocol, export. Record: frames seen vs applied in the gateway's shutdown line, the
+`runlog-check` of the export, controller 0x70 count vs host, anything odd in the Console. Without a
+controller, skip the Studio and run `pixi run -e lyrical rig -- --sim` + `pixi run -e lyrical studio-emulator`
+instead (what the harness does, by hand).
+
+## 4. The soak (overnight)
+
+First a six-minute smoke so the Windows stop path (CTRL_BREAK through the process group) is seen
+working before anyone leaves it overnight:
+
+```bash
+pixi run -e lyrical soak -- --hours 0.1 --every 1
+```
+
+Expect `N/N emulator runs ok` and `graph clean exit` on the last line. Then the real thing:
+
+```bash
+pixi run -e lyrical soak -- --hours 8 --every 10          # simulated FicTrac at 100 Hz + BIAS simulator
+pixi run -e lyrical soak -- --hours 8 --every 10 --fictrac  # real FicTrac on 60000 instead of the simulator
+```
+
+`tests/soak_graph.py` keeps one graph running for the whole time and drives a recorded run through
+the gateway every 10 minutes with the Studio emulator (the Studio's exact conversation), then stops
+the graph the way a rig does. It refuses to start if the ports are taken. Results land in
+`rig\runlogs\soak-<stamp>\`: `soak.json` (every run: ok / applied frames / seconds; the gateway's
+shutdown accounting; `stop`), `rig.log`, one report per run. What a problem looks like in `soak.json`:
+`timed_out: true` or `ok: false` on a run (an rmw hang or a stuck gateway), `applied` dropping over
+the night, `stop` not starting with `clean`, or the graph exiting on its own. Leave the PC alone
+while it runs; the Studio must stay disconnected (the emulator owns the gateway).
+
+For comparison, the same command in the `default` environment (`pixi run soak -- --hours 8 --every 10`)
+is the Jazzy baseline; if there is a second night, run it.
+
+## 5. Results and cleanup (10 min)
+
+- Results file: `rigs/<rig-id>/lyrical-bench-<date>.md` in the private `reiserlab/arena-experiments`
+  (rig records left this repository on 2026-10-29, LAB-227): the runtime print, install time, the
+  counts from steps 2–3, the `soak.json` summary line and the `summary_lines`, every error verbatim.
+  Attach `soak.json`.
+- Then in rig-nodes (a small PR on top of #8 or after it): the host table row in `SETUP.md` and a
+  line in `docs/ros-distro-switch.md` §8 with the PC's results.
+- Cleanup: `rmdir /s /q rig\.pixi\envs\lyrical` (or leave it; it is 1–2 GB and harms nothing).
+  The `default` environment was never touched; `pixi install --locked -e default` confirms.
+
+## Known already
+
+- The one Lyrical-only defect found so far was in `rosbag2_py` (positional stamps bound to a new
+  `sequence_number` argument) and is fixed on the branch; the recorder test checks it.
+- rig-nodes' CI Lyrical job passed on win-64 and linux-64; one host-sampler disk-busy check flaked
+  on osx-arm64 once (timing, not Lyrical).
+- The rig-nodes channel publishes Jazzy only; the `lyrical` environment always builds from git.
+- Do not `pixi update` on the rig PC; a changed lock is a finding to report, not a fix.