From f5cf5a668f9179fb8eed431fa9975e1c3a34e002 Mon Sep 17 00:00:00 2001 From: Michael B Reiser Date: Wed, 28 Oct 2026 17:20:56 -0400 Subject: [PATCH 1/4] =?UTF-8?q?rig:=20host=20monitor=20node=20=E2=80=94=20?= =?UTF-8?q?the=20PC's=20pressure=20on=20/rig/host,=20recorded=20in=20every?= =?UTF-8?q?=20run's=20bag?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A generic node (nothing FicTrac- or arena-specific) that publishes, once a second, CPU per core (window mean + busiest 100 ms sub-window, P/E-core class, sampler lateness, context switches / interrupts / DPCs), memory + page file + hard faults, log-volume free space, per-disk and per-adapter rates, NVIDIA GPU (incl. NVENC/NVDEC) and the top processes plus pinned groups (FicTrac, BIAS, MATLAB, browser, Foxglove, router, each graph node). The method is atlas-bench tools/cpu_sampler.py: cycle counters and one NtQuerySystemInformation call for all processes on Windows; GetIfEntry2 for network; NVML through ctypes; psutil for the rest and on Linux/macOS. - rig/nodes/host_sampler.py (no ROS) + host_monitor.py: rig-nodes material, prototyped here. Typed rig_interfaces/HostStats when installed (requested for rig-nodes 0.3.0; spec in rig/docs/host-stats-request.md), else String JSON with the same field names. - bridge_node.py records /rig/host into the bag; rig.py starts the monitor by default (--no-host-monitor, --host-monitor-args). - Cost on ww18: 0.9 % of one core (median); 4.9 % of the harness run's compressed payload, about 1.0 MB/h. Gated in the harness (<= 2 % of a core, <= 5 % of the payload). - tests/test-host-sampler.py (root suite, no ROS/psutil): fields and types against the .msg spec, rates, CPU mean vs peak, process selection, strict JSON. Co-Authored-By: Claude Opus 5.5 --- docs/development/rig-host-monitor.md | 85 ++++ rig/README.md | 1 + rig/docs/host-stats-request.md | 105 +++++ rig/nodes/bridge_node.py | 13 + rig/nodes/host_monitor.py | 175 ++++++++ rig/nodes/host_sampler.py | 639 +++++++++++++++++++++++++++ rig/nodes/rig.py | 6 + rig/tests/test_ros2_poc.py | 68 +++ tests/test-host-sampler.py | 215 +++++++++ 9 files changed, 1307 insertions(+) create mode 100644 docs/development/rig-host-monitor.md create mode 100644 rig/docs/host-stats-request.md create mode 100644 rig/nodes/host_monitor.py create mode 100644 rig/nodes/host_sampler.py create mode 100644 tests/test-host-sampler.py diff --git a/docs/development/rig-host-monitor.md b/docs/development/rig-host-monitor.md new file mode 100644 index 00000000..10cda472 --- /dev/null +++ b/docs/development/rig-host-monitor.md @@ -0,0 +1,85 @@ +# Rig host monitor — the PC's "pressure" in every run + +Status: v1, 2026-10-28 (`rig/nodes/host_sampler.py` + `rig/nodes/host_monitor.py`, ww18). The message +spec sent to rig-nodes: `rig/docs/host-stats-request.md`. + +## What and why + +A generic node that publishes how loaded the rig PC is, like Task Manager or Performance Monitor +but recorded: CPU per core, memory, disk, network, GPU, and the processes responsible. It knows +nothing about FicTrac or the arena. `pixi run rig` starts it by default (`--no-host-monitor` turns +it off) because it costs little enough to leave on (§4). The arena gateway records `/rig/host` into +every run's MCAP bag, so a run that went wrong carries the PC's state alongside it, and a question +like "does Foxglove cost anything" is answered by comparing bags instead of a separate experiment. + +The method is atlas-bench `tools/cpu_sampler.py` (commit 306fa6d, used on ww18 for the BIAS +CPU-spike work): cycle counters, 100 ms CPU sub-windows, one system call for all processes. + +## What one sample holds (1 Hz, `rig_interfaces/HostStats`) + +| area | fields | source (Windows · elsewhere) | +|---|---|---| +| CPU | per-core busy, window mean **and** busiest 100 ms sub-window; whole-PC busy cores (mean, peak); P/E-core class per CPU; how late the sampler woke (a PC-wide stall); context switches, interrupts, DPCs per s | idle-cycle counters (exact) + `GetSystemCpuSetInformation` + kernel DPC counts · psutil (tick-based) | +| memory | total, used, available, page file used, hard page faults per s | psutil + Σ per-process hard faults · psutil (no hard faults) | +| storage | per physical disk: read/write MB/s, IOPS, busy fraction; free space on the run-log volume | psutil | +| network | per adapter that is up (hardware, not loopback): rx/tx MB/s, errors + drops | `GetIfEntry2` per adapter · psutil | +| GPU | per NVIDIA GPU: core + memory-controller utilization, memory, NVENC/NVDEC load, temperature, power, P-state, throttle reasons | the driver's NVML through ctypes (no package) | +| processes | top 3 by CPU, by memory, by disk I/O (`--top`), plus one summed row per pinned group (router, FicTrac, BIAS `test_gui.exe`, MATLAB, browser, Foxglove app, foxglove_bridge, each graph node as `py: