Feat/data api

[wyzula-jan]

Description

This PR ports all plotting widgets to the new Data API introduced in bec-project/bec#1020: Waveform,
ScatterWaveform, Heatmap, Image, MultiWaveform and
MotorMap now receive their data exclusively through client.data_api subscriptions.

Architecture

One bridge, one render slot. A new QtDataSubscription (bec_widgets/utils/) wraps a Data API subscription and
marshals every update onto the Qt main thread as a signal. Widgets connect one slot and render the immutable,
pre-aligned columnar snapshot they receive — all buffering, pairing, trimming and scan-change logic that widgets
previously re-implemented on raw Redis endpoints is gone. update.reason tells the widget whether to append ("live")
or rebuild ("backfill", "rebind", "history"); since every update is full-state, dropped intermediate updates are
harmless by design.

Legacy data paths removed — intentionally not backwards compatible. The old direct-Redis data access in the plot
widgets is deleted, not kept as a fallback. This is deliberate: a fallback would let data access silently leak through
the old system and undermine the single data contract (correct pairing, size gating, history re-route). Consequently
this branch requires bec from the feat/data_api branch (bec-project/bec#1020), and the test suite must be run
against it — against bec main, the widgets cannot import the Data API and the suite fails at collection.

Live ↔ history unification. Widgets no longer distinguish live and historic data: a subscription bound to a running
scan is served live and re-routes itself to the authoritative history snapshot when the scan finishes; loading an old
scan uses the same render path.

Additional improvements on top of the port

• update_rate property on PlotBase (1–100 Hz, per-widget defaults, settable at runtime from code, RPC or
  Designer) — wired through every bridge via the subscription's coalescing interval.
• Large-dataset guard on the Data API size gate: oversized history loads are estimated up front and confirmed per
  dataset instead of silently freezing the GUI; confirmed scans are remembered.
• Oversized-curve rendering (Waveform): min/max envelope decimation keeps extremes visible, zooming re-renders the
  visible window down to raw samples (no data loss for inspection), and a labelled progress bar below the plot shows the
  chunked load progress.
• Numpy-native rendering: history columns arrive as ndarray and are rendered without conversion round-trips.
• Incremental async rendering matching main's update rates, with a throughput benchmark
  (tests/unit_tests/benchmarks/test_data_api_widget_throughput.py).
• Colorbar autorange fix (Image): frame renders no longer disable autorange and spam the log on every update —
  pre-existing bug, kept as its own commit.
• MultiWaveform: device and signal selection with class-based routing, replacing the monitor-only selection.
• Test isolation fixtures for Data API instances and device signal info, removing order-dependent test pollution.

Related Issues

Companion PR: bec-project/bec#1020 (required).

Type of Change

• Refactor (widget data paths) — breaking with respect to bec main by design, see above
• New features (update_rate, size-gate UX, decimation + load progress)
• Bug fix (colorbar autorange)

How to test

Install bec_lib from the companion branch, then run the suite:

pip install -e "<bec-checkout-on-feat/data_api>/bec_lib[dev]"
pytest --random-order tests/unit_tests/

• plot live/historic data in all widgets with different readout priorities
• gui should be not freezing on any update
• compare side by side with old version on main (in worktree for example)
• try to load some huge dataset to waveform (personally tested with 1GB dataset created by async Waveform) -> loading
  should be incremental with progressbar + zooming should render details

Potential side effects

• Requires bec feat/data_api — both packages must be upgraded together in one environment. Server-side deployments
  are unaffected (no message format changes).
• The RPC surface changed accordingly (regenerated cli/client.py): removed legacy monitor-selection API on
  MultiWaveform, added update_rate and the size-gate properties.

[Copilot]

Pull request overview

This PR refactors multiple plot widgets to receive and render data via the BEC DataAPI subscription model (instead of legacy dispatcher endpoints / periodic fetch paths), introducing a Qt-thread marshaling bridge (QtDataSubscription) and updating the unit tests to validate the new update semantics (including scan rollover behavior, history pinning, and dataset-size gating).

Changes:

• Add bec_widgets.utils.qt_data_subscription.QtDataSubscription to marshal DataAPI updates onto the Qt thread, drop stale scan payloads, and manage subscription lifecycle.
• Migrate widgets (Waveform-related tests, Heatmap, ScatterWaveform, MotorMap, MultiWaveform, Image) to DataAPI-driven rendering paths with updated scan/history handling.
• Extend/adjust test infrastructure and test suites to use DataAPI-style SubscriptionUpdate snapshots and isolate DataAPI/device-signal state across tests.

Reviewed changes

Copilot reviewed 22 out of 23 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/unit_tests/test_waveform.py	Replaces legacy waveform fetch/update tests with DataAPI snapshot-based rendering tests (sync/async/x-mode/history/size-guard coverage).
tests/unit_tests/test_scatter_waveform.py	Updates scatter waveform tests to validate DataAPI subscription setup, replacement, history binding, and live switching.
tests/unit_tests/test_qt_data_subscription.py	Adds focused unit tests for the new QtDataSubscription bridge behavior (thread marshalling, stale dropping, size-gate, idempotent close).
tests/unit_tests/test_multi_waveform_next_gen.py	Reworks MultiWaveform tests for DataAPI routing (monitor_1d vs preview vs async) and snapshot/scan rollover rendering.
tests/unit_tests/test_motor_map_next_gen.py	Updates MotorMap tests to validate DataAPI readback stream handling, ordinal filtering, and subscription lifecycle.
tests/unit_tests/test_image_view_next_gen.py	Migrates Image widget tests from dispatcher endpoints to DataAPI updates, including preview/async routing and rollover behavior.
tests/unit_tests/test_heatmap_widget.py	Migrates heatmap tests to DataAPI subscriptions and validates history pinning, cached re-rendering, and legacy-path removal.
tests/unit_tests/conftest.py	Adds autouse fixtures to isolate DataAPI instances and restore shared mock device signal configs between tests.
tests/unit_tests/benchmarks/test_data_api_widget_throughput.py	Introduces a DataAPI rendering throughput benchmark-style test for waveform update processing.
bec_widgets/widgets/plots/waveform/curve.py	Removes unused legacy slice_index state from waveform curve.
bec_widgets/widgets/plots/scatter_waveform/scatter_waveform.py	Refactors ScatterWaveform to render from DataAPI updates and manage a subscription lifecycle (live vs history).
bec_widgets/widgets/plots/multi_waveform/toolbar_components/monitor_selection.py	Removes legacy monitor-selection toolbar component in favor of shared device+signal selection.
bec_widgets/widgets/plots/multi_waveform/toolbar_components/init.py	(Module presence in PR context) aligns toolbar components package structure after monitor_selection removal.
bec_widgets/widgets/plots/multi_waveform/multi_waveform.py	Refactors MultiWaveform to use shared DeviceSelection toolbar and DataAPI subscriptions, adding routing + connection state.
bec_widgets/widgets/plots/motor_map/motor_map.py	Refactors MotorMap trail updates to DataAPI readback stream snapshots and adds subscription lifecycle management.
bec_widgets/widgets/plots/image/toolbar_components/device_selection.py	Generalizes DeviceSelection to support ndim filters and an optional monitor_1d sentinel entry.
bec_widgets/widgets/plots/image/image.py	Refactors Image widget to DataAPI subscriptions for preview/async signals, with rollover and cached 1D buffer logic.
bec_widgets/widgets/plots/heatmap/heatmap.py	Refactors Heatmap to DataAPI column updates, cached re-rendering on property changes, and live/history bridge switching.
bec_widgets/widgets/control/device_input/signal_combobox/signal_combobox.py	Adds support for static “extra” signal entries and makes get_signal_config() resilient to editable combobox text changes.
bec_widgets/widgets/control/device_input/device_combobox/device_combobox.py	Adds option to include async-readout devices in device listings (for monitor_1d-style streams).
bec_widgets/utils/qt_data_subscription.py	Adds the new QtDataSubscription bridge implementation.
bec_widgets/cli/client.py	Updates generated CLI surface for MultiWaveform changes (plot signature + monitor_signal property).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.