A live fit another program controls: MicroClaw support (smappy 0.3.0)

CLAUDE.md

@@ -101,6 +101,7 @@ shape, so read the one closest to what you are writing:
 | runs the ROI manager's segment, evaluate and analyse as one | `npc_analysis.chain.yaml` (a shipped chain: an evaluator step runs over every ROI before the next; `docs/batch.md`, "ROIs: segment, evaluate, analyse") |
 | opens an interactive window of the GUI's | `bead_calibration.py` (`Plugin.window`: the button opens what the GUI registered in `session.window_openers`) |
 | runs an external program in its own Python | `uipsf.py` (`smappy.uipsf`: a worker script under that program's interpreter, progress read from its output, the result converted into smappy's own file) |
+| is run by MicroClaw during an acquisition | `gui/live_session.py` (the GUI on a fit another program controls through `Context.stop` and `writer_finished`; the package and its protocol runner are `ries-lab/smappy-microclaw`) |
 
 A chain of plugins that runs as one, and running one over many files
 (`smappy-batch`, the batch window): `docs/batch.md`.
@@ -184,6 +185,9 @@ written this way; it is also what makes the tests readable.
   never assume a session exists.
 * `ctx.report(text)` for progress, `ctx.emit(event, payload)` to hand partial
   results on.  Both are no-ops when nobody is listening.
+* `ctx.stop` and `ctx.writer_finished` -- `threading.Event`s or None, set from
+  another thread: end the run early keeping what it has, and "the live source
+  is complete".  The Fit plugin honours both; a long plugin may check `stop`.
 * The **grouped** table (one row per blink) is a table of its own, with its own
   filter, at `session.layers[i].state.sets["grouped"]`; it exists only once the
   user has switched that layer to grouped, and `state.grouped_stale` says it no

NOTES.md

@@ -728,9 +728,30 @@ mid-write raises instead of returning half an image -- and the file list is
 re-globbed on each poll, so the `_1.ome.tif` that appears when the current file
 fills up is picked up without reopening anything.
 
+**Unless somebody says when it ended.**  A program that drives the microscope
+(MicroClaw) knows when the writer has finished, and then a pause -- a refocus,
+a laser change, a time-lapse interval -- must not end the fit: `finished`, an
+event handed to the watch, replaces the timeout altogether, and once it is set
+one more pass reads everything and the stream ends.  A longer timeout would
+only move the pause that ends the fit too early, which is why MicroClaw's
+protocol (its `design/84`) asks for the event and not for a number.  The
+Fit plugin takes it from `Context.writer_finished`, and `Context.stop` ends a
+run early with its file closed and nothing finished (no drift correction of
+half an acquisition; the caller allows ten seconds).
+`smappy.gui.live_session` is the GUI opened on such a run: the same session
+and windows, the run started without its preflight question, and the fit's
+file added to `Session.protected`, because the caller hands it on with a
+digest while the window stays open and a Ctrl+S would change what that digest
+vouches for.
+
 **The engine is flushed on a timer as well as when its ROI buffer fills.**  The
 buffer holds 15000 ROIs; a sparse sample would take minutes to fill it, and
-nothing would appear in the meantime.
+nothing would appear in the meantime.  The timer is checked once per block, so
+during a pause the watch sends empty blocks (`idle_blocks`) for it to be checked
+on -- without them, what was buffered before a pause stayed off the screen
+until the pause was over.  The GUI's live fit flushes every 5 s
+(`LIVE_FLUSH_SECONDS`); `smappy.live.LiveFit` still has its own loop and does
+not send idle blocks yet.
 
 Checked against the offline path on 300 frames of the astigmatic dataset,
 replayed frame by frame into a growing two-file series: same 28724

pyproject.toml

@@ -12,7 +12,7 @@ build-backend = "setuptools.build_meta"
 # package (a Smappee energy-monitor wrapper, last released in 2018).  The import
 # name is unaffected: `pip install smappy-smlm`, then `import smappy`.
 name = "smappy-smlm"
-version = "0.2.0"
+version = "0.3.0"
 description = "Single-molecule localization fitting pipeline (Python port of the SMAP fast-simple workflow)"
 readme = "README.md"
 license = "BSD-3-Clause"

src/smappy/gui/app.py

@@ -817,7 +817,7 @@ def open_image(self) -> None:
 
     def save(self) -> None:
         from ..io.formats import writer_for
-        if self.session.path is None:
+        if self.session.path is None or self.session.is_protected(self.session.path):
             return self.save_as()
         try:
             writer_for(self.session.path)
@@ -830,11 +830,18 @@ def save(self) -> None:
 
     def save_as(self) -> None:
         start = str(self.session.path) if self.session.path else folders.start()
+        if self.session.is_protected(start):
+            here = Path(start)          # offer a new name beside the kept file
+            start = str(here.with_name(f"{here.stem}_edited{here.suffix}"))
         path, _ = QFileDialog.getSaveFileName(self, "Save localizations", start,
                                               "HDF5 (*.h5 *.hdf5)")
         if path:
             folders.remember(path)
-            self.session.save(path)
+            try:
+                self.session.save(path)
+            except ValueError as error:
+                QMessageBox.warning(self, "Not saved", str(error))
+                return
             self._on_session("locs")
 
 

src/smappy/gui/live_session.py

@@ -0,0 +1,252 @@
+"""The GUI, opened on a fit that is already running, for a program that drives it.
+
+MicroClaw runs the microscope and starts an analysis while the acquisition is
+being written; the SMAPpy package it installs (`ries-lab/smappy-microclaw`)
+calls this to open the ordinary GUI on that dataset.  It is the same session,
+the same two windows and the same Fit plugin a user would start by hand -- the
+point is that the person at the microscope sees the reconstruction build up
+and can work with it in the program they already know -- with three things a
+hand-started fit does not need:
+
+* **The run is started here, with no question asked.**  Nobody may be at the
+  keyboard, so the plugin's preflight is skipped; the caller checks what it
+  would have asked before calling.
+* **It is controlled from another thread.**  `stop` ends the fit early and
+  keeps the file; `writer_finished` says the acquisition is complete, after
+  which the watch reads what is left and the fit ends -- not after an idle
+  timeout, because a pause in an acquisition is not its end
+  (`smappy.io.watch`).  Both are `threading.Event`s, so a reader thread can set
+  them without touching Qt.
+* **The fit's file is frozen once it is done.**  The caller hands it on with a
+  digest, and the window stays open, so the session refuses to save over it
+  (`Session.protected`); what the user does afterwards goes to a new file.
+
+`on_finished` is called exactly once, on the GUI thread, when the fit has
+ended and its file is closed -- also when the windows were closed while it ran,
+which is a cancellation.  `run` returns only once the windows are closed.
+"""
+from __future__ import annotations
+
+import sys
+import threading
+import time
+import traceback
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Any, Callable, Dict, List, Optional
+
+from .. import plugins
+from ..plugins import param_specs, settings_from, settings_values
+
+
+@dataclass
+class LiveOutcome:
+    """How a fit started by `LiveSession` ended."""
+
+    state: str                    # "succeeded", "cancelled" or "failed"
+    complete: bool                # every frame the writer wrote was read
+    path: Optional[Path] = None   # the fit's file, closed; None: nothing saved
+    stats: Dict[str, Any] = field(default_factory=dict)
+    error: str = ""               # one line, for "failed"
+    traceback: str = ""
+
+
+def fit_settings(plugin_path: str, values: Optional[Dict[str, Any]] = None):
+    """The settings of the fitter at ``plugin_path``: its defaults under ``values``.
+
+    ``values`` is a flat dotted map (``{"fit.roisize": 13}``).  Unlike
+    `plugins.settings_from`, which forgives a saved name the plugin no longer
+    has, this refuses one: the values come from a call somebody just made, and
+    a misspelt camera offset that silently kept its default would be a wrong
+    fit with nothing to say so.
+    """
+    cls = plugins.get(plugin_path)
+    values = dict(values or {})
+    known = set(_dotted_names(cls.Settings))
+    unknown = sorted(set(values) - known)
+    if unknown:
+        raise ValueError(f"{plugin_path} has no setting {', '.join(unknown)}; "
+                         f"it has {', '.join(sorted(known))}")
+    settings = settings_from(cls.Settings, values)
+    got = settings_values(settings)
+    refused = sorted(k for k, v in values.items() if got.get(k) != v)
+    if refused:
+        # settings_from falls back to the defaults on a value the class
+        # refuses; here that is an error, not a reset
+        raise ValueError(f"{plugin_path} refused {', '.join(refused)}")
+    return settings
+
+
+def _dotted_names(settings_cls, prefix: str = "") -> List[str]:
+    out = []
+    for name, spec in param_specs(settings_cls).items():
+        if spec.children is not None:
+            out += _dotted_names(spec.type, f"{prefix}{name}.")
+        else:
+            out.append(prefix + name)
+    return out
+
+
+class LiveSession:
+    """The smappy GUI with the fitter at ``plugin_path`` running ``settings``."""
+
+    def __init__(self, plugin_path: str, settings, *,
+                 on_finished: Optional[Callable[[LiveOutcome], None]] = None,
+                 on_progress: Optional[Callable[[str], None]] = None,
+                 on_closed: Optional[Callable[[], None]] = None,
+                 stop: Optional[threading.Event] = None,
+                 writer_finished: Optional[threading.Event] = None):
+        self.plugin_path = plugin_path
+        self.settings = settings
+        self.on_finished = on_finished
+        self.on_progress = on_progress
+        self.on_closed = on_closed
+        # the caller's own events when it has them -- a protocol reader that
+        # sets them as the messages arrive -- else ours, set through these
+        self.stop = stop or threading.Event()
+        self.writer_finished = writer_finished or threading.Event()
+        self.outcome: Optional[LiveOutcome] = None
+        self.session = None
+        self.control = None
+        self.render = None
+        self.panel = None
+        self._window = None
+
+    # ------------------------------------------------------------- running
+    def open(self) -> None:
+        """Build the windows and start the fit; the event loop is the caller's.
+
+        `run` is this plus the loop.  Split so a test can drive the loop
+        itself.
+        """
+        from PySide6.QtWidgets import QApplication
+
+        from ..session import Session
+        from .app import ControlWindow, RenderWindow
+        from .collector import collect_on_gui_thread
+        from .widgets import apply_style
+
+        app = QApplication.instance() or QApplication([sys.argv[0]])
+        apply_style(app)
+        collect_on_gui_thread(app)
+        self.session = Session()
+        out = self.settings.output.resolve(self.settings.source.path)
+        if out is not None:
+            self.session.protected.add(Path(out))
+        self.render = RenderWindow(self.session)
+        self.control = ControlWindow(self.session, self.render)
+        screen = app.primaryScreen().availableGeometry()
+        self.control.move(screen.left(), screen.top())
+        self.render.move(screen.left() + self.control.width() + 20, screen.top())
+        self.render.show()
+        self.control.show()
+
+        self.panel = self._panel()
+        self.panel.form.set(self.settings)
+        self.panel.ended.connect(self._ended)
+        self.panel.progressed.connect(self._progressed)
+        if not self.panel.start_run(ask=False, stop=self.stop,
+                                    writer_finished=self.writer_finished):
+            self._finish(LiveOutcome("failed", False,
+                                     error=self.panel.status.text() or
+                                     "the fit did not start"))
+
+    def run(self) -> int:
+        """Open, and run the event loop until the windows are closed."""
+        from PySide6.QtWidgets import QApplication
+
+        try:
+            self.open()
+        except Exception as error:
+            # a failure to build the windows is the fit's failure: the caller
+            # is owed an outcome whatever happens
+            self._finish(LiveOutcome("failed", False, error=_one_line(error),
+                                     traceback=traceback.format_exc()))
+            if self.on_closed is not None:
+                self.on_closed()
+            return 1
+        code = QApplication.instance().exec()
+        self.wait()
+        if self.on_closed is not None:
+            self.on_closed()
+        return code
+
+    def wait(self) -> None:
+        """After the windows have closed: end a fit that is still running.
+
+        Closing the windows before the fit has finished is a cancellation --
+        the user has decided not to look -- and the fit closes its file and
+        reports as a stopped run does.  Its last signals are queued for the
+        GUI thread, so they are delivered here by hand.
+        """
+        from PySide6.QtWidgets import QApplication
+
+        if self.outcome is None:
+            self.stop.set()
+        app = QApplication.instance()
+        while self.outcome is None:
+            app.processEvents()
+            time.sleep(0.02)
+        thread = getattr(self.panel, "_thread", None)
+        if thread is not None:
+            thread.wait()
+
+    # ------------------------------------------------------------ internals
+    def _panel(self):
+        """The fitter's panel in the Localize tab, or a window of its own.
+
+        The tab is where a user would look for it; a workspace that has
+        unpinned it still gets the fit, in the window the Plugins menu opens.
+        """
+        for index in range(self.control.tabs.count()):
+            tab = self.control.tabs.widget(index)
+            slots = getattr(tab, "slots", None)
+            if not slots:
+                continue
+            for instance in tab.tab.instances:
+                if instance.plugin != self.plugin_path:
+                    continue
+                panel = slots[instance.id].ensure()
+                if panel is not None:
+                    self.control.tabs.setCurrentIndex(index)
+                    tab.open_section(instance.id)
+                    return panel
+        from .plugin_panel import PluginWindow
+        self._window = PluginWindow(plugins.get(self.plugin_path), self.session,
+                                    self.control)
+        self._window.show()
+        return self._window.panel
+
+    def _progressed(self, text: str) -> None:
+        if self.on_progress is not None and self.outcome is None:
+            self.on_progress(text)
+
+    def _ended(self, kind: str, payload) -> None:
+        if self.outcome is not None:
+            return                       # a later run the user started by hand
+        if kind == "failed":
+            lines = str(payload).strip().splitlines()
+            # stopped before it had a file -- while the dataset was still
+            # being waited for, say -- is still a stop, not a broken fit
+            state = "cancelled" if self.stop.is_set() else "failed"
+            self._finish(LiveOutcome(state, False,
+                                     error=lines[-1] if lines else "failed",
+                                     traceback=str(payload)))
+            return
+        data = payload.data or {}
+        stopped = bool(data.get("stopped"))
+        live = self.settings.source.live
+        complete = not stopped and (self.writer_finished.is_set() or not live)
+        path = data.get("path")
+        self._finish(LiveOutcome("cancelled" if stopped else "succeeded", complete,
+                                 path=Path(path) if path else None,
+                                 stats=dict(data.get("stats") or {})))
+
+    def _finish(self, outcome: LiveOutcome) -> None:
+        self.outcome = outcome
+        if self.on_finished is not None:
+            self.on_finished(outcome)
+
+
+def _one_line(error: BaseException) -> str:
+    return f"{type(error).__name__}: {error}".splitlines()[0]