Bound concurrent runs and remove idle terminal-history scans

docs/contracts/external-recipes.md

@@ -98,7 +98,10 @@ retry means a fresh preflight, key and approval, not reuse of the old grant.
 
 ## Process custody and recovery
 
-The sole Runs queue starts one pinned single-threaded Linux subreaper helper.
+The sole Runs supervisor starts one pinned single-threaded Linux subreaper helper
+per dispatched attempt. Resource-aware dispatch can overlap independent helpers;
+each has its own control channel, process identity, custody directory and receipt.
+See [Run resource scheduling](workspace-runs.md) for CPU, CUDA and memory budgets.
 A private bounded socket protocol carries READY, GO, cancellation and custody
 receipts; recipe stdout cannot declare completion. The helper and executable
 bytes use sealed descriptors. Recipe code receives its own stdin/stdout/stderr,

docs/contracts/managed-torch-runtime.md

@@ -99,3 +99,10 @@ snapshot as a bound trainer. This keeps fresh CPU restore and its typed Run
 artifact inputs in the current Project contract. Bound evidence still requires
 Project v2; unbound field-evidence recipes and other ordinary runtime profiles
 are not admitted through this Project v2 path.
+
+Explicit managed CUDA recipes may reserve1–1800seconds of execution. Ordinary
+CPU recipes retain their600second bound. The neutral custody owner and recipe
+admission share the CUDA maximum; operators still choose a smaller exact
+per-attempt deadline. Exceeding the requested deadline stops/drains the attempt
+and is not a numerical accuracy result. Installed-runtime verification before
+leader creation retains its separate finite preparation allowance.

docs/contracts/workspace-runs.md

@@ -19,7 +19,36 @@ catalog transaction serialize competing admissions.
 A committed queued receipt whose acknowledgment was lost wakes the existing
 owned, unblocked queue when the caller explicitly reconciles the same request.
 It never creates a second run. Restarted catalogs do not replay old queue entries.
-There are at most eight active admitted attempts and one active training child.
+There are at most eight active admitted attempts. The single catalog supervisor
+dispatches independent attempts concurrently through an owned executor: at most
+one CUDA attempt and a configurable number of CPU attempts. `run_capacity.py`
+owns resource selection; `Runs` retains admission, job authorization, execution,
+publication and recovery. A queued CUDA attempt cannot block an eligible CPU
+attempt behind it. Unknown custody fences all further dispatch until recovery.
+
+`TULDOK_CPU_RUN_SLOTS` selects 1–7 CPU attempts (default up to seven, derived from
+available CPUs with two reserved for interactive work). CPU placements are disjoint
+within this supervisor and inherited by child processes. Recipes can reserve
+`resources.cpu_cores` (1–256); otherwise CPU attempts receive one core and CUDA
+attempts up to two. Numerical thread settings remain with their adapter. Affinity
+is cooperative trusted-local placement, not a security sandbox. Placement and
+admission budgets are retained as `execution_resources` on each dispatched Run.
+
+`TULDOK_CPU_RUN_MEMORY_MIB`, `TULDOK_CUDA_RUN_MEMORY_MIB` and
+`TULDOK_RUN_MEMORY_RESERVE_MIB` select positive budgets of at least64 MiB; defaults
+are1024/4096/1024 MiB. A recipe's optional `resources.memory_bytes` can increase
+its reservation but cannot reduce the operator floor. Dispatch requires current
+host/cgroup available memory to cover all active reservations, the new attempt
+and the host reserve. Missing memory observation leaves work queued. Reservations
+are conservative admission accounting, not per-process OS memory limits or an
+OOM guarantee; trusted recipes must separately bound their allocations. The
+64 GiB lab profile selects4096/8192/8192 MiB and explicit per-recipe budgets
+for larger CPU batches. Other applications and catalogs do not share these leases.
+
+The coordinator holds every executor future until the attempt finishes its
+publication/cessation path. Shutdown requests cancellation of every active
+attempt and joins the entire executor before releasing catalog metadata. Restart
+reconciles each durable attempt independently and never replays old queue entries.
 
 `list()` and `get(id)` return real mode, dataset/run IDs, saved graph snapshot,
 graph and release SHA, graph revision, adapter/runtime/parameters, attempt and
@@ -175,7 +204,7 @@ and produce reasons without comparative metric values. Current record editing
 continues through Workbench; retained inputs and prior evidence stay immutable.
 
 Built-in model-aware trials use the same owner-only Runs admission, finite Study
-reservation transaction, eight-attempt queue and one worker. A membership/budget
+reservation transaction and eight-attempt queue with resource-aware dispatch. A membership/budget
 or external execution grant cannot launch a built-in trainer as an agent. The
 supervisor captures and checks the new evidence automatically; users configure
 neither logging nor manual artifact imports. Runtime installation is separate

docs/plans/concurrent-cpu-cuda/plan.md

@@ -0,0 +1,57 @@
+# Concurrent local CPU and CUDA Runs
+
+Status: Accepted; operation start. User authorized simultaneous CPU Runs alongside
+one CUDA fit and continuing the pressure-model sequence. Owner: root; branch
+performance/concurrent-cpu-cuda-20261017, base8af5a6a1e288e5b2f86a75b53cab93ca71222a04.
+Retain this branch for review/integration; existing PR89 and its source branch
+remain unchanged. No rewrite, merge or cleanup.
+
+Objective: remove the serial queue bottleneck while preserving exact admissions,
+per-attempt custody, cancellation, drainage, restart and immutable publication.
+Canonical consumer contract: docs/contracts/workspace-runs.md.
+
+Composition: Runs remains the sole catalog supervisor and owner of durable Run
+state, execution futures and shutdown. The existing standard-library executor
+provides bounded threading; run_capacity owns only admission budgets, CPU placement
+and device classification. Custodians remain one per attempt. Authoritative
+recipe resource requests are captured with material; observed placement is Run
+execution metadata. No second queue, process supervisor or authority registry.
+Changing a capacity budget affects run_capacity and its configuration/tests;
+changing custody affects recipe_process independently. Removing capacity would
+move resource accounting/placement back into Runs; removing the executor would
+restore serial execution. Necessary interleaving is admission before dispatch,
+and confirmed cessation/publication before lease release. All copied numerical
+sources, runtime assets, model weights and scientific thresholds are unchanged.
+
+Write set: runs.py, run_capacity.py, recipe_contract.py, affected tests,
+docs/contracts/{workspace-runs,external-recipes}.md and this plan directory.
+No storage schema/version change; prior rows remain retained. New resource
+requests use the current recipe decoder and material hash; stale deployments
+reject those fields rather than guessing them. No runtime compatibility shim.
+
+Acceptance: focused resource/accounting tests; actual independent overlapping CPU
+custodians; disjoint inherited CPU affinity; independent and queued cancellation;
+CPU bypass of a blocked CUDA slot; exclusive CUDA dispatch; shutdown joins all
+attempts; existing storage-loss/restart/unknown-state lifecycle tests. Actual CUDA
+computation and CPU analysis overlap must be observed in the lab before deployment
+is qualified for that workload. Source/Spec review and exact staged diff required.
+
+Current phase: terminal for scheduler implementation and hardware qualification.
+Next slice: none in this plan; deployment and model experiments continue in the
+linked lab plan. All required process checks and both independent review axes
+pass. Real CUDA pressure training overlapped managed CPU computation for1.2556s;
+the one-update checkpoint restored on CPU with zero updates and CUDA uninitialized.
+Exact restored weights/normalization/history and joined executor workers are
+recorded in actual-device-overlap-03.json (270.023s total). The old
+TinyCNN runtime is unavailable locally; its test fails at runtime preflight before
+the scheduler. No package install or fallback to the lab's CUDA runtime is used.
+
+Independent review found a late dispatch/fence race and one uncoordinated drain
+fence writer. The real red startup regression, passing13-case built-in lifecycle
+receipt10,14-case external/concurrency receipt09 and the final review retain both
+findings and their closure. Commit correction preserves the original history.
+
+Related scientific plan: rheon-surrogate-lab/docs/plans/pressure-throughput-feedback-v1/plan.md.
+Standards route snapshot:v1:bc3877fc-df9b-41cc-9a60-ec680a5ccf55,
+21 selected policies, zero unresolved routing facts. Reservations are admission
+budgets, not an OS memory sandbox; other applications/catalogs do not share leases.