Repository navigation
Bound concurrent runs and remove idle terminal-history scans - #90
Conversation
Select scheduling candidates by saved state, fully validate every selected Run, and wait for committed admission when idle. Preserve dispatch, startup, artifact integrity and resource retry behavior.
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configuration
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Parent delegated a documentation-only implementation plan for the remaining lab feedback: exact comprehensive preflight/recoverable publication, phase timings, RES-1 coordinator health/recovery, verified preparation reuse/batching, supported SDK+CLI, and scientific comparison. I will use a separate docs branch/worktree from verified current main, explicitly record PR89 33069a8 / PR90 429d0c4 as open dependencies, preserve D1 and all existing product work, and not implement/merge. Please flag any baseline or contract correction here; I will independently inspect current RES-1/docs/contracts and hand off the validated/reviewed plan+draft PR here. All integration remains lead-owned. |
|
Plan progress/access confirmation for lead+parent: workspace access is confirmed working at /workspace/Tuldok-lab-workflow-plan; docs branch docs/lab-workflow-reliability-plan preserves the complete six-milestone draft. Exact upstream validate_plan was executed and PASSES (checker SHA256 c7266c0564b9ee02944fc4eb12d66cc20d604fd4ddb7f051ed77d01e9fe4466e, current plan62d33606...). Independent review found one real recovery-consumer gap; corrected with an exact versioned attached publication-result Ref/new negotiated collection+comparison readers while preserving legacy failed Run semantics and explicitly refusing candidate/package promotion initially. Focused re-review is running. Full links/whitespace/docs-only checks will run after final review/JSON files exist. Application-purpose Standards Engine route is unavailable; no Engine certification/product/GPU/scientific test is claimed. No branch, implementation or merge changes outside the new plan directory. Draft plan PR follows after review and validation. |
|
Completed documentation-only plan handoff: draft PR91 #91 at branch docs/lab-workflow-reliability-plan, exact commit b446d9a. Canonical plan: https://github.com/MrScripty/Tuldok/blob/b446d9a19dbd553b80bf9f3af574b8b34ddbd5cb/docs/plans/lab-workflow-reliability/plan.md . Six milestones: M1 exact preflight/bounded diagnostics/same-Run publication recovery; M2 RES-1 health/custody-safe coordinator recovery; M3 honest phase timing/wait reasons; M4 measured verified reuse+bounded batching; M5 thin supported SDK+matching CLI; M6 compatible scientific comparison in existing UI/lineage. Exact C1–C6 contracts/T1–T8 acceptance, security/resources, error/cancel/retry, migration/rollback and missing-evidence dispositions included. Independent GPT-6.1 Sol High review CLEAR after recovery-reader contract correction; reportSHA b6666f6036f798cb62bead680f1ddb4cceb3049aed87358b4478d2f5de8b9d10. Original structural validator executed PASS independently and by author; all23 relative links/anchors, whitespace, eight-file docs-only scope PASS. Full Engine route unavailable and not claimed. Main300f12d and your PR89/90 exact heads remain baseline/open dependencies; no product/GPU/scientific test, implementation, merge or branch deletion by me. Branch/worktree clean and retained. Implementation requires a subsequent instruction; please retain lead ownership of baseline disposition/integration. Parent receives the complete result via final handoff. |
Run coordination repeatedly validated terminal history under the catalog lock, delaying ordinary Run and artifact reads. This keeps the existing coordination owner, selects queued/active candidates before fully validating each selected record, and waits on the existing wake event when idle. Explicit reads, dispatch, startup recovery and artifact integrity checks retain their validation.
This dependent PR preserves the full published branch through de0f1b8: independent CPU/CUDA capacity, startup/GO custody fences, drain coordination, the explicit managed-CUDA deadline, transient output-rename handling, and the scoped scheduling repair. It also includes PR89's reviewed compatibility, browser-observation and registry-durability fixes. No source commits were dropped or rebased.
Retained lab evidence under docs/plans/request-latency-v1 reports same-Run/same-189-byte-artifact observations around 7.7 s before and 0.394 s after, with sampling caveats. These are not a controlled whole-pipeline speedup or a general artifact latency guarantee. A fresh historical C67 CPU restore completed with drained custody and zero optimizer updates: nine immutable outputs matched, while the tenth receipt differed only in timing/RSS. That new Run's artifact reads were 2.1–2.3 s. Paired preflight was 15.104→14.263 s, so most preparation cost remains.
Current integration validation requires independent source review, targeted capacity/managed-process/output/scheduling checks, full exact-head hosted PostgreSQL 17 CI, and post-merge CI. Historical CUDA/scientific and live-deployment receipts are inspected evidence; they are not newly reproduced hardware execution. No real catalog was opened by this review.
Remaining limitation RES-1 is explicit in the retained findings: a catalog advisory-lock timeout can terminate the coordinator, and _start cannot replace its assigned dead thread. That existing fail-closed availability gap is not repaired here. ENV-1's absent legacy runtime is recorded as unavailable rather than passed. Further preparation profiling, diagnostics and model comparison work require their own scope.
Final qualification of exact
76d3c6a2ec46f36bb5dee5f2d40d1ddf4d017114: hosted PostgreSQL 17 lifecycle 39302006431 succeeded with every step successful. Agent contracts/HTTP ran 74 tests and official MCP ran 19, all passed; the full Python suite ran 1535 tests in 1582.490s, OK with 72 explicitly skipped. Required browser/model-development gates succeeded. Captured CPU workflow 39302006445 also passed actual finite CPU training, checkpoint/probe collection, correction/new release, exact package preparation and separate captured CPU inference verification. Independent source review and final archived-evidence review remain distinct. Main post-merge qualification is still required.