diff --git a/docs/plans/lab-workflow-reliability/execution-ledger.md b/docs/plans/lab-workflow-reliability/execution-ledger.md new file mode 100644 index 00000000..af5ef664 --- /dev/null +++ b/docs/plans/lab-workflow-reliability/execution-ledger.md @@ -0,0 +1,43 @@ +# Lab workflow reliability execution ledger + +## 2026-10-19 — Plan preparation + +- Parent requested a standards-compliant implementation plan only, with a + separate documentation branch/worktree, independent review, validation and + draft publication. No product implementation or merge is authorized here. +- Inspected local status/worktrees, repository tree, current main and PR89/90. + Main is `300f12df6f48e028fd9379911703f7c89337952a`; PR89/90 remain open at + `33069a859836953960a06dfd6001cba54aba0aa0` / + `429d0c4ff44dba16bc95313d2a0980169dbc9b9f`. +- Created `docs/lab-workflow-reliability-plan` at main in the separate + `/workspace/Tuldok-lab-workflow-plan` worktree. Existing worktrees and refs + were preserved. Only this plan directory is an allowed write set. +- Coordinated with the integration lead in + [PR90 comment](https://github.com/MrScripty/Tuldok/pull/90#issuecomment-6272255247). + Lead retains PR89/90 and all integration/merge authority. +- Inspected completed D1, current material/preflight/publication/CLI/evidence/ + lineage ownership and PR90's RES-1. Wrote six implementation milestones; + no new design-discovery phase or product worker was started. +- Read adopted Coding Standards at `dcc56f26e884ade260770beceba2501d3746200d`. + The available skill is standards-authoring-specific and explicitly excludes + ordinary application work. No repository AGENTS.md/local skills were found. + Application-purpose Engine invocation is unavailable; exact source structural + validation and manual review are recorded separately, without Engine certification. +- Independent review and actual validation results are recorded in + [reports](reports/validation.md). All product claims A1–A8 remain pending. +- Independent GPT-6.1 Sol High review found one blocking draft gap: legacy + completed-state gates made recovered evidence inaccessible. Revised C2/M1/T2 + with an exact attached publication-result Ref/new readers and explicit legacy + failed/promotion-refusal semantics. Also made scientific per-case criteria and + recovery operation budgets explicit. Focused re-review cleared all findings; + [the exact subject hashes and limits](reports/independent-plan-review.md) remain + distinct from future implementation acceptance. +- A disconnect notice was investigated: actual commands and source reads + succeeded, all draft hashes were intact, and no work was lost. Reported this + to lead/parent in [PR90](https://github.com/MrScripty/Tuldok/pull/90#issuecomment-6272545966). +- Final plan-creation validation passed: unchanged upstream structural checker, + all23 relative links/anchors, new-file whitespace and all8 paths confined to + the new documentation directory. No application/runtime/catalog test ran. + The coherent documentation commit contains the plan, exact contract/test + handoff, issues, ledger and review/validation evidence. The draft PR identifies + its final Git SHA; branch/worktree remain retained for lead review, with no merge. diff --git a/docs/plans/lab-workflow-reliability/issues.md b/docs/plans/lab-workflow-reliability/issues.md new file mode 100644 index 00000000..786fe9d1 --- /dev/null +++ b/docs/plans/lab-workflow-reliability/issues.md @@ -0,0 +1,15 @@ +# Lab workflow reliability issues + +| ID | Severity / evidence | Objective and owner | Disposition / verification / revisit | +| --- | --- | --- | --- | +| RES-1 | High availability/safety boundary; PR90 retained findings describe advisory-lock timeout terminating `_loop`, while `_start` retains a dead assigned thread. | A3; Runs coordinator owner. | Fix in M2 after integration baseline is verified. Inject actual PostgreSQL lock timeout and active custody failure; prove unhealthy/readiness fence, no duplicate coordinator or GO, safe drainage/reconciliation. Never count an exited coordinator as idle performance. | +| PUB-1 | High lost-work cost; lab reports completed training with diagnostic evidence beyond selected shard bounds. Exact private failure is not present here. | A1/A2; evidence profile and Runs publication owners. | M1 classifies one preserved failure before changing profiles. Bounded indexed producer output plus explicit same-Run publication continuation; invalid historical outputs stay invalid. Re-plan if mandatory data cannot fit admitted limits. | +| DEP-1 | Integration dependency; PR89/90 are open at the recorded SHAs, not merged. | All milestones; repository lead. | Verify current merged baseline/CI and remove duplicate tasks before implementation. Plan author changes neither branch. Revisit on any semantic dependency change. | +| PERF-2 | Measurement need; historical artifact timings are single-workload observations, preflight/restore still costly. | A5; performance owner. | M3 exposes phases; M4 measures cold/warm/history-size costs before retaining cache or batch machinery. No invented target/general speedup. | +| SCI-1 | Missing actual compatible baseline/candidate pair and fresh CUDA/scientific evidence. | A7 and any hardware-specific claim; lab/evidence owner. | Deterministic compatible/mismatch UI fixtures may proceed, labeled correctly. Lab-specific acceptance stays unavailable until exact artifacts, case/split/units/metrics and hardware procedure exist. No training rerun is authorized by plan creation. | +| TOOL-1 | Standards tooling limitation; no MCP tools exposed; application-purpose invocation returns unavailable. | Plan validation; documentation author. | Use exact checked-in structural validator and manual policy/contract review, retaining tool outcome. Do not claim full Engine or application compliance certification. Revisit if a supported application connection becomes available. | +| REVIEW-R1 | Blocking draft-contract gap; existing artifact/evidence/candidate readers require completed Run state. | Recovery usability/compatibility; Runs and evidence/selection owners. | Resolved in the reviewed plan: exact versioned attached result Ref for new readers, old failed projections unchanged, candidate/package promotion and new recovered-input bindings unsupported initially; T2/T7 cover all dispositions. Product work remains pending. | +| REVIEW-R2 | Review recommendation: scientific per-case pass/fail was not explicit beside training pass-counts. | Scientific interpretation; criterion/evidence owner. | Resolved in C6/T7 with owned criterion Ref, units/threshold/direction, qualified/diagnostic distinction and pass/fail/unknown; no invented criterion. | +| REVIEW-R3 | Review recommendation: publication recovery needed explicit independent operation bounds. | Resources/recovery; Runs owner. | Resolved in C2/T2 with finite wall/I/O/copy/concurrency/history limits, paused exhaustion and unchanged original compute envelope. Implementation must verify those proposed limits. | + +No unrelated product issue is silently repaired by this documentation branch. diff --git a/docs/plans/lab-workflow-reliability/plan.md b/docs/plans/lab-workflow-reliability/plan.md new file mode 100644 index 00000000..387c4fdc --- /dev/null +++ b/docs/plans/lab-workflow-reliability/plan.md @@ -0,0 +1,300 @@ +# Plan: Reliable lab submissions and scientific feedback + +**Plan status:** `Planned` + +**Current phase:** M1 — exact preflight and recoverable publication + +**Next slice:** After a subsequent implementation instruction and lead verification of PR89/90 integration, implement M1's exact preflight-to-publication path using the contract and acceptance procedures below. + +**Acceptance status:** `pending` + +**Execution ledger:** [execution-ledger.md](execution-ledger.md) + +**Issues:** [issues.md](issues.md) + +## Objective + +Let a lab submit an inspectable, exactly identified experiment, catch known +runtime/input/resource/evidence failures before expensive compute, see where +time is spent, and recover publication of valid completed work without training +again. Reduce measured preparation and collection overhead while preserving +integrity. Provide a small supported Python client and matching CLI, and make +baseline/candidate scientific comparison usable in the existing workspace and +Experiment progression graph. + +This is an implementation plan, not another D1 design exercise. The accepted +[architecture](../end-to-end-model-development/reports/architecture-design.md), +[evidence contracts](../end-to-end-model-development/reports/evidence-contract-design.md) +and existing domain owners remain the starting point. This documentation task +does not start implementation, execute a lab workload, migrate a catalog or +authorize a merge. A subsequent instruction must select this plan for execution. + +## Objective Acceptance + +Every row is proposed product acceptance; none is satisfied by writing this plan. +Exact cases, oracles and execution procedures are in +[contracts and acceptance](reports/contracts-and-acceptance.md#acceptance-procedures). + +| ID | Observable criterion | Kind | Environment | Mode | Status | Evidence | +| --- | --- | --- | --- | --- | --- | --- | +| A1 | A real submission reports complete bounded preflight tied to its exact material; known invalid runtime/input/output/evidence/resource cases refuse before process creation or grant consumption. Dispatch still rechecks mutable facts. | integration | representative | automated | pending | T1 | +| A2 | An authorized recovery publishes and exposes valid retained outputs via an exact attached result Ref under the same Run/reservation, with zero new compute or charge; legacy failed projections and candidate/package refusal remain explicit. | system | required-real | automated | pending | T2 | +| A3 | Lock-timeout and coordinator-failure tests expose unhealthy state, fence admission, reconcile actual process custody and recover one coordinator without duplicate execution; unknown custody stays fenced. | system | required-real | automated | pending | T3 | +| A4 | UI and CLI show preparation, queue/wait, execution, validation and publication spans, open/stale/unknown intervals and wait reasons; overlapping work is represented without adding durations into false elapsed totals. | user-workflow | representative | either | pending | T4 | +| A5 | Paired cold/warm/history-size measurements establish whether verified reuse and bounded collection batching improve their named costs, while byte mutation, stale authority and partial failures retain the same integrity/refusal outcomes. | integration | representative | automated | pending | T5 | +| A6 | A clean consumer uses the supported SDK and matching CLI to generate routine manifests, submit once, monitor, cancel, collect verified artifacts and compare exact results, without implementing a scheduler or gaining owner authority. | system | representative | automated | pending | T6 | +| A7 | A real browser user compares compatible baseline/candidate evidence per case with owned pass/fail/unknown criteria and recorded passes/iterations, sees time/memory/config/logs/diagnostics and incompatibility reasons, and follows immutable lineage with honest external/managed labels. | user-workflow | representative | either | pending | T7 | +| A8 | Supported older clients and retained catalogs preserve identities and access; explicit upgrade/restore, interrupted migration, cold reads and rollback procedures preserve raw evidence and do not replay compute. | contract | required-real | automated | pending | T8 | + +## Scope + +### In Scope + +- Comprehensive exact preflight; bounded indexed diagnostic capture; resumable, + integrity-checked publication of the same completed attempt. +- RES-1 coordinator health and recovery within Runs, plus live phase timing and + wait reasons in existing Run/Project views. +- Identity-bound verified preparation reuse and bounded artifact metadata/read + collection, selected and retained only when measured value justifies them. +- A supported thin Python workflow client and the existing `project_cli.py`; + existing AgentClient/operation negotiation remain the transport authority. +- Scientific evidence compatibility and comparison in existing evidence views, + comparison endpoint and Experiment progression graph. + +### Out Of Scope + +- Repeating D1, reimplementing PR89's managed Torch/live Projects or PR90's + CPU/CUDA capacity, custody fences and idle-history selection repair. +- A new scheduler, parallel orchestration service, installer, runtime registry, + generic metrics platform or custom HTML report system. +- Automatic retraining, grant issuance, budget changes, selecting a scientific + winner, changing a model/physics method, new hardware support or a sandbox claim. +- Guaranteeing arbitrary output success, raising old limits to accommodate one + workload, or treating historical observations as a pipeline speedup. + +## Constraints And Assumptions + +- Baseline main is `300f12df6f48e028fd9379911703f7c89337952a`, checked remotely + on 2026-10-19. PR89 is open at `33069a859836953960a06dfd6001cba54aba0aa0`; + dependent PR90 is open at `429d0c4ff44dba16bc95313d2a0980169dbc9b9f`. + They are proposed dependencies, not merged capabilities. The lead owns their + exact-head CI, serial integration and any resulting baseline correction. +- Pumas owns installed runtime acquisition/updates; Tuldok verifies selected + runtime identity. Runs owns attempts, process custody and publication; Studies + owns selection/budget; Pipeline owns recipes/material preparation. Reuse and + client conveniences reference those authorities and cannot replace them. +- Retain exact training-to-inference hashes, source/input identity, current + transitive access, cancellation/drainage evidence and persistent catalog state. + Existing profile limits remain authoritative. Hash/permission/custody checks + may move only when their proof lifetime is preserved, never just to save time. +- Preflight can reject known impossibilities, not predict arbitrary program + outputs or future free space. Unknown output sizes require an admitted bound + and execution-time enforcement. No project code runs during inspection. +- Lab reports of 70.9 s training in a 275.1 s workflow and oversized diagnostics + motivate the work. Historical artifact observations about 7.7 to 0.394 s are + not controlled whole-workflow speedups; 15.104 to 14.263 s preflight and + 2.1–2.3 s fresh restore reads identify remaining costs. See + [baseline and evidence limits](reports/baseline-and-standards.md). +- No fresh CUDA or scientific verification was established by this plan review. + CPU fixtures qualify lifecycle/transport only. A CUDA claim needs an actual + selected Pumas installation/device and recorded hardware evidence. + +## Binding Decisions + +| Decision | Owner | Evidence / rationale | Supersedes | +| --- | --- | --- | --- | +| Extend existing preflight and material construction; do not add a parallel submission validator or approval authority. | Pipeline + Runs | `pipeline.py` preparation and `runs.py` exact admission already bind material. | none | +| Keep diagnostic indexing and bounded sharding in the selected evidence profile; keep publication continuation in Runs. | Evidence profile owners + Runs | Actual shard-limit failure; profile caps and verifiers must agree before compute. | none | +| Default publication recovery is an explicit owner-authorized same-Run action, never an execution retry. | Runs; Studies reconciles once | Preserves existing one-use grant and finite reservation semantics. | none | +| A recovered legacy failed Run exposes a versioned attached publication-result Ref to new readers; its old completed-state gates are not widened, and candidate/package promotion remains unsupported. | Runs + evidence readers; Experiments selection | Makes recovery usable for collection/comparison without silently changing old Run meaning. | none | +| Repair RES-1 inside the existing coordinator lifecycle; readiness requires reconciliation, not a live-thread assertion. | Runs | PR90's retained RES-1 finding and process custody contract. | none | +| Record owner phase spans and derive views; do not create another Run state machine. | Runs, with Pipeline preparation spans | Spans explain concurrent work without altering execution authority. | none | +| Cache only disposable verified preparation derived from immutable owned bytes; a mutable installed-runtime observation is not durable verification. | Pipeline / runtime verification owners | Explicit identity and invalidation in C4. | none | +| Extend AgentClient with one thin `tuldok_client.py` facade and the existing CLI. Keep `tuldok_extension_sdk` producer/conformance responsibility separate. | Client/API owner | Existing negotiation, private config and verified download paths. | none | +| Reuse verified evidence comparison and Experiment progression identities; generic UI displays declared scientific meaning. | Evidence owners + Experiments lineage | Existing comparison selectors and immutable Trial lineage. | none | +| One lead integrates material changes serially; new plan preparation owns only its directory. | Repository lead | Avoids conflicts with PR89/90 and shared API/UI files. | none | + +Full proposed contracts C1–C6, consumer dispositions, failure semantics and +security limits are in [contracts and acceptance](reports/contracts-and-acceptance.md). +Those are planning decisions; implementation must update the named canonical +contract documents and registered operations rather than making this report a +second runtime authority. + +## Design Observations And Remaining Evidence + +| Assumption / claim | Decision affected | Owner | Smallest adequate procedure and stopping condition | Environment / state | +| --- | --- | --- | --- | --- | +| PR89/90 integrate with their reported contracts and RES-1 remains open. | Implementation baseline and write sets | Lead | Inspect merge SHAs, current contracts and exact-head CI before M1; stop when actual baseline/dependency disposition is recorded. | Remote Git/CI; pending integration | +| Diagnostic failure is a bound/format issue, not a training error. | Profile planning and recovery eligibility | M1 owner with lab | Inspect one preserved failure and selected output/profile declarations read-only; classify it without replaying training. Stop at the first precise failed invariant. | Retained lab receipt; missing here | +| Completed outputs can survive a publication fault durably and within existing resource envelopes. | M1 retention/continuation | Runs owner | Inject one post-drain pre-commit fault on a disposable real custodian attempt, reopen and validate retained bytes. If retention requires unbounded disk or weaker custody, re-plan. | Isolated PostgreSQL + Linux process host | +| Repeated hashing/preparation and per-artifact requests dominate remaining overhead. | M4 cache/batch selection | Performance owner | Capture phase and call-count profiles, then paired cold/warm/large-history samples; stop after the proposed mechanism's contribution is distinguished. | Representative local workflow; pending | +| Comparison operands declare compatible scientific meaning. | M6 supported comparison domain | Evidence owner + lab | Read one real baseline/candidate pair's task/data/split/case/units/metric contracts; show precise mismatch rather than infer equivalence. Stop with an explicit supported/missing disposition. | Lab artifacts; unavailable here | + +These observations do not reopen D1 or delay independent deterministic work. +Missing lab/hardware inputs block only the dependent claim; they do not permit +fixtures to stand in for scientific acceptance. + +## Simplicity And Ownership Review + +**Applicability:** `applicable` + +- Independent concepts and dimensions: submission material (Pipeline), process attempt and publication (Runs), installed environment (Pumas), finite experiment policy (Studies), and scientific interpretation (evidence owner/lab) change for different reasons. UI/client compose their typed projections; they own no execution policy. +- State, identity, value, time, policy, and mechanism: exact material hashes reference source/runtime/input/profile authorities; health and spans are observations; publication has a separately owned continuation record. No single manifest becomes canonical for runtime installation, permission, budget and science. New operation versions own only changed wire promises; evidence versions own interpretation. Old supported versions retain exact projections. Material/profile/host/input changes invalidate preparation; credentials and mutable installation facts are rechecked at their own boundaries. +- Caller and composition-root knowledge: callers choose graph/Trial, exact material, grant and request key; they receive bounded results and typed failures. They do not know file-copy ordering, recovery locks, shard offsets, internal futures or cache layout. Dataset/Runs lifecycle remains the sole coordinator composition root. +- Representative change paths and forced owners: changing an evidence shard policy touches its producer/verifier/readers and public version, not Pumas or Study policy; changing installed Torch bytes touches Pumas selection and verification, not metric meaning; changing a metric touches its evidence contract and comparison, not custody. A queue-lock failure touches Runs health/recovery and projections, not another scheduler. Broader required propagation triggers re-plan. +- Stable Interfaces versus hidden knowledge: immutable refs, exact negotiated operations, existing custody receipts, metric definitions and bounded artifact descriptors are the stable boundaries. SDK and UI consume these; raw staging directories, thread handles, file mtimes and process-local clocks stay private. +- Independent evolution, testing, failure, and replacement: publication can fail while successful compute remains recorded; timing unavailability cannot grant readiness; a cache miss uses the authoritative preparation path; an incompatible comparison refuses without affecting Runs. Each milestone can be disabled without deleting retained authority. Real transport/process/store tests supplement focused contracts. +- Necessary complexity and containment: Runs contains at-most-once GO, custody reconciliation and publication atomicity; profile owners contain bounded evidence; Pipeline contains proof reuse; clients contain routine manifest assembly. Their composition retains one admission path and one coordinator, with no cross-cutting workflow engine. +- Deletion and cumulative machinery result: deleting the disposable cache loses only measured speed, so omit/remove it if its benefit is unproven. Deleting the batch operation restores bounded individual reads. Deleting the SDK would spread manifest/collection obligations across callers, justifying one thin facade. Publication continuation and health are necessary to prevent lost work/unsafe restart, so remain in Runs. Reject a new registry, universal validation framework, scheduler or extra model store; reuse existing descriptors, verifiers and lineage. + +## Milestones + +Each milestone is a useful reviewed vertical outcome. No branch or commit count +is prescribed. All write sets below also allow this plan directory; edits to +shared API/UI/schema files are integrated by one named owner. Proposed new paths +are explicit and carry no claim that they exist now. + +### M1: Exact preflight and recoverable bounded publication + +**Goal:** Known failures refuse before compute; valid completed work survives publication interruption without retraining. + +**Allowed write set:** `pipeline.py`, `recipe_contract.py`, `runs.py`, `extension_evidence.py`, `model_evidence.py`, `field_evidence.py`, `tuldok_extension_sdk/evidence.py`, `agent_api.py`, `app.py`, `experiments.py`, `experiment_lineage.py`, `metadata.py`, `backup_metadata.py`, `static/workspace-external.js`, `static/workspace-real.js`, `static/experiment-lineage.js`, `static/workspaces.html`, `docs/contracts/{external-recipes,model-development,agent-operations,workspace-runs,workspace-storage,experiment-lineage}.md`; new `docs/contracts/run-preflight-publication.md`; `tests/test_external_{runs,artifacts}.py`, `tests/test_study_admission.py`, `tests/test_experiment_lineage.py`, `tests/test_extension_sdk_evidence.py`, `tests/test_model_evidence.py`, `tests/test_candidate_{selection,packages}.py`, `tests/test_metadata_migration.py`, and new `tests/test_run_preflight_publication.py`. + +**Tasks:** Extend C1's canonical checks and structured diagnostics; pre-admit bounded C2 diagnostic indexing; stage a publication receipt/continuation with finite wall/I/O/history/copy budgets; expose recovery status, exact result Ref and explicit owner action in the existing Run panel and lineage links; keep original failures and settle the existing reservation exactly once. Implement new negotiated result readers; retain old failed-reader and candidate/package promotion refusals. Use versioned evidence when indexed sharding changes a closed profile; do not change old verifiers in place. + +**Acceptance gate:** T1/T2/T8, independent integrity/authority review and actual post-drain fault/reopen proof. Demonstrate oversized-declaration refusal and one valid indexed diagnostic round trip. Invalid/over-limit original evidence remains failed; recovery does not repair its scientific content. Re-plan if required diagnostic information cannot fit a bounded declared profile or durable recovery changes custody/accounting authority. + +**Status:** `Planned` + +### M2: Coordinator health and safe RES-1 recovery + +**Goal:** A coordinator exception is visible and recoverable only after safe process reconciliation. + +**Allowed write set:** `runs.py`, `recipe_process.py`, `run_capacity.py` (PR90 dependency), `agent_api.py`, `app.py`, `static/workspace-real.js`, `docs/contracts/{workspace-runs,external-recipes,agent-operations}.md`; new `tests/test_run_coordinator_recovery.py`; existing `tests/test_run_{scheduling_reads,concurrency}.py` and `tests/test_recipe_process.py` if present on integrated baseline (otherwise record the exact replacement before edits). + +**Tasks:** Implement C3 health/generation and a serialized explicit recovery action; stop admission on failure, retain/cancel/drain owned futures, reconcile durable attempts and custody, and replace only a proven dead joined coordinator. Separate lifecycle shutdown from recoverable fault state. Do not reset stop flags or fences merely to revive a thread. + +**Acceptance gate:** T3 and affected PR90 capacity/custody tests on the real PostgreSQL/process boundary. Prove no two coordinators, no duplicate GO, preserved unknown fence and bounded shutdown. Re-plan if recovery needs external custody ownership or changes capacity policy. + +**Status:** `Planned` + +### M3: Live phase timing and wait visibility + +**Goal:** Users can distinguish compute from preparation, queue, validation and publication delays. + +**Allowed write set:** `pipeline.py`, `runs.py`, `recipe_process.py`, `agent_api.py`, `app.py`, `project_cli.py`, `metadata.py`, `backup_metadata.py`, `static/workspace-{real,external,service}.js`, `static/workspaces.html`, `static/workspaces.css`, `docs/contracts/{workspace-runs,project-cli,agent-operations,workspace-storage}.md`; new `tests/test_run_phase_timings.py`, `tests/test_run_phase_controller.cjs`, `tests/browser_run_phase_timings.cjs`. + +**Tasks:** Implement C3 spans from owners, including preparation before Run creation; bounded live projections and safe polling cleanup; display wait reason, last update, incomplete spans and publication recovery. Retain coarse old timing as unknown-granularity rather than fabricate history. + +**Acceptance gate:** T4/T8: clock-controlled overlapping-span assertions plus real browser/backend lifecycle under queued capacity, lock wait, cancel and publication fault. Re-plan if span retention requires an independent telemetry service or materially affects request latency. + +**Status:** `Planned` + +### M4: Measured preparation reuse and batched collection + +**Goal:** Remove demonstrated duplicate work while preserving each proof and artifact outcome. + +**Allowed write set:** `pipeline.py`, `runtime_snapshots.py`, `torch_runtimes.py` (PR89 dependency), `runs.py`, `agent_api.py`, `app.py`, `docs/contracts/{captured-cpu-runtime,managed-torch-runtime,external-recipes,agent-operations}.md`; new `tests/test_preparation_reuse.py`, `tests/test_artifact_collection.py`, `tests/benchmark_lab_workflow.py`. + +**Tasks:** Run T5 baseline first. Implement C4 only for repeated measured work with explicit keys/invalidation; retain provenance and current safety checks. Add C5 bounded collection if request counts contribute materially. Keep sequential bounded service by default; batching is not permission to parallelize compute or bypass per-item validation. Remove an unhelpful cache rather than retain machinery for a hypothetical gain. + +**Acceptance gate:** T5 and current artifact/revocation/material-integrity regressions; publish cold/warm/history-size distributions, CPU/RSS/I/O and integrity equality. No invented duration or percentage target. A mechanism is retained only for a measured useful contribution accepted by the performance owner with no integrity regression. Re-plan if a trusted immutable store cannot support safe reuse or measurements identify a different owner. + +**Status:** `Planned` + +### M5: Supported SDK and matching CLI + +**Goal:** Ordinary users complete routine submit/monitor/cancel/collect/compare workflows with small inspectable calls. + +**Allowed write set:** new `tuldok_client.py`, `agent_api.py`, `project_cli.py`, `docs/contracts/{project-cli,agent-operations}.md`, new `docs/contracts/workflow-client.md`, `README.md` client entrypoint only; new `tests/test_workflow_client.py`, `tests/test_workflow_client_http.py`; existing `tests/test_agent_project_cli.py`. + +**Tasks:** Implement C6 through negotiated operations; generate ordinary submission/collection manifests without custom script scaffolding; export exact JSON before submission; preserve explicit owner approval and explicit reconciliation after unknown mutations. Keep existing CLI and low-level AgentClient compatible. Add capabilities only alongside working server paths. + +**Acceptance gate:** T6 plus old-client contract tests; actual clean Python consumer and CLI use one admitted neutral attempt, cancellation, denied calls, interrupted collection and lost-reply reconciliation. No app/DB/Torch startup on import and no new install dependency for routine CPU client use. Re-plan if client logic duplicates server policy or requires a second credential/transport system. + +**Status:** `Planned` + +### M6: Scientific baseline/candidate feedback in existing UI and lineage + +**Goal:** Replace custom HTML comparison for supported scientific evidence with reproducible existing-workspace views. + +**Allowed write set:** `field_evidence.py`, `extension_evidence.py`, `model_evidence.py`, `experiment_lineage.py`, `experiments.py`, `runs.py`, `agent_api.py`, `app.py`, `static/workspace-real.js`, `static/workspace-external.js`, `static/experiment-lineage.{js,css}`, `static/workspaces.html`, `static/workspaces.css`, `docs/contracts/{experiment-lineage,model-development,agent-operations}.md`, new `docs/contracts/scientific-comparison.md`; `tests/test_experiment_lineage.py`, `tests/test_field_evidence.py`, `tests/test_experiment_lineage_controller.cjs`, new `tests/test_scientific_comparison.py`, `tests/browser_scientific_comparison.cjs`. + +**Tasks:** Implement C6's scientific compatibility gate and exact operand selection; per-case values/regressions and owned pass/fail/unknown criteria, iteration/pass views with timings/memory/config/log links; link baseline/candidate through existing lineage. Keep external activity clearly separate from managed Run custody. Use a tabular accessible equivalent for every chart and explicit missing-data states. + +**Acceptance gate:** T7/T8 with a real browser and a small independently calculated compatible pair, isolated mismatches, stale UI delivery, revocation and retained-root removal. Run the lab's recorded pair only when provided; synthetic arithmetic proves UI/contract behavior, not scientific improvement. Re-plan if generic display requires interpreting physics, mixing evaluation cohorts or inventing absent measurements. + +**Status:** `Planned` + +## Compatibility, Migration And Rollback + +Preserve all installed operation identities/projections. New closed outputs use +new negotiated versions or distinct operations, selected once from capabilities; +auth, corrupt responses and timeouts never trigger a second mutation dispatch. +Older retained evidence stays on its original verifier and remains inspectable. +No historical span, cache verification, scientific unit or lineage is synthesized. + +Before M1/M3 add persistence, inventory actual supported schema versions and +backup/restore consumers on the integrated baseline. Extend the existing explicit +stopped-writer migration, atomic ledger and storage contract; never migrate at +inspection/startup by surprise. Validate upgrade, interrupted acknowledgement and +stopped restore with raw bytes preserved. The lead records backup and target +schema before activation. An older binary must refuse an unsupported new schema; +rollback uses the recorded stopped backup plus retained new evidence, or a proved +forward repair. Never reopen a new catalog with an incompatible binary or erase +authority to downgrade. Cache data is independently disposable; remove only +identified unreferenced cache entries after no reader owns them. + +Roll out milestones separately through exact candidate review/tests/CI. Disable +new convenience operations on failure while retaining underlying inspection, +custody fences and evidence. Rollback must not restart compute, resend GO, reuse +a consumed grant or pretend a publication fault undid training. Future contract +docs own the operational details; this plan owns their required acceptance. + +## Blockers + +- No blocker to preparing/reviewing this plan. Implementation requires a later + user instruction and lead disposition of open PR89/90; no merge is inferred. +- Actual preserved lab diagnostic failure and compatible scientific pair are + absent in this executor. M1 classification and lab-specific M6 acceptance must + record missing inputs until supplied. Fresh CUDA verification remains absent; + no CUDA claim may be accepted from CPU substitutes. + +## Re-Plan Triggers + +- Integrated PR89/90 changes an owner/contract or resolves a proposed task. +- Preflight/recovery needs project execution, a new scheduler, weaker byte or + authority checks, unbounded storage or a second accounting lifecycle. +- Required scientific compatibility lacks an evidence owner/units/case identity; + a lower-fidelity fixture is being used for a hardware/science claim. +- Measurements do not support reuse/batching, or cross-owner change propagation + materially exceeds the composed design. Update scope and downstream gates + before broadening implementation; routine failures inside a valid slice are + repaired without restarting D1. + +## Review, Integration And Repository Isolation + +One implementation owner is assigned per admitted milestone; API/schema/shared +UI changes stay serial under the lead. Independent reviewers read the complete +changed invariant and its consumers, not merely one happy-path test. Any future +delegation needs an explicit non-overlapping write set and report; this plan +creates no duplicate implementation workers. + +Plan-author branch `docs/lab-workflow-reliability-plan` and worktree +`/workspace/Tuldok-lab-workflow-plan` isolate documentation from PR89/90. The +documentation author owns only `docs/plans/lab-workflow-reliability/`; target is +main; the lead owns integration. The shared draft branch/worktree remains retained +for review and implementation handoff. This task authorizes no merge, deletion, +history rewrite or worktree pruning. Later cleanup requires clean-state, +reachability and safely merged dependent-branch evidence under Commit. + +Plan review/validation evidence: [baseline and standards](reports/baseline-and-standards.md), +[independent review](reports/independent-plan-review.md) and +[validation](reports/validation.md). Validation of these documents is not A1–A8 +product evidence. No dependency or application test is weakened for this plan. + +## Final Acceptance + +- Acceptance status: `pending` +- Deferred follow-ups: CUDA/scientific claims require their actual inputs; no promise of new hardware or physical validity is admitted. +- Final status: `Planned` diff --git a/docs/plans/lab-workflow-reliability/reports/baseline-and-standards.md b/docs/plans/lab-workflow-reliability/reports/baseline-and-standards.md new file mode 100644 index 00000000..ac547c66 --- /dev/null +++ b/docs/plans/lab-workflow-reliability/reports/baseline-and-standards.md @@ -0,0 +1,91 @@ +# Baseline, sources and standards application + +## Source baseline + +Observed 2026-10-19 through GitHub PR reads and anonymous read-only Git refs: + +| Subject | Exact revision / state | Meaning | +| --- | --- | --- | +| main | `300f12df6f48e028fd9379911703f7c89337952a` | Base of this docs-only worktree; merge of completed D1 CLI-baseline documentation. | +| [PR89](https://github.com/MrScripty/Tuldok/pull/89) | `33069a859836953960a06dfd6001cba54aba0aa0`, open/unmerged | Proposed managed Torch registration, live Project activity and compatibility repairs. No fresh CUDA/science result is claimed by integration review. | +| [PR90](https://github.com/MrScripty/Tuldok/pull/90) | `429d0c4ff44dba16bc95313d2a0980169dbc9b9f`, open/unmerged, based on PR89 | Capacity/custody/deadline/idle-history work; retains explicit RES-1. | +| Adopted Coding Standards | `dcc56f26e884ade260770beceba2501d3746200d` | Local Core/Router, canonical policies, plan template and structural validator. | + +Inspected repo Git trees and local workspace instructions: no AGENTS.md or +repository-local `.agents/skills` were present in main/PR89/PR90. Workspace +`.agents`/`.codex` expose no task instructions. The standards repository's +`standards-engine` skill is specifically for authoring the standards library, +and says ordinary application agents use application tools; it is not applied +as an authoring workflow to this application documentation task. + +Read current D1 architecture/evidence/handoff and existing contracts: +[Project CLI](../../../contracts/project-cli.md), +[external recipes](../../../contracts/external-recipes.md), +[model development](../../../contracts/model-development.md), +[Run lifecycle](../../../contracts/workspace-runs.md), +[lineage](../../../contracts/experiment-lineage.md), +[agent operations](../../../contracts/agent-operations.md) and +[storage](../../../contracts/workspace-storage.md). +Inspected `pipeline.py` material/preflight, `runs.py` admission/publication and +artifact readers, `extension_evidence.py`/`model_evidence.py` profile limits, +`project_cli.py`/AgentClient and existing lineage/comparison UI. Read the exact +PR90 source and request-latency findings without checking out or changing its +branch. Existing owners are sufficient; no new scheduler or runtime installer +is justified. + +The following observations are not new benchmark/hardware acceptance: + +- Lab feedback: 70.9 seconds training within 275.1 seconds workflow; completed + training could not publish diagnostics exceeding evidence shard limits; custom + HTML was needed for scientific comparison. Raw private receipt inputs are + absent here, so classification remains an explicit M1 observation. +- PR90 retains historical same-Run 189-byte artifact reads around 7.7→0.394 seconds, + paired preflight 15.104→14.263 seconds, and new restore reads 2.1–2.3 seconds. + Different workloads, sampling caveats and the unhealthy-coordinator observation + prevent a controlled whole-workflow speedup claim. See + [the exact retained findings](https://github.com/MrScripty/Tuldok/blob/429d0c4ff44dba16bc95313d2a0980169dbc9b9f/docs/plans/request-latency-v1/issues.md). +- Fresh CUDA/scientific training was not reproduced by this plan review. Existing + actual lab receipts retain their original source/environment scope. + +## Standards route and design application + +The observable task is implementation planning plus durable docs/commit/review, +not standards maintenance or product execution. Future milestones affect a +Python SDK/CLI, browser UI, persisted/public contracts, lifecycle, resources and +scientific interpretation. There is no language-specific, FFI, framework, new +third-party dependency or package-publication change in this documentation task. +No concurrent mutable-plan proposals exist; the independent reader and one +serial writer do not select concurrent-plan-integration machinery. + +Selected canonical guidance and concrete application: + +| Guidance | Applied decision | +| --- | --- | +| Core, Planning, Implementation, Development Proportionality | Preserve requested outcomes/owners; six useful vertical milestones, exact write sets, one next slice; classify missing inputs without repeating D1. | +| Verification + Independent Oracles + GUI/Platform evidence | A1–A8 separate evidence kind/environment/mode; T1–T8 assert exact failures and no-effect invariants; real process/store/browser evidence cannot be replaced by fixtures. | +| Documentation + Commit | One repository-native plan directory, current decision owner, separate ledger/issues/reports; isolated docs worktree; no unrelated branch changes or merge. | +| Architecture + Code Design + Immutable Authority Closure | Complete eight-probe composition review; one authority per material/custody/runtime/budget/science concern; refs and disposable caches do not acquire authority. | +| Contracts + Evolution + Protocols/Schemas + IPC/Persistence | Preserve installed closed contract identities, explicitly version changed representations, update real consumers, explicit stopped migration/reopen/rollback. | +| Concurrency + Resilience | At-most-once GO, serialized health recovery, owned futures/drain, unknown outcome reconciliation; no automatic retraining or thread restart that clears custody uncertainty. | +| Security + Cross-platform + delegated-authority detail | Current scopes and transitive disclosure; private no-follow bounded files, no credential/path leakage; trusted-local process scope and no stronger sandbox promise. | +| Diagnostics + Performance | Bounded indexed detail and accurate overlapping spans; measured cold/warm/history-size costs; no guessed budget or speed claim. | +| Library / Launcher / Frontend profiles + Accessibility | Thin client and existing CLI preserve outcomes/ownership; real browser lifecycle, keyboard/table/narrow-screen paths and stale-response cleanup. | + +Requires closures follow the local Router and canonical policy metadata. Proposed +product implementation must re-route if baseline, deployment, dependencies or +publication facts change. No external source/library was adopted here. + +## Tooling availability and validation boundary + +No Standards Engine MCP tool is exposed in this session. The checked-in public +helper can list application operations and schema, but actual application-purpose +`route` invocation returns exit2: `Application invocation is unavailable; use the +published interface contract.` No authoring-purpose retry, direct standards +mutation, application certification or guessed successful route was substituted. +Manual routing above records current observed facts and policy application. + +Use the exact local `validate_plan` function with its original constants/helpers +for structure, then independent prose/contract review and link/scope checks. +[Validation](validation.md) records the exact source hash, command and limits. +This is standards-guided planning plus structural validation, not a full Engine +checkpoint, semantic completeness certificate or product acceptance result. diff --git a/docs/plans/lab-workflow-reliability/reports/contracts-and-acceptance.md b/docs/plans/lab-workflow-reliability/reports/contracts-and-acceptance.md new file mode 100644 index 00000000..6325bd49 --- /dev/null +++ b/docs/plans/lab-workflow-reliability/reports/contracts-and-acceptance.md @@ -0,0 +1,469 @@ +# Proposed contracts and acceptance procedures + +This report specifies the implementation plan's intended behavior. It does not +register operations, change supported schemas or certify existing workloads. +M1–M6 must move the implemented contract into its existing canonical owner and +contract document. Names of new operation fields below are semantic requirements, +not a claim that an endpoint already exists. + +## C1 — Exact submission preflight + +Pipeline's material builder remains the single source of prepared execution +meaning; Runs revalidates admission/dispatch. Extend existing external preflight +and Study material preparation with one bounded structured report. The report +references the exact catalog, Project snapshot, recipe/graph revision and +execution identity, Model/Trial where applicable, source/helper/host closure, +runtime profile/device/inventory, input content hashes and sizes, parameters, +output roles/names/limits, evidence profile/version and publication requirements. +Presentation-only graph layout/name stays outside execution identity as today. + +Each check has an owner, stable code, phase, bounded explanation and evidence +reference. Its state is checked, blocked, unsupported, invalid or deferred to a +named later boundary. The aggregate is ready only when all required pre-compute +checks pass. A deferred dynamic check is explicit and never appears as verified. +Report size and diagnostic count are capped by the selected operation; overflow +returns a bounded summary and paginated/indexed detail, never unbounded errors. + +Required check families: + +| Family | Before expensive work | Still checked later | +| --- | --- | --- | +| Source/input | Exact selected refs, content/size, closure, containment and current access; frozen splits/evaluation identities when declared. | Mutable root changes before capture; retained bytes at dispatch/read; current transitive access. | +| Runtime | Explicit Pumas-selected installed profile/device/inventory or captured CPU profile; registration/host compatibility; availability and supported limits. No implicit CPU fallback from CUDA. | Installed-byte verification and actual device readiness at existing READY/GO boundaries; runtime mutation remains refused. | +| Resource | Existing capacity declarations, input/capture/output/evidence/publication copies plus staging/reserve, deadline and cancellation support. Distinguish conservative envelope from measured usage. | Current capacity, disk and grant validity at admission/publication; preflight is not a reservation or quota guarantee. | +| Output/evidence | Required output names/roles, selected decoder/version, bounded shard/index plan, fixed-shape minimum sizes, aggregate byte/file bounds and required notice/provenance fields. | Actual output shape, count, semantics and hashes; stream/output enforcement while running; no prediction of arbitrary code behavior. | +| Publication | Supported durable staging, manifest/receipt/index bounds, completion/drain requirement and required retention/access rules. | Atomic fsync/commit/reopen proof, integrity verification and current authority. | +| Authority | Report required scopes, exact owner grant/material and Study continuation/budget requirements. | Authenticate and consume authority only in the existing atomic admission transaction; preflight cannot issue or reserve approval. | + +The report's identity includes the exact prepared material and check-contract +version. It is not a reusable grant. Any execution-affecting source, input, +parameter, runtime, host/profile/limit or policy change invalidates its proof. +Missing runtime/device or required storage is unavailable; incompatible evidence +is unsupported; malformed declarations are invalid; known identity drift uses the +existing stale/conflict outcome. Preserve actual operation error mappings. +Failed preflight consumes no execution grant/attempt reservation and creates no +recipe process. Its own bounded report may be retained for inspection. + +## C2 — Bounded diagnostics and publication continuation + +Profile owners define indexed diagnostics with exact role, schema/media type, +shard byte/count/aggregate limits, deterministic order, case/checkpoint/phase +references and each shard's size/hash. Reuse the existing artifact inventory as +the byte authority. Mandatory scientific evidence cannot be sampled or silently +truncated. Optional log/diagnostic detail may use a declared bounded capture +policy, with explicit omitted count/range and reason; an index cannot claim full +coverage when detail is absent. The shared SDK writer and host verifier consume +the same profile contract. Old closed v1 evidence remains unchanged; a new +version is required when an index/shard shape changes its meaning. + +Use current profile/output limits, including the current 8 MiB evidence shard +ceiling where selected, instead of widening them. Derive the permitted shard +population from the selected profile's existing file/count and total byte caps. +Reject a declaration whose required evidence cannot fit. Runtime outputs that +exceed their declared limit fail under the current bounded process/output owner. +The producer may write an already admitted bounded index; publication cannot +invent new scientific shards or silently remove invalid/large outputs. + +Runs owns a durable publication continuation attached to the original Run, +material/request identity, exact successful execution/custody receipt and +candidate output inventory. It records verification, staging, publishing, +published or blocked/failure outcome, recoverability reason, original fault and +publication attempts. These are publication states, not a new compute scheduler. +Original Run events and execution outcome remain immutable. A legacy failed +Run is not rewritten as if its first publication succeeded; a verified recovery +is an explicit attached result with a separate published/recovered label. +The new attached publication-result Ref binds schema version, catalog, original +Run, material, output inventory, custody receipt and result-content hash. New +negotiated artifact/evidence/compare readers accept that exact Ref and reuse the +supported verifier; they do not bypass a legacy completed-state gate by silently +substituting a different Run state. The Run panel, SDK collection/comparison and +lineage outcome links expose the same Ref. Old Run/artifact/evidence projections +retain their failed/unavailable outcome. This initial scope explicitly refuses +candidate selection/package promotion from a recovered legacy failed Run. It +also adds no new recovered-result input binding; those consumers require a later +explicit contract admission. Existing completed Run candidate/inference paths +and their training-to-inference hashes remain unchanged. + +Eligibility requires proved successful compute, valid terminal drainage, exact +retained inputs/material and stable valid outputs, and a recoverable publication +fault (for example disk exhaustion, interrupted copy or lost commit reply). +Malformed/oversize scientific output, unproved custody, cancelled work, changed +bytes or incompatible profiles are blocked. An old invalid diagnostic file is +not made acceptable by this continuation. If its raw output cannot satisfy the +original contract, report that limit; a new workload/profile requires its normal +reviewed execution path, outside publication recovery. + +Stage privately, enforce finite retention and include staging/publication copies +in admitted storage/accounting envelopes. Start with a named finite 60-second +per-recovery wall budget, one active recovery per catalog and per Run, and at +most one temporary publication copy within the existing copy reservation. Derive +the I/O budget from the exact closed inventories: one source-verification pass, +one copy and one destination-verification pass, with bounded receipt/index I/O; +no unbounded rescans. Keep at most 20 detailed continuation attempts, plus the +immutable original fault and final result proof; existing mandatory audit +retention is unchanged. Budget or history-cap exhaustion returns a paused, +limit-exhausted continuation, not success or an automatic restart. Raising a +finite operation bound requires explicit owner policy; it cannot enlarge the +original material/settled resource envelope. On fault, retain the authoritative +inventory/receipt and recoverable bytes within that envelope; expose the retention +deadline and bytes. No automatic deletion of referenced authoritative outputs. +An operator-directed abandonment records loss explicitly. Disposable incomplete +copies may be removed only after ownership/reference checks. If durable storage +cannot be established, report unavailable rather than promise recovery. + +An explicit owner-authorized publication-recovery operation uses a stable +actor/operation/Run/request-key/body identity and a single per-Run serialized +continuation. It checks current transitive input disclosure and retained proofs. +It creates no recipe process, sends no GO and consumes no second execution grant, +Study slot or reservation. Studies reconciles the original reservation exactly +once using the actual retained accounting rule; recovery cannot refund compute +or silently hide publication storage charges. Lost replies reconcile the same +continuation before copying again. Concurrent identical requests converge; +conflicting bodies refuse. Revoke/expiry after preparation still blocks new +recovery/disclosure, without changing proved past execution. + +Cancel has explicit phase semantics: queued cancellation prevents dispatch; +running cancellation requests custody drain; publication cancellation stops +copies at a safe boundary and leaves a labelled paused/blocked continuation. +If publication committed first, cancellation returns that actual committed +outcome, never pretends to erase it. Shutdown joins owned work and retains an +interrupted continuation. Retry of compute is a separate fresh approved attempt; +no client or startup path infers it from a publication failure. + +## C3 — Coordinator health, recovery and phase timing + +RES-1 belongs to Runs. Expose a bounded inspect-authorized coordinator health +projection: generation, healthy/idle/blocked/recovering/stopped/faulted state, +last transition/update, bounded fault category and readiness/admission fence. +Sensitive exception context remains owner-only. A thread-alive flag or old +heartbeat is not proof of readiness. Normal idle event waiting is healthy and +does not require polling/scanning terminal history. + +Select explicit owner-triggered recovery as the first supported recovery mode. +On an advisory-lock timeout or unexpected coordinator exception, transition to +faulted, fence new admission/dispatch, retain the original fault and ownership +of active futures/custodians, and use the existing cancellation/drain protocol. +Keep final application shutdown distinct from recoverable coordinator stop. +Do not catch-and-continue through unknown storage state or repeatedly restart. + +Serialize recovery under the existing runtime/catalog ownership and thread +lifecycle lock. Establish that the prior coordinator and futures are joined or +otherwise authoritatively ceased, reacquire required catalog/runtime ownership, +then reconcile durable admitted/queued/running/cancelling attempts and custody +receipts before creating one new coordinator generation. A same-boot missing +receipt or uncertain GO remains unknown and fenced according to the current +custody contract. Dead PID/empty process scans never clear it. Preserve request +receipts and queue order; do not admit another attempt, resend GO or automatically +replay a retained external Run. Failed reconciliation stays faulted with an +actionable reason. All joins, DB waits and recovery operations are bounded and +cancellable; HTTP readers do not hold catalog locks while waiting on them. + +Timings are owner-authored spans for preparation, queue, execution, validation +and publication, with operation/material or Run identity, parent/attempt, +phase, source, start/end instants, measured duration, current/terminal status, +bounded wait reason and last update. Preparation can precede Run creation: its +receipt binds exact material and is referenced when attached to a Run. No new +experiment lifecycle is introduced. A cache hit records actual lookup/validation +time and reuse provenance, not the time of the original preparation. + +Use monotonic clocks for same-process elapsed durations and deadlines; use UTC +instants for cross-process correlation. Keep clock domain/boot/source and +uncertainty; never subtract persisted monotonic readings from another process. +Represent child overlap explicitly. Parent elapsed is measured directly; +subphase work durations are not summed into workflow elapsed. When clocks cannot +support an interval union or cross-process delta, show unknown rather than infer +an exclusive partition. Report wait reason such as capacity, custody fence, +catalog lock, runtime verification, output validation, storage or publication +reconciliation. Unknown reason is allowed and visible. Producer-reported +training/pass timing is labelled separately from host-managed execution. + +Historical missing phases stay unknown, open spans remain open/stale after a +fault, and cancellation/interruption terminates only phases whose owner can +prove cessation. Timing channel failure cannot establish or erase Run success, +grant readiness or drive recovery. Bound event/detail retention and polling; +stop timers/requests on navigation, selection change and client cancellation. + +## C4 — Verified runtime/material preparation reuse + +First measure repeated work. Prefer existing content-addressed retained captures +over a second cache. Any added cache is private disposable derived state owned +by Pipeline/runtime verification, with an entry schema and verifier version, +catalog/trust scope, exact source/input/Model/recipe/host/runtime/profile identities, +parameters and effective limits that can affect the prepared result. Reference +immutable owned source bytes and their provenance; do not key solely by path, +mtime, recipe name or graph ID. Do not store live permission, grant validity, +free-space/capacity, process readiness or scientific acceptance as cached facts. + +Reuse only an intact verified object under its proof lifetime. Build/publish an +entry atomically; partial/corrupt/missing entries produce a miss through the +canonical preparation path or an unavailable source outcome, not weaker checks. +Changing any execution-affecting key component, verifier/format or trust scope +invalidates reuse. Current authorization and mutable facts are checked at their +own boundaries on every operation. Concurrent identical preparation may share +only a completed proven entry; cancel one caller without cancelling another +owner's work or leaking temporary state. + +Pumas still owns installed runtimes. For mutable installed Torch assets, do not +persist a last-known-good hashing result as proof of current bytes. Keep current +prelaunch/READY/GO checks and mutation refusal. Invocation-local reuse can avoid +duplicate work on the same held, stable verified representation if its boundary +and lifetime are proved; a cross-call speed claim requires genuinely immutable +owned storage. Record hit/miss/invalidated, original material/inventory, verification +source/version and bytes/time actually avoided. Entries have finite count/byte +budgets, no credentials, private permissions and reference-safe eviction. If no +safe measured benefit remains, do not add the cache. + +## C5 — Batched metadata and artifact reads + +Use one negotiated, closed collection operation backed by Runs' current artifact +owners. Default to at most 20 exact items, following current bounded list windows; +derive the aggregate encoded response ceiling from the operation transport cap. +Each whole/segment item retains the existing per-item reader's size limit (the +current segment ceiling is 8 MiB where selected); the batch total never exceeds +its declared aggregate cap. No unbounded archive, recursive path selection or +implicit whole-Run export. Pagination/continuation pins the immutable inventory +identity; a changed inventory refuses stale continuation. + +Each request item identifies Run (or the exact supported attached publication-result +Ref), artifact and expected inventory/hash plus +segment offset/length when applicable. Authenticate once but check current +transitive disclosure for every selected resource. Return ordered, correlated +per-item success or typed failure, with verified size/hash/offset and provenance. +One item failing never upgrades others to success or erases valid completed items. +Whole collection has explicit partial status. An aggregate response alone is not +an integrity proof: verify each artifact's complete hash before local publication. + +Collection is read-only and never starts/adopts execution. Cancellation stops +remaining reads and cleans provisional files; a resume manifest lists exact +verified successes and outstanding/refused items. Re-check authority on resumed +items. Never concatenate different inventory revisions or treat a missing segment +as zero bytes. Corrupt/unauthorized items cannot be cached as successful results. +Sequential bounded service is the default; any bounded I/O concurrency needs +separate measured justification and ownership, without changing Run capacity. + +## C6 — SDK/CLI and scientific comparison + +Add one small `tuldok_client.py` facade over existing AgentClient, with explicit +`submit`, `monitor`, `cancel`, `collect` and `compare` workflows. Extend +`project_cli.py` with matching concepts and machine-readable output; retain its +existing commands and private configuration. Import performs no application, +database, Torch, process or network startup. The facade generates routine exact +submission/collection manifests from verified selected domain results, exposes +them for inspection/export, and delegates each effect to its canonical owner. +It cannot create an owner grant, mutate Study budgets, install runtimes, choose +a scientific winner or implement its own queue. The extension author SDK remains +the evidence producer/conformance kit; do not mix its role with this consumer. + +`submit` accepts an already reviewed material/Trial and grant, retains a stable +request key/body, and returns an acknowledged Run or explicit unknown outcome. +An unknown mutation is reconciled by the same identity; no automatic new key, +new attempt or fallback dispatch. `monitor` polls read-only with a finite overall +deadline, cancellation and capped server-aware backoff; stopping monitoring does +not imply cancelling compute. `cancel` is an explicit scoped server action and +distinguishes requested from drained. `collect` uses C5 when negotiated and the +existing verified single-artifact reader when selected capabilities genuinely +lack batching; auth/decoding/server errors do not trigger alternate requests. +Write mode0600 files via private provisional paths, verified complete hash and +atomic no-clobber publication. `compare` selects exact operands and delegates +compatibility/metrics to the evidence owner. Preserve typed failure/partial/ +unknown outcomes through Python and CLI exit/output contracts. No shell eval or +credentials in generated manifests/logs. Client timeouts are not execution facts. + +Comparison extends the existing `model_evidence.compare_evidence`/field-evidence +readers and Experiment progression actions. An operand binds catalog, Model/ +Trial/Run/material, checkpoint or observation identity, evaluation data/split/ +case/metric-definition/profile versions, units and preprocessing/output policy. +Compare only compatible meanings and a declared case matching policy; missing +cases remain missing. No inference from filenames, cohort labels or matching +array shapes. Reject incompatible data, split, units, metric direction/definition, +case correspondence or normalization with an actionable typed reason. Explicit +unit conversion is permitted only by an existing owned conversion contract; +otherwise refuse. Do not fit/refit normalization or inspect holdout for selection. + +For scientific per-case pass/fail, bind an owned criterion Ref with threshold, +direction, units, evaluation scope and criterion authority; distinguish qualified +acceptance from diagnostic-only checks. Show pass/fail/unknown for each case and +operand. Without a supported owned criterion or complete evidence, show unknown +or unavailable with its reason; do not infer a threshold from a plotted value. +This is separate from the training iteration/pass-count axis. + +Show per-case baseline/candidate values and regression deltas, aggregate coverage, +accuracy/error versus actual recorded iteration or pass count, elapsed timings, +peak-memory source/scope, exact config differences, and links to retained logs +and diagnostics. Zero-pass closed-form fits stay zero-pass; no synthetic epochs. +Missing timing/memory is unknown, never zero. Host versus producer measurements, +RSS versus GPU peak, and overlapping wall-time versus cumulative work retain their +units/source and may be incommensurate. Sort/paginate/filter bounded populations; +sampled charts label their coverage and preserve access to exact data tables. + +Lineage remains immutable Trial relationships owned by Experiments: retain exact +baseline refs, distinguish derivation from comparison, and separate Run retries +from new Trials. PR89 Project activity stays explicitly external observation, +with freshness/provenance; it never receives managed custody, budget or numerical +acceptance labels. External evidence may be displayed under its declared trust +class but cannot masquerade as a managed Run or enter an authoritative managed +comparison without the same supported capture/verification contract. Selection +or navigation launches no training. Keyboard, narrow-screen and text/table paths +must reach the same result; stale deliveries/actor changes cannot overwrite +current selection or disclose revoked evidence. + +## Consumers and security/resource limits + +| Owner / producer | Consumers to change or preserve | Disposition | +| --- | --- | --- | +| Pipeline material + preflight | Run/Study admission, browser Pipeline, AgentClient/CLI/MCP | One builder and exact binding; new negotiated output when closed schema changes; old projections and refusal semantics retained. | +| Runs execution/custody/publication | Study reconciliation, artifact/evidence/compare readers, SDK/Run UI/lineage result links, candidate selection/package guards | Exact versioned attached result Ref for new readers; legacy failed gates unchanged; candidate/package promotion and new recovered-input bindings explicitly unsupported; one attempt/grant/reservation, no execution retry. | +| Pumas/runtime verification | Preflight, supervisor/bootstrap, captured material | Installation owner unchanged; reference exact inventories; retain current device and mutation checks. | +| Evidence profile/writer/verifier | SDK producer, host publication, cold readers and comparison | Coordinated new profile version when needed; old profiles continue exact reading, no inferred migration. | +| Runs health and phase spans | UI, CLI, observer/operator projection | Read-only bounded projections; permission/redaction; observation cannot grant readiness or prove science. | +| Experiments/lineage + comparison | Existing lineage graph, UI/SDK/CLI | Exact identities/current access; stable baseline relation; evidence owner decides comparability. | +| Metadata/storage + backup | Explicit migration/restore, historical readers | Supported-version inventory and stopped backup/upgrade/reopen/rollback tests; raw evidence never overwritten. | + +Keep existing inspect/edit/execute/cancel/owner distinctions at every trusted +operation. Owner-only recovery is discoverable but unavailable to ordinary agents. +Enforce catalog/actor scoping, revocation and current transitive input disclosure +on preflight, status, batch, recovery and comparison. Cache identity conveys no +authority. Use established no-follow/regular-file/size/stability checks, reject +path traversal/symlinks/special files, and retain private files/directories. +Diagnostics/log snippets are bounded and escaped; no HTML/code execution or raw +credential/host-path disclosure. Mandatory audit commits remain distinct from +best-effort interface observation. No new package dependency, installer or +network egress is required by this plan's default design. + +## Acceptance procedures + +These are future exact tests/procedures, not results. Each fixture starts with a +valid case and changes only the tested condition; assert its exact failing owner, +code and no-effect proof. Use isolated catalog/database/roots/credentials/ports +and join owned work; do not use the persistent lab catalog as a failure fixture. + +### T1 — Preflight and bounded evidence (A1) + +Extend current external/Study tests with a supported neutral valid material, then +individually mutate runtime inventory/device, input/source SHA, Model/profile, +fixed-size output minimum, shard/index/total bounds, required evidence capability +and storage reserve. Assert precise checked/invalid/unsupported/unavailable +diagnostic and material binding; zero process starts, GO, grant consumption and +reservation/Run count delta on refusal. Interleave drift between report and +admission/dispatch; assert fresh authoritative refusal, no use of stale proof. +Round-trip a valid multi-shard index through the actual writer, host verifier and +cold reader; test reordered/missing/duplicate/oversize/corrupt shards separately. + +### T2 — Publication recovery without retraining (A2) + +Use one isolated real Linux custodian attempt with finite output and an optimizer +or execution counter that is independently observed. Inject publication failure +after valid exit/drain at copy, fsync/rename and DB commit acknowledgement seams. +Record original failure and hashes; stop/reopen the service and explicitly recover +the same Run. Compare all output bytes and verified evidence against the original +staging inventory. Assert the same Run/request/material/reservation, one grant +consumption/charge, no additional child/GO/counter increment and exact recovered +publication labeling. Interleave two recoveries, cancel and revocation; prove +one publication or the precise safe blocked outcome. Tampered/oversize outputs +or missing drain must refuse, never complete. Verify finite staging accounting, +retention/abandonment and cold access after Project roots disappear. + +Also read the same recovered Ref through new negotiated artifact/evidence/compare +operations via HTTP/AgentClient, then cold reopen with roots removed. Assert identical exact +Ref/hashes, old selected readers still report failed/unavailable, and candidate +selection/package promotion refuses that recovered legacy Run without changing +selection/reservation/counts. Exercise every publication-result Ref identity +mismatch independently. Exhaust wall/I/O/history/concurrency/temporary-copy bounds; +assert the precise paused/limit outcome, no success and no extra compute charge. + +### T3 — RES-1 (A3) + +With real PostgreSQL advisory locks, hold the required lock on a separate +connection until the coordinator reaches its configured timeout. Observe its +health/fault through authenticated HTTP and prove dispatch is fenced. Repeat +with an actual active neutral custodian and at the queued-to-GO seam; verify +cancel/drain ownership. Release the lock, explicitly recover concurrently from +two requests, and assert one joined replacement generation, unchanged pending +identities/capacity and no duplicate GO/process. Missing/invalid custody receipt, +unknown commit and same-boot uncertainty must remain fenced. Concurrent close +wins over recovery, leaves no coordinator/futures/children and never clears a +shutdown flag. Run the integrated PR90 capacity/unknown-custody/scheduling tests. + +### T4 — Honest timings and visibility (A4) + +Use controllable clocks to assert monotonic durations, UTC/clock jumps, overlap, +open/unknown spans, cache-hit preparation and recovery publication intervals. +Compare owner event instants with independently recorded harness boundaries; +never derive expected elapsed by calling the production summarizer. In a real +browser/backend workflow, observe queued capacity, coordinator fault/wait, +execution/cancel/drain, validation and publication recovery. Assert live labels, +source/units, no false sum, stale polling cleanup, focus/keyboard and narrow-screen +access. Both CLI and UI must show matching semantic states. Producer training +time is separately labelled and cannot be substituted for total workflow time. + +### T5 — Controlled cost and integrity comparison (A5) + +`tests/benchmark_lab_workflow.py` will record baseline/candidate commits, same +inputs/runtime/environment, hardware, catalog history population, active/queued +state and measured phase/request boundary. Exercise cold process+cache miss, +warm verified material reuse, runtime mutation/miss, and terminal-history sizes +representative of empty/small/grown lab catalogs. Include the observed55-terminal +case when reproducible, without making55 a product bound. Alternate baseline and +candidate order; preserve raw samples, warmup policy, call/hash counts, wall and +process CPU time, RSS/I/O and hit/miss provenance. Choose repetitions from observed +variability and report distributions rather than one best sample. + +Measure preparation, Run reads, metadata collection and whole/segment collection +separately, then one end-to-end supported workflow. Include same-size metadata +and many-artifact cases, corruption/revocation and a partial failed item. Verify +identical typed outcomes and per-item hashes plus no admission/custody changes. +Agree a workload-specific retention decision after baseline variance is known; +do not invent a target or infer a whole-pipeline speedup from189-byte reads. +Unsupported/missing runtime/hardware or baseline is unavailable, not a pass. + +### T6 — Public client workflow (A6) + +From an isolated base Python environment, import the client with app/DB/Torch/ +subprocess/network startup tripwires. Use actual authenticated HTTP and CLI against +an isolated supported server to export routine manifests, submit an authorized +neutral attempt once, monitor with deadline, cancel/drain, collect and compare +exact retained results. Assert generated JSON is inspectable and deterministically +binds the selected meaning. Test lost submit reply with stable-key reconciliation, +conflicting body, revoked/expired/insufficient scope, unavailable capabilities, +mixed batch results, hash mismatch, interrupted/resumed segments and no-clobber +destination races. No silent new request key or implicit mutation retry. Also run +frozen old client/selected operation contracts and current CLI negotiation tests. + +### T7 — Scientific comparison and lineage (A7) + +Create a small compatible scientific pair with explicit per-case expected values +calculated independently of the comparison code. Through actual artifact readers +and the real UI, select exact baseline/candidate/checkpoints; assert case deltas, +coverage, real iteration/pass coordinates, timings/memory scope, config differences, +log/diagnostic links and exact lineage refs. Mutate each data/split/unit/metric/ +case/normalization compatibility dimension separately, asserting its precise +unsupported/incompatible reason and no invented delta. Exercise missing cases, +zero-pass fit, missing memory/timings, external activity versus managed evidence, +revocation, stale overlapping responses, keyboard/table equivalence and narrow +screens. No UI selection starts compute. A lab pair, if supplied, is a separately +recorded real artifact/read acceptance; these fixtures do not establish physics, +model quality or new CUDA execution. + +Include one owned scientific per-case criterion with independently calculated +pass and fail, one missing criterion yielding unknown, and diagnostic-only versus +qualified criteria. Assert threshold/direction/units/owner identity and coverage +remain visible; the comparison cannot invent an acceptance rule. Include an exact +recovered publication-result Ref and test legacy promotion refusal separately +from valid collection/comparison. + +### T8 — Compatibility, retained state and release gates (A8) + +Record supported client, evidence and catalog versions on the integrated source. +Prove old selected operation documents/wire projections unchanged; new clients +negotiate implemented versions and refuse unsupported servers without a mutation +fallback. Copy an isolated stopped old catalog, run the explicit supported upgrade, +inject interruption before/after ledger commit, reopen and restore through real +PostgreSQL/filesystem adapters. Compare raw artifact/material/receipt hashes and +business counts; old missing fields remain unknown. Prove an older binary refuses +new unsupported state and document stopped backup/forward-repair rollback, with +no data destruction or compute replay. Full exact-head required CI, independent +cross-owner review and lead approval are required before production activation. + +No acceptance procedure run is implied by this document. Fresh CUDA/hardware and +scientific claims require their own actual environment/input receipt; missing +inputs stay visible rather than replaced by a synthetic successful workload. diff --git a/docs/plans/lab-workflow-reliability/reports/independent-plan-review.md b/docs/plans/lab-workflow-reliability/reports/independent-plan-review.md new file mode 100644 index 00000000..dfb80ca1 --- /dev/null +++ b/docs/plans/lab-workflow-reliability/reports/independent-plan-review.md @@ -0,0 +1,129 @@ +# Independent implementation-plan review + +Reviewer: independent GPT-6.1 Sol High review worker. Date: 2026-10-19. +Disposition: **clear for documentation publication and a later explicitly +authorized implementation handoff; no remaining blocking plan finding**. +This is not implementation, product acceptance, integration approval or a +Standards Engine certification. + +## Exact reviewed subject + +The subject is the uncommitted draft in the separate +`docs/lab-workflow-reliability-plan` worktree, based on main +`300f12df6f48e028fd9379911703f7c89337952a`. The final binding material has these +SHA256 identities: + +| File, relative to this plan directory | SHA256 | +| --- | --- | +| `plan.md` | `62d33606c5829d7bb26562985fdc8d13e8f8f3e7380ab5ef9e31fabd96752af3` | +| `reports/contracts-and-acceptance.md` | `815dc87cfb438b83563c3671e0c91ab8ae51fc5b035cf030a7544eb8dfe257c5` | +| `reports/baseline-and-standards.md` | `60a887e2169ad9966242dc70c6edb133df115d9f809debb4549cb131ff12f4a4` | + +Also read the current issues, execution ledger and validation report. Their +subsequent recording of this review or final validation does not change the +binding subject above. Material semantic changes require affected re-review; +recording unchanged evidence does not impose a new commit-topology requirement. + +Inspected adopted Coding Standards at +`dcc56f26e884ade260770beceba2501d3746200d`: Core, Router, canonical Planning, +Architecture and Development Proportionality, with relevant Implementation, +Verification, contract/proof-lifetime/evolution, resilience, security and +performance guidance. Read the accepted D1 architecture/evidence design and +the existing Run, external-recipe, Model/Study, lineage, storage and agent/client +contracts and affected source. Read the exact PR90 objects at +`429d0c4ff44dba16bc95313d2a0980169dbc9b9f`, including RES-1 and coordinator +startup/loop/shutdown. PR89/90 remain proposed dependencies in this plan; +reviewing their objects does not establish a merge or current CI success. + +## Findings and dispositions + +### R1 — Recovered evidence needed an explicit supported consumer contract + +**Originally blocking; resolved in the reviewed revision.** Keeping a legacy +Run failed while merely publishing recovered bytes did not define how actual +completed-state-gated artifact/evidence readers or candidate selection would +consume those bytes. Existing `runs.py` readers and +`Experiments.select_candidate` cannot acquire that meaning through a UI label. + +C2 now owns an exact versioned attached publication-result Ref binding catalog, +Run, material, inventory, custody receipt and result identity. New negotiated +artifact/evidence/compare readers, collection, Run presentation and lineage +links consume that Ref through the supported verifier. Legacy failed projections +remain failed/unavailable. Candidate/package promotion and new recovered-result +input bindings are explicitly unsupported in this initial scope. M1's consumer +and write-set inventory and T2/T7 now cover new and old readers, cold/rootless +access, Ref mismatches and promotion refusal. This makes recovery useful without +silently weakening legacy completion or selection contracts. + +### R2 — Distinguish scientific per-case criteria from training passes + +**Recommendation; resolved.** C6 and T7 now bind an owned criterion Ref, +threshold, direction, units and scope; show per-case pass/fail/unknown; and keep +diagnostic checks distinct from qualified acceptance. Missing criteria stay +unknown. This is separate from the actual training iteration/pass-count axis; +neither UI arithmetic nor synthetic fixtures invent scientific thresholds. + +### R3 — Bound publication recovery independently from compute admission + +**Recommendation; resolved.** C2 now specifies a named finite initial wall +budget, serialized catalog/Run recovery, inventory-derived verification/copy +I/O, one reserved temporary copy and bounded detailed history. Exhaustion pauses +with a limit outcome. Original/final proofs and mandatory audits remain retained; +owner policy cannot enlarge the original settled resource envelope. T2 exercises +these limits and unchanged compute charges. These are proposed operational +bounds to verify during implementation, not measured throughput claims. + +## Review conclusions + +- C1 binds preflight to exact submission material and required checks while + preserving current admission/dispatch authorization, storage, capacity and + mutable runtime/device rechecks. It promises no prediction of arbitrary + output success or future resource availability. +- C2 preserves the original attempt, fault and successful-drain proof, uses + explicit owner recovery and stable reconciliation identity, and forbids new + GO, training, grants, reservations or compute charges. Invalid/oversize + required evidence cannot be repaired by relabelling or silent truncation. +- C3 addresses actual RES-1 inside Runs: unhealthy readiness fences, owned + futures/custodians, bounded joins, serialized replacement and durable custody + reconciliation precede readiness. Unknown GO/custody and final shutdown are + not cleared by a live thread, dead PID or automatic restart. +- Phase spans distinguish elapsed time from cumulative/overlapping work, retain + clock domains and uncertainty, cover preparation before admission, and expose + stale/open/unknown intervals. Producer timing is not workflow timing. +- M4 measures before retaining machinery. C4 does not cache grants, access, + capacity or mutable installed-runtime proof. C5 preserves per-item authority, + inventory/hash identity, partial outcomes, segment bounds and complete-hash + local publication within a bounded batch. +- The SDK/CLI remain thin negotiated clients with stable mutation identities, + explicit unknown outcomes, finite monitoring and cancel-versus-drained + semantics. They acquire no scheduler, installer, owner budget or grant role. +- Scientific comparison binds compatible data/splits/cases/metrics/units and + exact evidence operands, preserves missing values and measurement scope, and + distinguishes external observations from managed custody. Lineage intent + does not establish quality or trigger compute. +- The six milestones are proportionate dependency-ordered vertical outcomes, + with named owners, write sets, acceptance paths and re-plan triggers. M1 uses + existing HTTP/AgentClient consumers rather than depending on the future M5 + facade. Supported public/persisted versions, stopped migration/restore, + interrupted commits and backup/forward-repair rollback remain explicit. + +## Checks actually performed and limits + +Rehashed the three binding documents and independently ran the unchanged +`validate_plan` AST/constants/helpers from the adopted local checker. Result: +**PASS**, returning `None`. Checker source SHA256: +`c7266c0564b9ee02944fc4eb12d66cc20d604fd4ddb7f051ed77d01e9fe4466e`. +This proves that checker's selected structural rules, not semantic completeness +or application behavior. Final directory-wide link/whitespace/scope validation +is recorded separately by the plan author. + +Only this allocated review report was written by this reviewer. No product, +shared branch, PR89/90 source, catalog, hardware/runtime, producer or application +test was changed or executed. No benchmark, scientific or CUDA result was +established. All required source and standards reads succeeded; no environment +access limitation affected this review. A1–A8 and T1–T8 remain future +implementation acceptance. Execution +still requires the later explicit plan instruction, lead disposition of +PR89/90/current baseline, actual required-environment tests, independent review +and applicable release gates. Missing lab receipts, compatible scientific pairs +and hardware remain visible dependent evidence needs. diff --git a/docs/plans/lab-workflow-reliability/reports/validation.json b/docs/plans/lab-workflow-reliability/reports/validation.json new file mode 100644 index 00000000..5d909ce2 --- /dev/null +++ b/docs/plans/lab-workflow-reliability/reports/validation.json @@ -0,0 +1,33 @@ +{ + "documentation_paths": [ + "docs/plans/lab-workflow-reliability/execution-ledger.md", + "docs/plans/lab-workflow-reliability/issues.md", + "docs/plans/lab-workflow-reliability/plan.md", + "docs/plans/lab-workflow-reliability/reports/baseline-and-standards.md", + "docs/plans/lab-workflow-reliability/reports/contracts-and-acceptance.md", + "docs/plans/lab-workflow-reliability/reports/independent-plan-review.md", + "docs/plans/lab-workflow-reliability/reports/validation.json", + "docs/plans/lab-workflow-reliability/reports/validation.md" + ], + "format": "documentation_plan_validation_v1", + "limits": [ + "Structural validation and local documentation checks only; no Engine certification or product acceptance.", + "External references independently inspected via GitHub/Git, not exercised as product behavior.", + "No application/catalog/runtime/hardware tests run." + ], + "material_sha256": { + "docs/plans/lab-workflow-reliability/issues.md": "5263b33da1a5c85e3794c8f34f4f3bb9b2bcfa8d8c1ea1ba36a19fad3340b471", + "docs/plans/lab-workflow-reliability/plan.md": "62d33606c5829d7bb26562985fdc8d13e8f8f3e7380ab5ef9e31fabd96752af3", + "docs/plans/lab-workflow-reliability/reports/baseline-and-standards.md": "60a887e2169ad9966242dc70c6edb133df115d9f809debb4549cb131ff12f4a4", + "docs/plans/lab-workflow-reliability/reports/contracts-and-acceptance.md": "815dc87cfb438b83563c3671e0c91ab8ae51fc5b035cf030a7544eb8dfe257c5" + }, + "passed": true, + "plan_sha256": "62d33606c5829d7bb26562985fdc8d13e8f8f3e7380ab5ef9e31fabd96752af3", + "relative_link_failures": [], + "relative_links_checked": 23, + "scope_failures": [], + "source_revision": "dcc56f26e884ade260770beceba2501d3746200d", + "structural_diagnostic": null, + "structural_validator_sha256": "c7266c0564b9ee02944fc4eb12d66cc20d604fd4ddb7f051ed77d01e9fe4466e", + "whitespace_failures": [] +} diff --git a/docs/plans/lab-workflow-reliability/reports/validation.md b/docs/plans/lab-workflow-reliability/reports/validation.md new file mode 100644 index 00000000..b1196109 --- /dev/null +++ b/docs/plans/lab-workflow-reliability/reports/validation.md @@ -0,0 +1,65 @@ +# Plan preparation validation + +Subject: documentation under `docs/plans/lab-workflow-reliability/`, based on +main `300f12df6f48e028fd9379911703f7c89337952a`. These checks concern the plan, +not the proposed product behavior. All A1–A8 acceptance claims remain pending. + +## Checks and evidence boundary + +| Check | Actual result / scope | +| --- | --- | +| Repository/worktree inspection | Separate docs branch/worktree; unrelated work preserved; no product, runtime, catalog or workflow writes. | +| Remote baseline and dependencies | Anonymous `git ls-remote` and GitHub reads agree on recorded main/PR89/PR90 SHAs and open/unmerged dependency state. No CI or merge success is inferred. | +| Exact upstream structural validator | PASS: `validate_plan(plan_text)` returns `None` using the original AST, constants and helpers at Coding Standards `dcc56f26e884ade260770beceba2501d3746200d`. | +| Manual policy/contract application | Current plan template, eight architecture probes, acceptance-kind/environment/mode distinctions, domain ownership and explicit compatibility/recovery decisions reviewed. See [baseline and standards](baseline-and-standards.md). | +| Independent review | CLEAR after the one blocking finding and two recommendations were resolved; [independent plan review](independent-plan-review.md) pins exact material identities. Report SHA256: `b6666f6036f798cb62bead680f1ddb4cceb3049aed87358b4478d2f5de8b9d10`. | +| Relative links, whitespace and scope | PASS: all23 relative links/anchors resolve, no whitespace findings, all8 changed/new paths confined to this plan directory. Exact material hashes and actual results are in [validation.json](validation.json). Code fences are excluded from Markdown-link parsing. | +| Standards Engine application route | Unavailable: no exposed MCP; local application-purpose invocation returns exit2, `Application invocation is unavailable; use the published interface contract.` No authoring bypass/full Engine certification was used. | +| Application, catalog, GPU and scientific tests | Not run: documentation-only change; no product pass, hardware reproduction, benchmark speedup or scientific acceptance is claimed. | + +## Structural checker provenance + +Canonical source: +`tools/standards_verifier/standards_verifier/checks/plan_contract.py` in the +adopted local Coding Standards checkout. Source SHA256: +`c7266c0564b9ee02944fc4eb12d66cc20d604fd4ddb7f051ed77d01e9fe4466e`. +No verifier logic, constants or expected rules were changed. As in the existing +D1 plan check, load just its standard-library AST before the first checker class +to avoid unrelated aggregate package initialization: + +```python +import ast +from pathlib import Path + +source = Path("/workspace/coding-standards/tools/standards_verifier/" + "standards_verifier/checks/plan_contract.py") +body = [] +for node in ast.parse(source.read_text()).body: + if isinstance(node, ast.ClassDef): + break + if (isinstance(node, (ast.Assign, ast.FunctionDef)) + or isinstance(node, ast.Import) + and all(alias.name == "re" for alias in node.names) + or isinstance(node, ast.ImportFrom) and node.module == "__future__"): + body.append(node) +namespace = {} +exec(compile(ast.Module(body=body, type_ignores=[]), str(source), "exec"), namespace) +plan = Path("docs/plans/lab-workflow-reliability/plan.md") +assert namespace["validate_plan"](plan.read_text()) is None +``` + +This proves selected lifecycle/structure rules, not semantic completeness or +future implementation conformance. Independent review and explicit T1–T8 +procedures cover the plan's intended meaning; future implementation requires +its own real acceptance evidence and exact candidate CI. + +## Reproduction and final scope + +Run the structural snippet above from the plan worktree; resolve every relative +Markdown link/anchor against its file; run `git diff --check` and full new-file +`git diff --no-index --check /dev/null FILE` (diff exit1 alone is normal); inspect +the exact staged diff and confirm every path is under this plan directory. +No permanent validator or custom test framework is added to the application. +The local disposable helper `/tmp/validate_lab_plan.py` combines these read-only +checks and emits the linked JSON, excluding self-referential validation/review/ +ledger hashes from its material-subject hash set.