pstack: add create-verification-skill and maintain-verification-skill#150
Conversation
Generalizes the control-glass approach (feature map, doctor, proof standards, harness-first) for any language or platform. The generator interviews the repo, writes a project-local verify skill + seeded feature map, and must prove its own output by running it once. The maintainer is the upkeep loop: source wave per feature, one live pass, at most one PR. setup-pstack gains an optional final step offering the generator. Validated by 4 cloud agents generating against real repos (go TUI, node CLI, HTTP service, full-stack web app) - all four proof runs passed, and their friction reports drove 6 revisions.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
|
|
||
| ### 7. Offer a verification skill (optional) | ||
|
|
||
| Check whether the project has a way to drive the real app for proof (a `verify-*` skill, or an existing harness). If not, offer once: "want a project-local verification skill, so agents can drive the app the way a user does and prove changes work? I can generate one with /create-verification-skill." On yes, read and follow `create-verification-skill`. On no, move on without pushing. |
There was a problem hiding this comment.
Harness check skips verify offer
Medium Severity · Logic Bug
Step 7's logic for offering a project-local verification skill is overbroad. It incorrectly interprets existing e2e tooling or non-app-driving verify-* skills (like verify-this) as fulfilling the need for a new, app-driving skill with a feature map. This prevents the offer from appearing for projects that could benefit, and the detection also omits common skill paths.
Triggered by learned rule: Skill detection must cover all installation paths
Reviewed by Cursor Bugbot for commit e12d7ee. Configure here.
There was a problem hiding this comment.
Dismissing: step 7 is a deliberate judgment call, not a detection algorithm. Enumerating skill paths and disambiguating verify-this-style names in the skill body trades a one-line offer for a brittle checklist; a wrong guess costs one extra question to the user. The offer text already scopes it to 'a way to drive the real app for proof'.
|
|
||
| 1. **Index hygiene.** Read the feature map README and glob its sibling files. Fix missing, extra, duplicate, or dead entries. Lightweight; no generated inventory. | ||
|
|
||
| 2. **Source wave.** One read-only subagent per feature file, launched concurrently. Each explains "how does this user-facing feature work?" from source, flags likely doc drift with citations, and returns one concise live-verification recipe. Children never drive the app and never edit files. Return shape: feature summary / source entry points / likely drift or none / one recipe. |
There was a problem hiding this comment.
Source wave not actually readonly
Medium Severity · Logic Bug
The "Source wave" step intends for subagents to be read-only and not edit files or drive the app. However, readonly: true isn't set, allowing these concurrent subagents to mutate the verification skill's files or run harness commands, which could race the coordinator's later triage.
Reviewed by Cursor Bugbot for commit e12d7ee. Configure here.
There was a problem hiding this comment.
Dismissing: the skill's contract is behavioral ('children never drive the app and never edit files') and runner-agnostic — pstack skills don't prescribe Task-parameter details because runners differ. The coordinator owns all writes either way.
…-failure cleanup, teardown after re-proof
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 4 potential issues.
There are 6 total unresolved issues (including 2 from previous reviews).
Bugbot Autofix prepared fixes for all 4 issues found in the latest run.
- ✅ Fixed: Mid-pass cleanup vs long-lived drives
- Failed long-lived drives now trigger cleanup followed by relaunch and doctor before serial coverage continues.
- ✅ Fixed: Doctor retry skips prior cleanup
- Doctor-drift recovery now cleans the failed attempt and retries the full Launch-Doctor sequence once.
- ✅ Fixed: Evidence never checked after cleanup
- Every cleanup now requires confirming all captured evidence still exists, with missing evidence invalidating the proof.
- ✅ Fixed: No re-doctor after failed drive
- A failed drive now requires a fresh session or a relaunched long-lived instance that passes doctor before further driving.
Or push these changes by commenting:
@cursor push 94dc37953a
Preview (94dc37953a)
diff --git a/pstack/skills/maintain-verification-skill/SKILL.md b/pstack/skills/maintain-verification-skill/SKILL.md
--- a/pstack/skills/maintain-verification-skill/SKILL.md
+++ b/pstack/skills/maintain-verification-skill/SKILL.md
@@ -30,7 +30,7 @@
3. **Reconcile.** Every feature file has a returned summary. Merge overlapping recipes into as few app states as practical. Spot-check cited drift; don't re-prove clean claims. Sweep recent churn for user-facing surfaces missing from the map — require a concrete source path before calling one missing.
-4. **Live pass.** Required even when source looks clean. The coordinator owns all driving; follow the verification skill's own launch model — one long-lived instance driven serially for servers and UIs, or a fresh isolated session per drive for short-lived CLIs (the skill's Launch section decides, not this one). Health-check at the skill's own granularity: doctor before driving a long-lived instance, and per session where sessions are the unit. Never drive past a failing doctor — but a doctor that fails because the *skill* drifted is itself drift: fix it under edit scope, retry once, and only then call the pass `blocked`. Exercise every feature at least once, cleaning up any failed drive immediately so a stuck session can't poison the rest of the pass. A feature that can't be reached is `verified-unreachable` only with the concrete prerequisite (auth, entitlement, OS, external state) and the route attempted; if the map omits that prerequisite, that's drift. Any harness fix from triage gets re-driven live before it ships. Final teardown happens after the last drive of the run — including those re-proofs — so nothing outlives the run (evidence stays, per the skill).
+4. **Live pass.** Required even when source looks clean. The coordinator owns all driving; follow the verification skill's own launch model — one long-lived instance at a time driven serially for servers and UIs, or a fresh isolated session per drive for short-lived CLIs (the skill's Launch section decides, not this one). Health-check at the skill's own granularity: doctor before driving a long-lived instance, and per session where sessions are the unit. Never drive past a failing doctor — but a doctor that fails because the *skill* drifted is itself drift: run Cleanup for anything the failed attempt started, fix it under edit scope, retry the Launch-Doctor sequence once, and only then call the pass `blocked`. Exercise every feature at least once. After a failed drive, run Cleanup immediately; before continuing, start and doctor a fresh isolated CLI session or relaunch and doctor the long-lived server/UI instance. A feature that can't be reached is `verified-unreachable` only with the concrete prerequisite (auth, entitlement, OS, external state) and the route attempted; if the map omits that prerequisite, that's drift. Any harness fix from triage gets re-driven live before it ships. Final teardown happens after the last drive of the run — including those re-proofs — so nothing outlives the run. After every Cleanup — failed doctor attempt, failed drive, or final teardown — confirm that all evidence captured so far still exists at the skill's named location; missing evidence fails the affected proof.
5. **Triage.** Wrong or missing user-POV description → doc drift, fix it. Working behavior the harness can't drive → harness gap, fix it; a harness fix follows the same helpers rule as generation (scripts executable, invocation documented in the skill body). App behavior that's actually broken → product gap; record it for the user, keep it out of this PR.You can send follow-ups to the cloud agent here.
Reviewed by Cursor Bugbot for commit 45f6601. Configure here.
|
|
||
| 3. **Reconcile.** Every feature file has a returned summary. Merge overlapping recipes into as few app states as practical. Spot-check cited drift; don't re-prove clean claims. Sweep recent churn for user-facing surfaces missing from the map — require a concrete source path before calling one missing. | ||
|
|
||
| 4. **Live pass.** Required even when source looks clean. The coordinator owns all driving; follow the verification skill's own launch model — one long-lived instance driven serially for servers and UIs, or a fresh isolated session per drive for short-lived CLIs (the skill's Launch section decides, not this one). Health-check at the skill's own granularity: doctor before driving a long-lived instance, and per session where sessions are the unit. Never drive past a failing doctor — but a doctor that fails because the *skill* drifted is itself drift: fix it under edit scope, retry once, and only then call the pass `blocked`. Exercise every feature at least once, cleaning up any failed drive immediately so a stuck session can't poison the rest of the pass. A feature that can't be reached is `verified-unreachable` only with the concrete prerequisite (auth, entitlement, OS, external state) and the route attempted; if the map omits that prerequisite, that's drift. Any harness fix from triage gets re-driven live before it ships. Final teardown happens after the last drive of the run — including those re-proofs — so nothing outlives the run (evidence stays, per the skill). |
There was a problem hiding this comment.
Mid-pass cleanup vs long-lived drives
Medium Severity · Logic Bug
The live pass maintains a single long-lived instance for serial server/UI drives. The instruction to immediately clean up failed drives causes the skill's Cleanup to tear down this shared instance mid-pass, preventing subsequent features from being driven and breaking serial coverage.
Reviewed by Cursor Bugbot for commit 45f6601. Configure here.
There was a problem hiding this comment.
Valid — fixed in follow-up #151: cleanup now happens at the granularity of what failed; a shared long-lived instance is never torn down mid-pass.
|
|
||
| 3. **Reconcile.** Every feature file has a returned summary. Merge overlapping recipes into as few app states as practical. Spot-check cited drift; don't re-prove clean claims. Sweep recent churn for user-facing surfaces missing from the map — require a concrete source path before calling one missing. | ||
|
|
||
| 4. **Live pass.** Required even when source looks clean. The coordinator owns all driving; follow the verification skill's own launch model — one long-lived instance driven serially for servers and UIs, or a fresh isolated session per drive for short-lived CLIs (the skill's Launch section decides, not this one). Health-check at the skill's own granularity: doctor before driving a long-lived instance, and per session where sessions are the unit. Never drive past a failing doctor — but a doctor that fails because the *skill* drifted is itself drift: fix it under edit scope, retry once, and only then call the pass `blocked`. Exercise every feature at least once, cleaning up any failed drive immediately so a stuck session can't poison the rest of the pass. A feature that can't be reached is `verified-unreachable` only with the concrete prerequisite (auth, entitlement, OS, external state) and the route attempted; if the map omits that prerequisite, that's drift. Any harness fix from triage gets re-driven live before it ships. Final teardown happens after the last drive of the run — including those re-proofs — so nothing outlives the run (evidence stays, per the skill). |
There was a problem hiding this comment.
Doctor retry skips prior cleanup
Medium Severity · Logic Bug
The new skill-drift retry path doesn't include a Cleanup or relaunch before retrying doctor. If Launch or Doctor itself drifted, this can leave half-started instances or stranded processes, causing the retry to fail and the pass to block unnecessarily.
Reviewed by Cursor Bugbot for commit 45f6601. Configure here.
There was a problem hiding this comment.
Valid — fixed in follow-up #151: the drift retry path now cleans up and relaunches before retrying doctor.
|
|
||
| 3. **Reconcile.** Every feature file has a returned summary. Merge overlapping recipes into as few app states as practical. Spot-check cited drift; don't re-prove clean claims. Sweep recent churn for user-facing surfaces missing from the map — require a concrete source path before calling one missing. | ||
|
|
||
| 4. **Live pass.** Required even when source looks clean. The coordinator owns all driving; follow the verification skill's own launch model — one long-lived instance driven serially for servers and UIs, or a fresh isolated session per drive for short-lived CLIs (the skill's Launch section decides, not this one). Health-check at the skill's own granularity: doctor before driving a long-lived instance, and per session where sessions are the unit. Never drive past a failing doctor — but a doctor that fails because the *skill* drifted is itself drift: fix it under edit scope, retry once, and only then call the pass `blocked`. Exercise every feature at least once, cleaning up any failed drive immediately so a stuck session can't poison the rest of the pass. A feature that can't be reached is `verified-unreachable` only with the concrete prerequisite (auth, entitlement, OS, external state) and the route attempted; if the map omits that prerequisite, that's drift. Any harness fix from triage gets re-driven live before it ships. Final teardown happens after the last drive of the run — including those re-proofs — so nothing outlives the run (evidence stays, per the skill). |
There was a problem hiding this comment.
Evidence never checked after cleanup
Medium Severity · Logic Bug
The live pass newly cleans up after failed drives and again at final teardown, but only states that evidence stays. Unlike the generator's prove loop, it never requires confirming proof artifacts still exist after those cleanups, so a cleanup that deletes evidence can still leave the run treating features as proven.
Reviewed by Cursor Bugbot for commit 45f6601. Configure here.
There was a problem hiding this comment.
Valid — fixed in follow-up #151: every cleanup is followed by an evidence-survival check, mirroring the generator's proof rule.
|
|
||
| 3. **Reconcile.** Every feature file has a returned summary. Merge overlapping recipes into as few app states as practical. Spot-check cited drift; don't re-prove clean claims. Sweep recent churn for user-facing surfaces missing from the map — require a concrete source path before calling one missing. | ||
|
|
||
| 4. **Live pass.** Required even when source looks clean. The coordinator owns all driving; follow the verification skill's own launch model — one long-lived instance driven serially for servers and UIs, or a fresh isolated session per drive for short-lived CLIs (the skill's Launch section decides, not this one). Health-check at the skill's own granularity: doctor before driving a long-lived instance, and per session where sessions are the unit. Never drive past a failing doctor — but a doctor that fails because the *skill* drifted is itself drift: fix it under edit scope, retry once, and only then call the pass `blocked`. Exercise every feature at least once, cleaning up any failed drive immediately so a stuck session can't poison the rest of the pass. A feature that can't be reached is `verified-unreachable` only with the concrete prerequisite (auth, entitlement, OS, external state) and the route attempted; if the map omits that prerequisite, that's drift. Any harness fix from triage gets re-driven live before it ships. Final teardown happens after the last drive of the run — including those re-proofs — so nothing outlives the run (evidence stays, per the skill). |
There was a problem hiding this comment.
No re-doctor after failed drive
Medium Severity · Logic Bug
For long-lived apps, doctor runs once before driving, then the pass continues after cleaning up a failed drive. There is no health re-check before the next feature, so a drive that left the shared instance hung, crashed, or corrupted can keep being driven and mislabeled as map, harness, or product issues.
Reviewed by Cursor Bugbot for commit 45f6601. Configure here.
There was a problem hiding this comment.
Valid — fixed in follow-up #151: a failed drive on a long-lived instance triggers re-doctor before the next feature.
… checks (#151) * maintain-verification-skill: cleanup granularity, re-doctor, evidence checks Follow-up to the four bugbot comments that landed on #150 seconds before merge: failed-drive cleanup now matches the granularity of what failed (never tearing down a shared instance mid-pass), a failed drive on a long-lived instance triggers re-doctor before the next feature, the doctor-drift retry includes cleanup and relaunch, and every cleanup is followed by an evidence-survival check. * State the live-pass recovery rules as invariants, not enumerated procedures * Restore the per-session doctor check inside invariant 1
1 participant
Summary
/create-verification-skill: generates a project-localverify-<app>skill for any language/framework/platform — interviews the repo (surface, run, drive, observe, isolate), writes launch/doctor/drive/evidence/cleanup sections with real selectors, seeds a user-POV feature map, and must prove its own output by executing it once. Generalizes control-skill ideas: the feature map as the maintained verification source, a doctor command, real-user-path proof standards, harness-first reuse./maintain-verification-skill: the upkeep loop — index hygiene, a read-only source wave per feature file, one coordinator-owned live pass driving every feature, drift/harness-gap/product-gap triage, clean/changed/blocked outcomes, at most one PR.setup-pstackstep 7 (optional): offers the generator when the project has no verification path.Eval (4 cloud agents, real repos, isolated VMs)
gum (go TUI), np (interactive node CLI), fastify-example-twitter (HTTP + mongo/redis), Reactive-Resume (react/hono/postgres). All four generated skills passed their own proof runs — fastify drove 9 real HTTP actions and verified evidence in both stores; resume brought up isolated postgres + chromium and proved an autosave round-trip. Two agents caught real upstream drift in their targets (gum's README submit key; np's misleading dry-run output). Their friction reports drove 6 revisions now in the skill: finite-CLI launch framing, seed-3-5-prove-ONE made explicit, evidence survives cleanup, dry-run hermeticity must be observed not assumed, marked scaffolding allowed for irrelevant blockers, helpers ship executable with documented invocation.
Test plan
Note
Low Risk
Documentation and agent skill definitions only; no runtime product code or auth/data paths change.
Overview
Adds agent playbooks for project-local “drive the real app and prove it” verification, plus wiring so setup can surface them.
/create-verification-skillinterviews the repo (surface, run, drive, observe, isolate), writes a.cursor/skills/verify-<app>/skill with launch/doctor/drive/evidence/cleanup/helpers grounded in real harnesses, seeds a user-POV feature map, and must execute the generated skill once (one mapped feature, evidence survives cleanup) before delivery./maintain-verification-skillis the upkeep loop: index hygiene, parallel read-only source pass per feature file, coordinator-owned live pass over every feature, triage into doc/harness/product gaps, and outcomes clean / changed (at most one PR of proven fixes) / blocked — edits stay inside the verify skill directory only.README documents both skills;
setup-pstackgains an optional step 7 that offers generation when noverify-*skill or harness exists. Plugin version 0.11.0.Reviewed by Cursor Bugbot for commit 45f6601. Bugbot is set up for automated code reviews on this repo. Configure here.