Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
bd0da97
test: reproduce admission, pin and cache invariants
Yehonal Nov 16, 2026
156ee7e
test: use supported local git source syntax
Yehonal Nov 16, 2026
38ccbc0
test: complete invariant fixture inputs
Yehonal Nov 16, 2026
cca0024
test: inventory Commander commands and reconcile runner test identities
Yehonal Nov 16, 2026
50f4520
test: verify effect snapshots and command attribution accounting
Yehonal Nov 16, 2026
6eb5abc
test: retain verified invariant failures and publish CI command matrix
Yehonal Nov 16, 2026
f31d029
test: resolve runner executable through its exported package metadata
Yehonal Nov 16, 2026
e7ddfec
test: require stale admission refusal before additional effects
Yehonal Nov 16, 2026
adb319d
test: include root permissions and raw Git index in effect snapshots
Yehonal Nov 16, 2026
3429cb1
test: run the declared package test command for matrix observations
Yehonal Nov 16, 2026
71101a9
test: correct invariant scope and expose matrix runner failures
Yehonal Nov 16, 2026
3f8bf8e
test: reproduce persistent effects around synthetic status reads
Yehonal Nov 16, 2026
c290660
test: use an unambiguous synthetic workspace selector
Yehonal Nov 16, 2026
cc2ec0c
test: expose cache deletion and lease effects before the read policy fix
Yehonal Nov 16, 2026
92e8cf0
fix: observe status caches without implicit refresh or builds
Yehonal Nov 16, 2026
bb1015e
test: qualify offline usefulness, retained snapshots and adapter refusal
Yehonal Nov 16, 2026
a7edab1
test: trigger snapshot pruning with a newer synthetic source revision
Yehonal Nov 16, 2026
6389ee8
test: preserve legacy status reporting with uncached version metadata
Yehonal Nov 16, 2026
667d6d2
fix: clarify cache freshness and incomplete checkout recovery
Yehonal Nov 16, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -70,4 +70,11 @@ jobs:
- name: Test
env:
SYNCWHEEL_SOURCE_ROOT: ${{ github.workspace }}/syncwheel
run: pnpm test
run: node scripts/test-command-matrix.mjs "${{ runner.temp }}/test-command-matrix.json"

- name: Upload command test matrix
if: always()
uses: actions/upload-artifact@v4
with:
name: test-command-matrix
path: ${{ runner.temp }}/test-command-matrix.json
27 changes: 27 additions & 0 deletions docs/read-effects.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Status observation and explicit refresh

`status` now observes local state and existing caches by default, regardless of cache TTL. `status --refresh` explicitly permits cache population, source fetches, snapshot maintenance, version discovery and composite member refresh. `--offline` wins over `--refresh`. Missing or stale evidence remains visible in the status protocol instead of causing an implicit refresh. Useful cached Git snapshots and local sources still support pending-work inspection; an absent cached snapshot returns a named refresh requirement.

## Consumer behavior change and cache refresh responsibility

Previously, a status read could refresh expired member and package-version caches automatically. It no longer does. The default `fleet-diagnostics` wrapper forwards an observation to `status`; after the member or version TTL expires it therefore shows stale evidence (`STALE`, or a degraded status derived from stale package metadata) until an explicit refresh succeeds. Repeated diagnostics reads do not make those caches fresh. Missing caches stay missing and are reported as unavailable.

The operator or caller requesting fresh evidence now owns the refresh: run `agentwheel status` with the **same workspace/fleet and profile selectors** plus `--refresh`, or explicitly opt into the diagnostics caller's existing `--refresh` option. The refresh path updates the member-report and version-index caches; subsequent default reads consume those caches without extending their freshness. No background refresher, timer, or new automated refresh owner is added here. Any change to fleet-control consumers or their refresh schedule is a separate maintainer decision and follow-up; this PR does not change fleet-control.

For example, `agentwheel status --fleet example-fleet --profile example-profile --refresh --json` refreshes that selected scope. `--offline` must be omitted when refreshing. Existing mutating commands retain their own cache behavior; this change concerns the read route.

## Effect policy

The AsyncLocalStorage scope in `src/effects/read-policy.ts` centralizes this distinction across graph sources, registry/version indexes, member reports, and cache writers. Observation never creates cache leases or maintenance locks and does not prune snapshots. Incomplete checkout directories are preserved even during refresh: repair is a separate explicit operation. The error identifies the exact checkout directory to move into a new backup location outside its cache directory. Confirm the original checkout path is absent, then rerun the same scoped command with `--refresh` to clone it again. Do not overwrite existing backups. No repair command or automatic move/removal is provided. Git status disables optional index locks. The optional update notifier does not fetch or persist on status reads.

Observation supports built-in local targets and cached composite reports. Unqualified programmatic adapters, source drivers other than local/Git, and direct remote transports refuse before execution; explicit refresh retains their existing admission and execution paths. This deliberately avoids promising zero effects for arbitrary executable code. Other CLI read/preview verbs have not yet been qualified by this batch.

Diagnostics consumers of the versioned `status --json` protocol inherit this policy. Tests exercise a synthetic child-process consumer, not a private fleet configuration. No fleet diagnostics source is changed by this PR.

`scripts/agentwheel-local.mjs` is the generic source-owned local launcher. It requires a prebuilt, current `dist/index.js`, or refuses with an explicit build instruction. It never runs a package manager, acquires a build lock, or builds implicitly, for any verb. `AGENTWHEEL_REPO_DIR` selects a checkout; the default is the repository containing the launcher. This PR does not replace any installed wrapper or retarget a runtime launcher.

## Evidence and limits

The B0a effects helper is reused from PR #201; this PR is based on that branch. Before the product changes, synthetic status fixtures reproduced clone/fetch/tag queries, persistent cache leases and version-index writes, global pruning when a newer source commit creates a snapshot, and removal of a sentinel from an incomplete checkout. A separately isolated legacy-launcher probe reproduced one implicit build on a status invocation with a dummy builder. That launcher was unversioned: the replacement above establishes a source-owned implementation without copying host-specific defaults into public source.

The fixture snapshots synthetic HOME, workspace, cache, Git metadata and temp before/after each invocation. A child preload records process calls and rejects Node network APIs; only its exact evidence file is excluded from tree comparison. Tests cover cold/warm/offline/refused reads, a diagnostics protocol consumer, wrapper admission, and explicitly bounded refresh writes. External Git network access is checked through command arguments; this is an application-boundary trace, not a kernel syscall sandbox. Concurrent refresh/prune versus a lease-free observer remains a separate cache-lifecycle qualification concern: observation may report missing evidence but cannot repair or delete it.
22 changes: 22 additions & 0 deletions docs/testing-invariants.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Invariant baseline and command attribution

Run `node scripts/test-command-matrix.mjs [output.json]` from the repository root after installing dependencies. Without an output argument, the report is saved under the operating system temporary directory. Parser scratch files also live there. The script runs the entire Vitest suite with the default reporter streamed to stdout/stderr; failures remain in the output and yield a nonzero exit. It loads the actual Commander tree with only its executable entrypoint removed, including root, groups and hidden commands. No command handler runs during inventory.

`scripts/test-command-map.json` currently links only three commands and is deliberately conservative: every link names exactly one runner-registered file/full-name pair and documents why that test covers the command. Imports and shared files do not imply command coverage. Add audited links as coverage is established; an unattributed test is not evidence of absent behavior. Unknown commands, missing or ambiguous cases, duplicate IDs and registration/result mismatches refuse generation. Parameterized cases remain individual runner IDs. Per-command counts overlap; only the union of IDs is a suite total. IDs are Vitest IDs, stable for the same runner/project/module/test order. The separate JSON reporter reconciles the overall count.

The report records input-byte and parser hashes, runner and Node versions. It excludes timing, absolute paths and stack traces. `unattributedCommands` lists commands with no declared test links; this is not a claim that their behavior is untested. Unattributed tests are explicitly listed to reconcile the full runner inventory. `it.fails` outcomes count as runner passes and carry `expectedFailure: true`; this is not proof that the invariant holds. Skipped and pending outcomes are separate from executed cases. Generated reports are observations of one environment and are not committed as a portable status baseline.

## Effects harness

`test/helpers/effects.ts` records file bytes in base64, permission bits, directories and symlink targets without following links. Explicit manifest paths record complete parsed claims as well as original bytes. With `git: true`, the snapshot also records the staged index (semantic entries and raw bytes), refs and symbolic HEAD; the filesystem tree includes raw Git metadata. Locks under the fixture root are included. Filesystem atime, mtime, ctime and birthtime are not recorded; all listed content and permission fields are compared exactly. There are no default path exclusions. A caller may exclude only an exact file or symlink path with a nonempty reason; directories are always traversed and manifest claims cannot be excluded. Compare snapshots with `diffEffects`; unknown changed paths remain visible. This is an isolated filesystem observer, not a syscall/network/process tracer.

## Invariant cases

The invariant tests use synthetic fixtures and no live targets. I2 now uses ordinary regression tests; I5 remains an expected failure. Each expected-failure test has an adjacent ordinary test that checks the exact current defect, so a broken setup cannot masquerade as an expected failure. A product fix must remove that marker in the same change: an unexpected pass fails CI. Do not replace the marker with skip or todo.

| Invariant | Case | Observed defect |
| --- | --- | --- |
| I2 | `read_missing_git_preserves_cache` | Fixed: an incomplete cache checkout is preserved and refuses implicit repair. |
| I5 | `stale_merge_metadata_rejected_in_all_paths` | Preview accepts unchanged source with stale contribution metadata while apply rejects the contribution. |

An unreachable source during pin preflight produces a warning by design, not a defect: it does not prove a new artifact collision. The existing pin-preflight tests retain that intended behavior.
23 changes: 23 additions & 0 deletions scripts/agentwheel-local.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
#!/usr/bin/env node
import { readdirSync, statSync } from "node:fs";
import { dirname, join, resolve } from "node:path";
import { fileURLToPath } from "node:url";
import { spawnSync } from "node:child_process";

const root = resolve(process.env.AGENTWHEEL_REPO_DIR ?? join(dirname(fileURLToPath(import.meta.url)), ".."));
const entry = join(root,"dist/index.js");
function newest(path) {
let stat;
try { stat = statSync(path); } catch { return 0; }
return stat.isDirectory() ? Math.max(stat.mtimeMs,...readdirSync(path).map(name => newest(join(path,name)))) : stat.mtimeMs;
}
let built = 0;
try { built = statSync(entry).mtimeMs; } catch { /* Missing builds are refused below. */ }
if (!built || ["src","package.json","pnpm-lock.yaml","tsconfig.json","tsup.config.ts"].some(path => newest(join(root,path)) > built)) {
console.error("agentwheel: built CLI is missing or stale. Build explicitly with pnpm build in the selected checkout; this wrapper never builds implicitly.");
process.exitCode = 127;
} else {
const result = spawnSync(process.execPath,[entry,...process.argv.slice(2)],{ stdio: "inherit" });
if (result.error) console.error(result.error.message);
process.exitCode = result.status ?? 1;
}
23 changes: 23 additions & 0 deletions scripts/test-command-map.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
{
"schemaVersion": 1,
"links": [
{
"file": "test/cli-verb-redesign.test.ts",
"name": "CLI verb redesign > previews and applies Git cache pruning",
"commands": ["cache prune"],
"evidence": "Runs cache prune in preview and apply modes and checks retained cache entries."
},
{
"file": "test/cli-verb-redesign.test.ts",
"name": "CLI verb redesign > drafts a registry publish submission from a GitHub URL without writing workspace state",
"commands": ["registry publish"],
"evidence": "Invokes registry publish and checks its draft and workspace effects."
},
{
"file": "test/cli-verb-redesign.test.ts",
"name": "CLI verb redesign > doctor suggests the Copilot user companion skill install without writing files",
"commands": ["doctor"],
"evidence": "Invokes doctor and checks the suggested install and unchanged tree."
}
]
}
137 changes: 137 additions & 0 deletions scripts/test-command-matrix.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
import { createHash } from "node:crypto";
import { readFile, writeFile, mkdtemp, rm } from "node:fs/promises";
import { join, relative, resolve } from "node:path";
import { tmpdir } from "node:os";
import { pathToFileURL } from "node:url";
import { spawnSync, execFileSync } from "node:child_process";
import { createRequire } from "node:module";


const root = process.cwd();
const require = createRequire(import.meta.url);
// Use the bundler shipped with the pinned, direct tsup development dependency.
const { build } = createRequire(require.resolve("tsup/package.json"))("esbuild");
const compare = (a, b) => a < b ? -1 : a > b ? 1 : 0;
const hash = bytes => createHash("sha256").update(bytes).digest("hex");
const read = path => readFile(join(root, path), "utf8");

export function counts(tests) {
const pass = tests.filter(t => t.status === "passed").length;
const fail = tests.filter(t => t.status === "failed").length;
return { registered: tests.length, executed: pass + fail, pass, fail,
skipped: tests.filter(t => t.status === "skipped").length,
pending: tests.filter(t => t.status === "pending").length };
}

export function reconcile(commands, registered, results, links) {
const byId = new Map(registered.map(t => [t.id, t]));
const resultIds = new Set(results.map(t => t.id));
if (byId.size !== registered.length || resultIds.size !== results.length
|| byId.size !== resultIds.size || [...byId.keys()].some(id => !resultIds.has(id))) {
throw new Error("Registered and executed runner IDs do not reconcile");
}
const attributed = new Map(commands.map(command => [command, new Set()]));
const assigned = new Set();
const selectors = new Set();
for (const link of links) {
const key = `${link.file}::${link.name}`;
if (selectors.has(key) || !link.evidence?.trim() || !Array.isArray(link.commands) || !link.commands.length) throw new Error(`Invalid mapping: ${key}`);
selectors.add(key);
const matches = registered.filter(t => t.file === link.file && t.name === link.name);
if (matches.length !== 1) throw new Error(`Mapping must match exactly one registered test: ${key} (${matches.length})`);
for (const command of link.commands) {
if (!attributed.has(command)) throw new Error(`Unknown parser command: ${command}`);
attributed.get(command).add(matches[0].id);
assigned.add(matches[0].id);
}
}
const rows = commands.map(command => {
const tests = results.filter(t => attributed.get(command).has(t.id)).sort((a,b) => compare(a.id,b.id));
return { command, ...counts(tests), tests };
});
return { rows, uniqueAttributed: assigned.size,
unattributed: results.filter(t => !assigned.has(t.id)).sort((a,b) => compare(a.id,b.id)) };
}

export async function parserCommands(scratch) {
const source = await read("src/cli/index.ts");
const marker = "try {\n await main();";
if (source.split(marker).length !== 2) throw new Error("CLI entry boundary changed; inspect inventory loader");
// Load the actual Commander tree, replacing only the executable entrypoint.
const code = `${source.slice(0, source.indexOf(marker))}\nexport { program };\n`;
const output = join(scratch, "parser.mjs");
await build({ stdin: { contents: code, resolveDir: join(root,"src/cli"), sourcefile: "index.ts", loader: "ts" },
bundle: true, plugins: [{ name: "resolve-temporary-parser-imports", setup(context) {
context.onResolve({ filter: /^[^./]/ }, args => ({
path: args.path.startsWith("node:") ? args.path : import.meta.resolve(args.path), external: true,
}));
} }], platform: "node", format: "esm", outfile: output, write: true });
const { program } = await import(pathToFileURL(output));
const commands = [program.name()];
function visit(parent, prefix) {
for (const command of parent.commands) {
const path = [...prefix, command.name()];
commands.push(path.join(" "));
visit(command, path);
}
}
visit(program, []);
if (new Set(commands).size !== commands.length) throw new Error("Duplicate parser command path");
return commands.sort(compare);
}

async function sourceReceipt() {
// Hash exact source/test/script/config bytes; no absolute paths, timestamps or self-referential output.
const tracked = execFileSync("git", ["ls-files", "-z"], { cwd: root, encoding: "utf8" }).split("\0").filter(Boolean);
const paths = tracked.filter(path => /^(src|test|scripts)\//.test(path)
|| ["package.json", "pnpm-lock.yaml", "vitest.config.ts", "tsconfig.json"].includes(path));
const files = await Promise.all(paths.sort(compare).map(async path => [path, hash(await readFile(join(root,path)))]));
return { inputDigest: hash(JSON.stringify(files.sort((a,b) => compare(a[0],b[0])))),
parserSha256: hash(await read("src/cli/index.ts")),
vitest: require("vitest/package.json").version,
node: process.versions.node };
}

export function runTests(report, json) {
return spawnSync("pnpm", ["test", "--maxWorkers=2", "--reporter=default", "--reporter=json",
"--reporter=./scripts/test-command-reporter.mjs", `--outputFile=${json}`], {
cwd: root, env: { ...process.env, COMMAND_MATRIX_REPORT: report }, stdio: "inherit",
});
}

export async function matrixDestination(path) {
return path ? resolve(path) : join(await mkdtemp(join(tmpdir(), "command-matrix-output-")), "matrix.json");
}

async function main() {
const scratch = await mkdtemp(join(tmpdir(), "command-matrix-parser-"));
const reportRoot = await mkdtemp(join(tmpdir(), "command-matrix-"));
try {
const commands = await parserCommands(scratch);
const report = join(reportRoot,"runner.json");
const json = join(reportRoot,"vitest.json");
const runner = runTests(report, json);
if (runner.error) throw runner.error;
const observed = JSON.parse(await readFile(report,"utf8"));
const official = JSON.parse(await readFile(json,"utf8"));
const officialTests = official.testResults.flatMap(file => file.assertionResults);
if (officialTests.length !== observed.tests.length) throw new Error("JSON reporter test count mismatch");
const mapping = JSON.parse(await read("scripts/test-command-map.json"));
if (mapping.schemaVersion !== 1) throw new Error("Unsupported mapping schema");
const matrix = reconcile(commands, observed.registered, observed.tests, mapping.links);
const output = { schemaVersion: 1, receipt: await sourceReceipt(),
commandCount: commands.length, unattributedCommands: matrix.rows.filter(row => !row.registered).map(row => row.command),
totals: counts(observed.tests), uniqueAttributed: matrix.uniqueAttributed,
runnerExitCode: runner.status, moduleErrors: observed.moduleErrors, unhandledErrors: observed.unhandledErrors,
commands: matrix.rows, unattributed: matrix.unattributed };
const destination = await matrixDestination(process.argv[2]);
await writeFile(destination, `${JSON.stringify(output,null,2)}\n`);
console.log(JSON.stringify({ commandCount: output.commandCount, unattributedCommands: output.unattributedCommands.length,
...output.totals, uniqueAttributed: output.uniqueAttributed, output: relative(root,destination) }));
if (runner.status !== 0 || observed.moduleErrors.length || observed.unhandledErrors) process.exitCode = 1;
} finally {
await rm(scratch, { recursive: true, force: true });
await rm(reportRoot, { recursive: true, force: true });
}
}
if (process.argv[1] && import.meta.url === pathToFileURL(resolve(process.argv[1])).href) await main();
Loading
Loading