Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -70,4 +70,11 @@ jobs:
- name: Test
env:
SYNCWHEEL_SOURCE_ROOT: ${{ github.workspace }}/syncwheel
run: pnpm test
run: node scripts/test-command-matrix.mjs "${{ runner.temp }}/test-command-matrix.json"

- name: Upload command test matrix
if: always()
uses: actions/upload-artifact@v4
with:
name: test-command-matrix
path: ${{ runner.temp }}/test-command-matrix.json
22 changes: 22 additions & 0 deletions docs/testing-invariants.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Invariant baseline and command attribution

Run `node scripts/test-command-matrix.mjs [output.json]` from the repository root after installing dependencies. Without an output argument, the report is saved under the operating system temporary directory. Parser scratch files also live there. The script runs the entire Vitest suite with the default reporter streamed to stdout/stderr; failures remain in the output and yield a nonzero exit. It loads the actual Commander tree with only its executable entrypoint removed, including root, groups and hidden commands. No command handler runs during inventory.

`scripts/test-command-map.json` currently links only three commands and is deliberately conservative: every link names exactly one runner-registered file/full-name pair and documents why that test covers the command. Imports and shared files do not imply command coverage. Add audited links as coverage is established; an unattributed test is not evidence of absent behavior. Unknown commands, missing or ambiguous cases, duplicate IDs and registration/result mismatches refuse generation. Parameterized cases remain individual runner IDs. Per-command counts overlap; only the union of IDs is a suite total. IDs are Vitest IDs, stable for the same runner/project/module/test order. The separate JSON reporter reconciles the overall count.

The report records input-byte and parser hashes, runner and Node versions. It excludes timing, absolute paths and stack traces. `unattributedCommands` lists commands with no declared test links; this is not a claim that their behavior is untested. Unattributed tests are explicitly listed to reconcile the full runner inventory. `it.fails` outcomes count as runner passes and carry `expectedFailure: true`; this is not proof that the invariant holds. Skipped and pending outcomes are separate from executed cases. Generated reports are observations of one environment and are not committed as a portable status baseline.

## Effects harness

`test/helpers/effects.ts` records file bytes in base64, permission bits, directories and symlink targets without following links. Explicit manifest paths record complete parsed claims as well as original bytes. With `git: true`, the snapshot also records the staged index (semantic entries and raw bytes), refs and symbolic HEAD; the filesystem tree includes raw Git metadata. Locks under the fixture root are included. Filesystem atime, mtime, ctime and birthtime are not recorded; all listed content and permission fields are compared exactly. There are no default path exclusions. A caller may exclude only an exact file or symlink path with a nonempty reason; directories are always traversed and manifest claims cannot be excluded. Compare snapshots with `diffEffects`; unknown changed paths remain visible. This is an isolated filesystem observer, not a syscall/network/process tracer.

## Expected failures

The invariant tests use synthetic fixtures and no live targets. Each expected-failure test has an adjacent ordinary test that checks the exact current defect, so a broken setup cannot masquerade as an expected failure. A product fix must remove that marker in the same change: an unexpected pass fails CI. Do not replace the marker with skip or todo.

| Invariant | Case | Observed defect |
| --- | --- | --- |
| I2 | `read_missing_git_preserves_cache` | Fetch removes an existing incomplete cache directory and its sentinel before a failing local clone. |
| I5 | `stale_merge_metadata_rejected_in_all_paths` | Preview accepts unchanged source with stale contribution metadata while apply rejects the contribution. |

An unreachable source during pin preflight produces a warning by design, not a defect: it does not prove a new artifact collision. The existing pin-preflight tests retain that intended behavior.
23 changes: 23 additions & 0 deletions scripts/test-command-map.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
{
"schemaVersion": 1,
"links": [
{
"file": "test/cli-verb-redesign.test.ts",
"name": "CLI verb redesign > previews and applies Git cache pruning",
"commands": ["cache prune"],
"evidence": "Runs cache prune in preview and apply modes and checks retained cache entries."
},
{
"file": "test/cli-verb-redesign.test.ts",
"name": "CLI verb redesign > drafts a registry publish submission from a GitHub URL without writing workspace state",
"commands": ["registry publish"],
"evidence": "Invokes registry publish and checks its draft and workspace effects."
},
{
"file": "test/cli-verb-redesign.test.ts",
"name": "CLI verb redesign > doctor suggests the Copilot user companion skill install without writing files",
"commands": ["doctor"],
"evidence": "Invokes doctor and checks the suggested install and unchanged tree."
}
]
}
137 changes: 137 additions & 0 deletions scripts/test-command-matrix.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
import { createHash } from "node:crypto";
import { readFile, writeFile, mkdtemp, rm } from "node:fs/promises";
import { join, relative, resolve } from "node:path";
import { tmpdir } from "node:os";
import { pathToFileURL } from "node:url";
import { spawnSync, execFileSync } from "node:child_process";
import { createRequire } from "node:module";


const root = process.cwd();
const require = createRequire(import.meta.url);
// Use the bundler shipped with the pinned, direct tsup development dependency.
const { build } = createRequire(require.resolve("tsup/package.json"))("esbuild");
const compare = (a, b) => a < b ? -1 : a > b ? 1 : 0;
const hash = bytes => createHash("sha256").update(bytes).digest("hex");
const read = path => readFile(join(root, path), "utf8");

export function counts(tests) {
const pass = tests.filter(t => t.status === "passed").length;
const fail = tests.filter(t => t.status === "failed").length;
return { registered: tests.length, executed: pass + fail, pass, fail,
skipped: tests.filter(t => t.status === "skipped").length,
pending: tests.filter(t => t.status === "pending").length };
}

export function reconcile(commands, registered, results, links) {
const byId = new Map(registered.map(t => [t.id, t]));
const resultIds = new Set(results.map(t => t.id));
if (byId.size !== registered.length || resultIds.size !== results.length
|| byId.size !== resultIds.size || [...byId.keys()].some(id => !resultIds.has(id))) {
throw new Error("Registered and executed runner IDs do not reconcile");
}
const attributed = new Map(commands.map(command => [command, new Set()]));
const assigned = new Set();
const selectors = new Set();
for (const link of links) {
const key = `${link.file}::${link.name}`;
if (selectors.has(key) || !link.evidence?.trim() || !Array.isArray(link.commands) || !link.commands.length) throw new Error(`Invalid mapping: ${key}`);
selectors.add(key);
const matches = registered.filter(t => t.file === link.file && t.name === link.name);
if (matches.length !== 1) throw new Error(`Mapping must match exactly one registered test: ${key} (${matches.length})`);
for (const command of link.commands) {
if (!attributed.has(command)) throw new Error(`Unknown parser command: ${command}`);
attributed.get(command).add(matches[0].id);
assigned.add(matches[0].id);
}
}
const rows = commands.map(command => {
const tests = results.filter(t => attributed.get(command).has(t.id)).sort((a,b) => compare(a.id,b.id));
return { command, ...counts(tests), tests };
});
return { rows, uniqueAttributed: assigned.size,
unattributed: results.filter(t => !assigned.has(t.id)).sort((a,b) => compare(a.id,b.id)) };
}

export async function parserCommands(scratch) {
const source = await read("src/cli/index.ts");
const marker = "try {\n await main();";
if (source.split(marker).length !== 2) throw new Error("CLI entry boundary changed; inspect inventory loader");
// Load the actual Commander tree, replacing only the executable entrypoint.
const code = `${source.slice(0, source.indexOf(marker))}\nexport { program };\n`;
const output = join(scratch, "parser.mjs");
await build({ stdin: { contents: code, resolveDir: join(root,"src/cli"), sourcefile: "index.ts", loader: "ts" },
bundle: true, plugins: [{ name: "resolve-temporary-parser-imports", setup(context) {
context.onResolve({ filter: /^[^./]/ }, args => ({
path: args.path.startsWith("node:") ? args.path : import.meta.resolve(args.path), external: true,
}));
} }], platform: "node", format: "esm", outfile: output, write: true });
const { program } = await import(pathToFileURL(output));
const commands = [program.name()];
function visit(parent, prefix) {
for (const command of parent.commands) {
const path = [...prefix, command.name()];
commands.push(path.join(" "));
visit(command, path);
}
}
visit(program, []);
if (new Set(commands).size !== commands.length) throw new Error("Duplicate parser command path");
return commands.sort(compare);
}

async function sourceReceipt() {
// Hash exact source/test/script/config bytes; no absolute paths, timestamps or self-referential output.
const tracked = execFileSync("git", ["ls-files", "-z"], { cwd: root, encoding: "utf8" }).split("\0").filter(Boolean);
const paths = tracked.filter(path => /^(src|test|scripts)\//.test(path)
|| ["package.json", "pnpm-lock.yaml", "vitest.config.ts", "tsconfig.json"].includes(path));
const files = await Promise.all(paths.sort(compare).map(async path => [path, hash(await readFile(join(root,path)))]));
return { inputDigest: hash(JSON.stringify(files.sort((a,b) => compare(a[0],b[0])))),
parserSha256: hash(await read("src/cli/index.ts")),
vitest: require("vitest/package.json").version,
node: process.versions.node };
}

export function runTests(report, json) {
return spawnSync("pnpm", ["test", "--maxWorkers=2", "--reporter=default", "--reporter=json",
"--reporter=./scripts/test-command-reporter.mjs", `--outputFile=${json}`], {
cwd: root, env: { ...process.env, COMMAND_MATRIX_REPORT: report }, stdio: "inherit",
});
}

export async function matrixDestination(path) {
return path ? resolve(path) : join(await mkdtemp(join(tmpdir(), "command-matrix-output-")), "matrix.json");
}

async function main() {
const scratch = await mkdtemp(join(tmpdir(), "command-matrix-parser-"));
const reportRoot = await mkdtemp(join(tmpdir(), "command-matrix-"));
try {
const commands = await parserCommands(scratch);
const report = join(reportRoot,"runner.json");
const json = join(reportRoot,"vitest.json");
const runner = runTests(report, json);
if (runner.error) throw runner.error;
const observed = JSON.parse(await readFile(report,"utf8"));
const official = JSON.parse(await readFile(json,"utf8"));
const officialTests = official.testResults.flatMap(file => file.assertionResults);
if (officialTests.length !== observed.tests.length) throw new Error("JSON reporter test count mismatch");
const mapping = JSON.parse(await read("scripts/test-command-map.json"));
if (mapping.schemaVersion !== 1) throw new Error("Unsupported mapping schema");
const matrix = reconcile(commands, observed.registered, observed.tests, mapping.links);
const output = { schemaVersion: 1, receipt: await sourceReceipt(),
commandCount: commands.length, unattributedCommands: matrix.rows.filter(row => !row.registered).map(row => row.command),
totals: counts(observed.tests), uniqueAttributed: matrix.uniqueAttributed,
runnerExitCode: runner.status, moduleErrors: observed.moduleErrors, unhandledErrors: observed.unhandledErrors,
commands: matrix.rows, unattributed: matrix.unattributed };
const destination = await matrixDestination(process.argv[2]);
await writeFile(destination, `${JSON.stringify(output,null,2)}\n`);
console.log(JSON.stringify({ commandCount: output.commandCount, unattributedCommands: output.unattributedCommands.length,
...output.totals, uniqueAttributed: output.uniqueAttributed, output: relative(root,destination) }));
if (runner.status !== 0 || observed.moduleErrors.length || observed.unhandledErrors) process.exitCode = 1;
} finally {
await rm(scratch, { recursive: true, force: true });
await rm(reportRoot, { recursive: true, force: true });
}
}
if (process.argv[1] && import.meta.url === pathToFileURL(resolve(process.argv[1])).href) await main();
28 changes: 28 additions & 0 deletions scripts/test-command-reporter.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
import { writeFileSync } from "node:fs";

// Uses runner-owned IDs, including parameter expansion, rather than source-text guesses.
export default class CommandMatrixReporter {
registered = new Map();
onTestModuleCollected(module) {
for (const test of module.children.allTests()) {
if (this.registered.has(test.id)) throw new Error(`Duplicate runner test ID: ${test.id}`);
this.registered.set(test.id, {
id: test.id, file: module.relativeModuleId, name: test.fullName,
expectedFailure: test.options.fails === true,
});
}
}
onTestRunEnd(modules, errors, reason) {
const tests = [];
for (const module of modules) for (const test of module.children.allTests()) {
const registered = this.registered.get(test.id);
if (!registered) throw new Error(`Unregistered runner test: ${test.id}`);
tests.push({ ...registered, status: test.result().state });
}
writeFileSync(process.env.COMMAND_MATRIX_REPORT, JSON.stringify({
registered: [...this.registered.values()], tests, reason,
moduleErrors: modules.filter(module => !module.ok()).map(module => module.relativeModuleId),
unhandledErrors: errors.length,
}));
}
}
56 changes: 56 additions & 0 deletions test/command-matrix.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
import { mkdtemp, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import { dirname, join, relative } from "node:path";
import { spawnSync } from "node:child_process";
import { expect, it, vi } from "vitest";
// @ts-expect-error The CLI support script is native ESM JavaScript.
import { reconcile, counts, runTests, matrixDestination, parserCommands } from "../scripts/test-command-matrix.mjs";

vi.mock("node:child_process", async importOriginal => ({
...await importOriginal<typeof import("node:child_process")>(), spawnSync: vi.fn(),
}));

it("reconciles shared command attribution without double-counting unique runner IDs", () => {
const registered = [{ id: "a", file: "case.ts", name: "parameter 1" }, { id: "b", file: "case.ts", name: "parameter 2" }];
const results = registered.map(test => ({ ...test, status: "passed" }));
const matrix = reconcile(["one", "two", "zero"], registered, results,
[{ file: "case.ts", name: "parameter 1", commands: ["one", "two"], evidence: "shared fixture" }]);
expect(matrix.uniqueAttributed).toBe(1);
expect(matrix.rows.map((row: { registered: number }) => row.registered)).toEqual([1,1,0]);
expect(matrix.unattributed.map((test: { id: string }) => test.id)).toEqual(["b"]);
expect(() => reconcile(["one"], registered, [results[0]], [])).toThrow(/reconcile/);
expect(() => reconcile(["one"], registered, results, [{ file: "case.ts", name: "missing", commands: ["one"], evidence: "fixture" }])).toThrow(/exactly one/);
expect(() => reconcile(["one"], registered, results, [{ file: "case.ts", name: "parameter 1", commands: ["unknown"], evidence: "fixture" }])).toThrow(/Unknown/);
expect(counts([{ status: "passed" }, { status: "failed" }, { status: "skipped" }, { status: "pending" }])).toEqual({ registered: 4, executed: 2, pass: 1, fail: 1, skipped: 1, pending: 1 });
});

it("rejects mappings without a commands array with an explicit validation error", () => {
const registered = [{ id: "a", file: "case.ts", name: "case" }];
for (const commands of [undefined, null, "one", []]) {
expect(() => reconcile(["one"], registered, registered,
[{ file: "case.ts", name: "case", evidence: "fixture", commands }])).toThrow(/Invalid mapping/);
}
});

it("streams default reporter output and runner errors to CI", () => {
runTests("runner.json", "vitest.json");
expect(vi.mocked(spawnSync)).toHaveBeenCalledWith("pnpm", expect.arrayContaining(["test", "--reporter=default"]),
expect.objectContaining({ stdio: "inherit" }));
});

it("keeps the default report outside the checkout in a temporary directory", async () => {
const output = await matrixDestination();
try {
expect(relative(tmpdir(), output)).toMatch(/^command-matrix-output-[^/]+\/matrix\.json$/);
expect(await matrixDestination("custom.json")).toBe(join(process.cwd(), "custom.json"));
} finally { await rm(dirname(output), { recursive: true, force: true }); }
});

it("loads the Commander tree from a scratch directory outside the checkout", async () => {
const scratch = await mkdtemp(join(tmpdir(), "command-matrix-parser-test-"));
try {
const commands = await parserCommands(scratch);
expect(commands).toEqual(expect.arrayContaining(["agentwheel", "cache prune", "registry publish", "doctor"]));
expect(new Set(commands).size).toBe(commands.length);
} finally { await rm(scratch, { recursive: true, force: true }); }
});
35 changes: 35 additions & 0 deletions test/effects-harness.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
import { execFileSync } from "node:child_process";
import { chmod, mkdir, mkdtemp, rm, symlink, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { expect, it } from "vitest";
import { diffEffects, snapshot } from "./helpers/effects.js";

it("detects bytes, modes, symlinks, claims, index and refs without following links", async () => {
const root = await mkdtemp(join(tmpdir(), "effects-harness-"));
try {
const git = (...args: string[]) => execFileSync("git", ["-C", root, ...args]);
git("init", "-b", "main");
git("config", "user.name", "Synthetic");
git("config", "user.email", "synthetic@example.invalid");
await writeFile(join(root, "file"), Buffer.from([0,255,1]));
await writeFile(join(root, "manifest.json"), JSON.stringify({ entries: [{ owner: "alpha" }] }));
await symlink("file", join(root,"link"));
git("add", "file"); git("commit", "-m", "fixture");
const policy = { manifests: ["manifest.json"], git: true };
const before = await snapshot(root, policy);
expect(diffEffects(before, await snapshot(root,policy))).toEqual([]);
await writeFile(join(root,"file"), Buffer.from([0,255,2]));
await chmod(join(root,"file"), 0o755);
await rm(join(root,"link")); await symlink("missing",join(root,"link"));
await writeFile(join(root,"manifest.json"), JSON.stringify({ entries: [{ owner: "beta" }] }));
git("add", "file"); git("branch", "other");
const changes = diffEffects(before, await snapshot(root,policy));
expect(changes).toEqual(expect.arrayContaining(["tree:file", "tree:link", "claims:manifest.json", "git:index", "git:refs"]));
await expect(snapshot(root, { exclude: { "**": "invalid" } })).rejects.toThrow(/exact/);
await expect(snapshot(root, { exclude: { "file": "" } })).rejects.toThrow(/reason/);
const excluded = { exclude: { "file": "This fixture explicitly ignores the mutable file payload." } };
expect((await snapshot(root,excluded)).tree.file).toBeUndefined();
await expect(snapshot(root, { manifests: ["manifest.json"], exclude: { "manifest.json": "invalid claim exclusion" } })).rejects.toThrow(/claims/);
} finally { await rm(root, { recursive: true, force: true }); }
});
Loading
Loading