gpu-proof merge: shard receipts, re-signed; tri-state config fix

CHANGELOG.md

@@ -4,6 +4,23 @@ All notable changes to pytest-gpu-proof are documented here. The format is
 based on [Keep a Changelog](https://keepachangelog.com/); versions follow
 [SemVer](https://semver.org/) (pre-1.0: minor bumps may break).
 
+## [0.2.0] — 2026-08-07
+
+### Added
+- `gpu-proof merge`: union N shard receipts from one commit into a single
+  re-signed receipt (per-module crash isolation / machine sharding). Refuses
+  shards that disagree on schema, commit SHA, fingerprint, mode, or
+  environment; duplicate node IDs across shards are a hard error. Records
+  per-shard provenance under `session.shards` (additive — the verifier is
+  unchanged). `repo.dirty` is OR-ed; `gpu_info` survives CPU-only shards.
+  Docs: `docs/sharding.md`.
+
+### Fixed
+- Explicit CLI values equal to their built-in defaults are no longer silently
+  ignored in favor of `[tool.gpu_proof]` (value-taking options now register a
+  `None` sentinel; `--gpu-proof-fail-on-skip` ORs with the toml value since a
+  store_true flag can only turn it on).
+
 ## [0.1.0] — 2026-07-07
 
 First public release.

docs/sharding.md

@@ -0,0 +1,62 @@
+# Sharded runs & merging receipts
+
+Large suites often can't (or shouldn't) run as one pytest session: per-module
+subprocesses give crash isolation (one CUDA abort no longer erases the whole
+run's results), and big projects split GPU tests across invocations or
+machines. Each invocation emits its own receipt; CI wants **one** artifact.
+
+## Emit one receipt per shard
+
+Point each invocation at its own output path:
+
+```bash
+pytest tests/gpu/test_a.py --gpu-proof-enable --gpu-proof-out=receipts/a.json
+pytest tests/gpu/test_b.py --gpu-proof-enable --gpu-proof-out=receipts/b.json
+```
+
+Every shard receipt is a complete, individually verifiable receipt.
+
+## Merge
+
+```bash
+gpu-proof merge --out gpu-proof.json receipts/a.json receipts/b.json
+```
+
+`merge` unions the shards' `tests`, spans `session.started_at`/`ended_at`
+across them, records per-shard provenance under `session.shards`
+(`source`, `node_count`, timestamps, and each shard's recorded signer), and
+**re-signs the merged payload with your local SSH key**. The result flows
+through `gpu-proof verify` completely unchanged — same schema, same seven
+checks.
+
+Options:
+
+- `--github-user USERNAME` — recorded signer identity for the merged receipt
+  (default: the first shard's `repo.github_username`).
+- `--key PATH` — SSH private key (default: `git config user.signingKey`, then
+  `~/.ssh/id_ed25519` / `id_ecdsa` / `id_rsa`).
+- `--unsigned` — write `signature: null`; verifies only with
+  `--allow-unsigned`, loudly.
+
+## What merge refuses
+
+A merged receipt must mean exactly what a single-session receipt means, so
+`merge` hard-refuses shards that disagree on anything a receipt pins:
+
+- `schema_version`, `repo.commit_sha`, `fingerprint` (digest + paths),
+  `mode`, or the `environment` the tests ran under (python/pytest/plugin
+  versions, platform);
+- **duplicate node IDs across shards** — two shards attesting the same test is
+  a sharding bug in the runner, never something to dedupe silently.
+
+`repo.dirty` is OR-ed: one dirty shard makes the merged attestation dirty, and
+your verify-time dirty policy applies honestly. `gpu_info` is taken from the
+first shard that has one, so a CPU-only shard doesn't erase the GPU record.
+
+## Trust model
+
+Consistent with the [security model](security_model.md): the merged receipt is
+an **attestation by the merger**. Shard signatures are recorded as provenance
+but not re-verified at merge time (merging is offline); the merged signature
+is what CI verifies. If shards were signed by someone else, verification of
+the merged receipt attests that *you* vouch for the union.

mkdocs.yml

@@ -33,6 +33,7 @@ nav:
   - Quickstart: quickstart.md
   - Local Mode: local_mode.md
   - CI-GPU Mode: ci_gpu_mode.md
+  - Sharding & Merge: sharding.md
   - Architecture: architecture.md
   - Security Model: security_model.md
   - Landscape: landscape.md

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "pytest-gpu-proof"
-version = "0.1.0"
+version = "0.2.0"
 description = "pytest plugin for GPU equivalence testing with signed receipts verified via GitHub SSH keys"
 readme = "README.md"
 requires-python = ">=3.11"

src/pytest_gpu_proof/__init__.py

@@ -1 +1 @@
-__version__ = "0.1.0"
+__version__ = "0.2.0"

src/pytest_gpu_proof/cli.py

@@ -72,8 +72,53 @@ def main():
         "require_gpu = true in [tool.gpu_proof].",
     )
 
+    # merge
+    mp = subparsers.add_parser(
+        "merge",
+        help="Merge shard receipts from ONE commit into a single re-signed receipt",
+        description=(
+            "Union the tests of N shard receipts (same commit SHA, fingerprint, "
+            "and environment) into one receipt, re-signed with your local SSH "
+            "key, that flows through `gpu-proof verify` unchanged. Shards that "
+            "disagree on anything a receipt pins are refused; duplicate node "
+            "IDs across shards are always an error."
+        ),
+    )
+    mp.add_argument("shards", nargs="+", metavar="RECEIPT",
+                    help="Shard receipt paths (two or more, typically)")
+    mp.add_argument("--out", required=True, metavar="PATH",
+                    help="Path for the merged receipt")
+    mp.add_argument("--github-user", default=None, metavar="USERNAME",
+                    help="Recorded signer identity for the merged receipt "
+                    "(default: the first shard's repo.github_username)")
+    mp.add_argument("--key", default=None, metavar="PATH",
+                    help="SSH private key to sign with (default: git "
+                    "user.signingKey, then ~/.ssh/id_ed25519 etc.)")
+    mp.add_argument("--unsigned", action="store_true", default=False,
+                    help="Write signature: null — the merged receipt then "
+                    "verifies only with --allow-unsigned, loudly")
+
     args = parser.parse_args()
 
+    if args.command == "merge":
+        from .merge import MergeError, merge_receipts
+
+        try:
+            receipt = merge_receipts(
+                args.shards, args.out,
+                github_user=args.github_user,
+                key_path=args.key,
+                unsigned=args.unsigned,
+            )
+        except MergeError as e:
+            print(f"gpu-proof merge: {e}", file=sys.stderr)
+            sys.exit(1)
+        n = len(receipt.get("tests", []))
+        shards = len(receipt.get("session", {}).get("shards", []))
+        print(f"merged {shards} shard(s), {n} tests -> {args.out}"
+              + (" (UNSIGNED)" if args.unsigned else ""))
+        sys.exit(0)
+
     if args.command == "verify":
         from .verify import verify_receipt
 

src/pytest_gpu_proof/config.py

@@ -47,10 +47,13 @@ def opt(name, default=None):
             return default
 
     def resolve(opt_name, toml_key, default):
-        """Precedence: CLI flag (when it differs from its built-in default),
-        then [tool.gpu_proof] in pyproject.toml, then the built-in default."""
-        cli = opt(opt_name, default)
-        if cli != default:
+        """Precedence: CLI flag (when explicitly passed), then [tool.gpu_proof]
+        in pyproject.toml, then the built-in default. Value-taking options
+        register with ``default=None`` so an explicit CLI value that happens to
+        equal the built-in default still wins — previously it was silently
+        ignored in favor of the toml value."""
+        cli = opt(opt_name, None)
+        if cli is not None:
             return cli
         toml_val = toml_cfg.get(toml_key)
         if toml_val is not None:
@@ -74,7 +77,10 @@ def resolve(opt_name, toml_key, default):
         signing_backend=resolve("--gpu-proof-signing-backend", "signing_backend", "ed25519"),
         policy_path=resolve("--gpu-proof-policy", "policy_path", None),
         required_marker=resolve("--gpu-proof-required-marker", "required_marker", "gpu_proof"),
-        fail_on_skip=bool(resolve("--gpu-proof-fail-on-skip", "fail_on_skip", False)),
+        # store_true flag: False just means "not passed", so OR with the toml
+        # value rather than sentinel-resolving (a CLI flag can only turn it ON).
+        fail_on_skip=bool(opt("--gpu-proof-fail-on-skip", False)
+                          or toml_cfg.get("fail_on_skip", False)),
         fingerprint_paths=paths,
         github_username=resolve("--gpu-proof-github-user", "github_username", None),
         max_age_days=max_age_days,

src/pytest_gpu_proof/merge.py

@@ -0,0 +1,161 @@
+"""
+Merge shard receipts from one commit into a single re-signed receipt.
+
+Motivation: large suites run their GPU tests as several pytest invocations
+(per-module crash isolation, machine sharding). Each invocation emits its own
+receipt; CI wants ONE artifact to verify. ``gpu-proof merge`` unions the shard
+receipts' ``tests`` and re-signs the result with the merger's local SSH key —
+so the merged receipt flows through the existing ``gpu-proof verify`` path
+completely unchanged (schema_version stays "1"; the only addition is the
+OPTIONAL ``session.shards`` provenance list, which the verifier ignores).
+
+Trust model (consistent with docs/security_model.md): the merged receipt is an
+attestation by the MERGER — shard signatures are recorded as provenance but are
+NOT verified here (merge is offline by design); the merged receipt's signature
+is what CI verifies. Merging refuses to combine shards that disagree on
+anything a receipt pins: schema, commit SHA, fingerprint, mode, or the
+environment the tests ran under. Duplicate node IDs are always an error — two
+shards attesting the same test is a sharding bug, not a merge input.
+"""
+
+import json
+import os
+from typing import List, Optional
+
+from .receipt import finalize_receipt, write_receipt
+
+
+class MergeError(ValueError):
+    """A refusal to merge, with an actionable message."""
+
+
+def load_receipt(path: str) -> dict:
+    try:
+        with open(path) as f:
+            data = json.load(f)
+    except (OSError, json.JSONDecodeError) as e:
+        raise MergeError(f"{path}: not a readable receipt ({e})") from e
+    if not isinstance(data, dict) or "tests" not in data:
+        raise MergeError(f"{path}: not a gpu-proof receipt (no 'tests' key)")
+    return data
+
+
+def _require_identical(receipts: List[dict], sources: List[str], getter, what: str):
+    values = [getter(r) for r in receipts]
+    first = values[0]
+    for src, val in zip(sources[1:], values[1:]):
+        if val != first:
+            raise MergeError(
+                f"shards disagree on {what}: {sources[0]}={first!r} vs {src}={val!r} — "
+                f"a merged receipt must come from ONE commit/config; re-run the "
+                f"divergent shard."
+            )
+    return first
+
+
+def merge_payloads(receipts: List[dict], sources: List[str]) -> dict:
+    """Union shard payloads into one UNSIGNED payload. Raises MergeError on any
+    disagreement over what a receipt pins."""
+    if len(receipts) < 1:
+        raise MergeError("nothing to merge")
+
+    _require_identical(receipts, sources, lambda r: r.get("schema_version"), "schema_version")
+    if receipts[0].get("schema_version") != "1":
+        raise MergeError(
+            f"unsupported schema_version {receipts[0].get('schema_version')!r} "
+            f"(this version merges schema '1' receipts)"
+        )
+    _require_identical(receipts, sources, lambda r: r.get("repo", {}).get("commit_sha"),
+                       "repo.commit_sha")
+    _require_identical(receipts, sources, lambda r: r.get("fingerprint"), "fingerprint")
+    _require_identical(receipts, sources, lambda r: r.get("mode"), "mode")
+    for key in ("python_version", "platform", "pytest_version", "plugin_version"):
+        _require_identical(receipts, sources,
+                           lambda r, k=key: r.get("environment", {}).get(k),
+                           f"environment.{key}")
+
+    # tests: union; duplicate node ids are a sharding bug, never silently deduped.
+    tests: List[dict] = []
+    seen: dict = {}
+    for src, r in zip(sources, receipts):
+        for t in r.get("tests", []):
+            nid = t.get("node_id")
+            if nid in seen:
+                raise MergeError(
+                    f"duplicate node_id {nid!r} in {src} (already attested by "
+                    f"{seen[nid]}) — shards must partition the suite."
+                )
+            seen[nid] = src
+            tests.append(t)
+    if not tests:
+        raise MergeError("merged receipt would contain zero tests")
+
+    sessions = [r.get("session", {}) for r in receipts]
+    started = min(s.get("started_at") for s in sessions)
+    ended = max(s.get("ended_at") for s in sessions)
+
+    merged = dict(receipts[0])
+    merged.pop("signature", None)
+    # repo: the commit SHA is asserted identical above; branch/remote come from
+    # the first shard. `dirty` is OR-ed — one dirty shard makes the merged
+    # attestation dirty, and the verifier's dirty policy then applies honestly.
+    merged["repo"] = dict(receipts[0].get("repo", {}))
+    merged["repo"]["dirty"] = any(r.get("repo", {}).get("dirty") for r in receipts)
+    # environment: fields asserted identical above; gpu_info from the first
+    # shard that has one (a CPU-only shard shouldn't erase the GPU record).
+    merged["environment"] = dict(receipts[0].get("environment", {}))
+    merged["environment"]["gpu_info"] = next(
+        (r["environment"]["gpu_info"] for r in receipts
+         if r.get("environment", {}).get("gpu_info")), None)
+    merged["tests"] = tests
+    merged["session"] = {
+        "started_at": started,
+        "ended_at": ended,
+        "node_ids": [t["node_id"] for t in tests],
+        # Provenance (additive; the verifier ignores unknown session keys).
+        # Shard signers are recorded, not verified — the merged signature is
+        # the attestation CI checks.
+        "shards": [
+            {
+                "source": os.path.basename(src),
+                "node_count": len(r.get("tests", [])),
+                "started_at": s.get("started_at"),
+                "ended_at": s.get("ended_at"),
+                "signer": (r.get("signature") or {}).get("signer"),
+            }
+            for src, r, s in zip(sources, receipts, sessions)
+        ],
+    }
+    return merged
+
+
+def merge_receipts(
+    paths: List[str],
+    out: str,
+    *,
+    github_user: Optional[str] = None,
+    key_path: Optional[str] = None,
+    unsigned: bool = False,
+) -> dict:
+    """Merge receipts at ``paths`` and write the re-signed result to ``out``.
+
+    ``github_user`` overrides the recorded signer identity (default: the first
+    shard's ``repo.github_username`` — correct when the merger is also the shard
+    runner). ``unsigned=True`` writes ``signature: null`` (verifies only with
+    ``--allow-unsigned``, loudly, same as the plugin's 'none' backend).
+    """
+    receipts = [load_receipt(p) for p in paths]
+    payload = merge_payloads(receipts, list(paths))
+    if github_user:
+        payload["repo"] = dict(payload.get("repo", {}))
+        payload["repo"]["github_username"] = github_user
+
+    if unsigned:
+        receipt = dict(payload)
+        receipt["signature"] = None
+    else:
+        from .signers.ed25519 import SSHSigner
+        signer = SSHSigner(key_path=key_path)
+        receipt = finalize_receipt(payload, signer)
+    write_receipt(receipt, out)
+    return receipt

src/pytest_gpu_proof/plugin.py

@@ -201,13 +201,13 @@ def pytest_addoption(parser):
     )
     group.addoption(
         "--gpu-proof-mode",
-        default="local",
+        default=None,
         choices=["local", "ci-gpu"],
         help="Execution mode: local (default) or ci-gpu",
     )
     group.addoption(
         "--gpu-proof-out",
-        default="gpu-proof.json",
+        default=None,
         metavar="PATH",
         help="Output path for the receipt JSON (default: gpu-proof.json)",
     )
@@ -219,7 +219,7 @@ def pytest_addoption(parser):
     )
     group.addoption(
         "--gpu-proof-signing-backend",
-        default="ed25519",
+        default=None,
         choices=["ed25519", "none"],
         help="Signing backend (default: ed25519 via SSH key)",
     )
@@ -231,7 +231,7 @@ def pytest_addoption(parser):
     )
     group.addoption(
         "--gpu-proof-required-marker",
-        default="gpu_proof",
+        default=None,
         help="Marker name that flags a test for the receipt (default: gpu_proof)",
     )
     group.addoption(
@@ -242,7 +242,7 @@ def pytest_addoption(parser):
     )
     group.addoption(
         "--gpu-proof-fingerprint-paths",
-        default="src,tests",
+        default=None,
         metavar="PATHS",
         help="Comma-separated paths to fingerprint (default: src,tests)",
     )