Cross-check Strata against independent readers (synthetic corpus)

.github/workflows/crosscheck.yml

@@ -0,0 +1,44 @@
+name: Cross-check
+
+# Reads the synthetic corpus with Strata and with independent readers (the
+# libyal libraries, qemu-img) and fails on any difference that
+# tests/crosscheck_baseline.json does not explain. See docs/CROSSCHECK.md.
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    paths:
+      - "engine/**"
+      - "tests/crosscheck*"
+      - "tests/imagebuild_*.py"
+      - ".github/workflows/crosscheck.yml"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+concurrency:
+  group: crosscheck-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  crosscheck:
+    name: Against reference readers
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      - uses: actions/setup-python@v7
+        with:
+          python-version: "3.12"
+      - name: Install the reference readers
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y qemu-utils
+          python3 -m pip install libewf-python libvhdi-python libqcow-python \
+            libvmdk-python libvsgpt-python libfsntfs-python libfshfs-python \
+            libfsext-python libfsfat-python
+      - name: List what is installed
+        run: python3 tests/crosscheck.py --list
+      - name: Compare
+        run: python3 tests/crosscheck.py --no-save

.gitignore

@@ -45,3 +45,6 @@ desktop.ini
 # Native sidecar build output and locally installed test binaries
 native/target/
 engine/native/*/
+
+# Output of tests/crosscheck.py; may describe real evidence.
+/crosscheck-out/

CHANGELOG.md

@@ -13,6 +13,14 @@ records, not how the code changed.
 
 ### Added
 
+- A repeatable comparison of Strata against independent readers
+  (`tests/crosscheck.py`; libewf, libvhdi, libqcow, libvmdk, libvsgpt,
+  libfsntfs, libfshfs, libfsext, libfsfat and qemu-img where installed). It
+  reads the same image both ways, layer by layer, and lists every difference;
+  the ones that are understood are recorded with their reasons and a new one
+  fails the run. It can also be pointed at real images, with file and folder
+  names withheld from its output. Synthetic images only so far; see
+  `docs/CROSSCHECK.md`.
 - **AFF4 disk images can be opened.** Both the single-stream and the more
   common Map-over-ImageStream shape are read, with zlib/Deflate-compressed
   or stored chunks; a gap in a Map (unallocated space) reads as zeros, a

docs/CROSSCHECK.md

@@ -0,0 +1,70 @@
+# Cross-checking Strata against other readers
+
+`tests/crosscheck.py` reads the same image with Strata and with independent
+readers and reports every place they disagree. It exists because passing
+Strata's own tests shows that Strata reads what Strata's authors expected, not
+that it reads what other tools read. A disagreement is a lead, not a verdict:
+either reader may be the one that is wrong.
+
+## What is compared
+
+Layer by layer, each side given the same input:
+
+| Layer | Strata | Reference |
+|---|---|---|
+| Disk container (EWF, VHD, QCOW2, VMDK, VDI, DMG) | the engine's reader | libewf, libvhdi, libqcow, libvmdk, qemu-img |
+| Partition table (GPT) | `volume.scan` | libvsgpt |
+| Filesystem tree (NTFS, FAT, ext, HFS+) | the walker search and hashing share | libfsntfs, libfsfat, libfsext, libfshfs |
+
+Containers are compared by size and by a digest of each 1 MiB chunk (all of
+them, or with `--chunks N` the first and last few and a seeded random sample,
+so a run can be repeated). Filesystems are compared by path, kind, size, a
+digest of the first `--hash-bytes` of content, and extra streams (NTFS
+alternate data streams, the HFS+ resource fork). Filesystem readers are given
+Strata's own view of the partition, so a container difference cannot show up
+as a filesystem one.
+
+The references are optional. Each is used only if its Python package (or
+program) is installed, and is reported as `missing` otherwise; `--list` says
+what is installed and how to get the rest. Strata itself needs only the
+standard library.
+
+## Running it
+
+    python3 tests/crosscheck.py                    synthetic corpus
+    python3 tests/crosscheck.py --list             readers and cases
+    python3 tests/crosscheck.py --case 'hfsplus*'  some cases
+    python3 tests/crosscheck.py --images A.E01 ... real images
+    python3 tests/crosscheck.py --strict           also fail on stale baseline entries
+
+The exit status is 1 when a difference is not in the baseline. CI runs it on
+every change to the engine (`.github/workflows/crosscheck.yml`).
+
+## The baseline
+
+`tests/crosscheck_baseline.json` lists the differences that are understood,
+each with a case, a facet (`container`, `volumes`, `fs`), a kind (`only_strata`,
+`only_ref`, `differs`, `error`), a key pattern and the reason it is accepted.
+An entry without a reason is refused. Many entries are about the synthetic
+images rather than the readers: a reference reader that refuses a fixture
+(because the builder leaves out something real writers always write) is
+recorded as an `error`, so the gap stays on the list instead of being quietly
+skipped.
+
+Add an entry only after finding out why the two disagree. If Strata is wrong,
+fix Strata and leave the baseline alone.
+
+## Real images
+
+`--images` runs the same comparison over images you supply. File and folder
+names are evidence, so for these they are replaced in everything printed and
+saved by short stable tokens (`--names` turns that off); nothing about a real
+image is written to the baseline, which refuses entries for anything but
+synthetic cases; and results are saved to `crosscheck-out/`, which git
+ignores. Review what you share from a real image before sharing it.
+
+## Not yet covered
+
+Encryption (BitLocker, LUKS), artefact parsers (event logs, registry, mail,
+shortcuts, prefetch), AFF4, APFS and exFAT have no reference wired in, and
+the ext4 fixture is not yet compared (see its baseline entry).

docs/ROADMAP.md

@@ -276,9 +276,10 @@ Agreeing with another tool is agreement, not verification.
 
 ## Open questions
 
-- **Independent validation.** None of the parsers has been validated against
-  reference tooling on shared evidence; a repeatable comparison against an
-  established tool is the candidate before any new filesystem.
+- **Independent validation.** `tests/crosscheck.py` now compares containers,
+  GPT and four filesystems against established readers on the synthetic
+  corpus (see `docs/CROSSCHECK.md`). Still open: running it over real images,
+  and references for encryption, artefacts, APFS and exFAT.
 - **Fuzz corpus growth.** The saved corpus is small; a standing fuzz job with
   coverage feedback would find the next #15 before a user does.