Skip to content

Read AFF4 disk images - #180

Merged
switch-nz merged 2 commits into
mainfrom
aff4-reader
Oct 2, 2026
Merged

switch-nz merged 2 commits into
mainfrom
aff4-reader

Conversation

@switch-nz

@switch-nz switch-nz commented Oct 1, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • Reads AFF4 (Advanced Forensic Format 4) disk images: a ZIP64 container whose image content is either a Map (a virtual address space over one or more ImageStreams) or a bare ImageStream, zlib/Deflate-compressed or stored, closing out the last of Read QCOW2, VDI, DMG and AFF4 images #47's four formats (QCOW2, VDI and DMG already shipped).
  • A damaged chunk or bevy index reads as zeros and is reported, not failing the whole image (same resilience EWF applies to a bad chunk). A striped or segmented multi-volume set, and any compression method other than Deflate, are refused with the reason.
  • Detection is by .aff4 extension plus interior markers (the Zip comment or a container.description member naming an aff4:// volume URI, and a parseable information.turtle) — never a generic Zip signature, which would misdetect (and then refuse) every ordinary Zip archive.

Two real discrepancies between the published spec and real output

Built from the AFF4 Standard v1.0a (github.com/aff4/Standard), but its prose doesn't match real files in two places — found by building real AFF4 files with pyaff4 (the reference implementation) and inspecting the bytes directly, not just trusting the text:

  • ImageStream bevy index: documented as an 8+4-byte (bevy offset, chunk size) struct. Real files use a 12-byte (offset, reserved, stored length) record per chunk.
  • Deflate compression method URI: documented against RFC1951 (raw DEFLATE). Real chunks are zlib-wrapped (RFC1950) — confirmed by a 0x78 0x9c zlib header on the actual compressed bytes.

A bug in pyaff4 itself was also found along the way (and worked around, not relied on): its AFF4Map.WriteStream() convenience path writes an incorrect target_offset, producing files its own reader can't read back. The test fixture builder computes correct values independently rather than reproducing that bug. Separately, real pyaff4 output omits the aff4:dependentStream property the standard says a Map should carry — the reader discovers ImageStreams from the whole graph instead of depending on that property.

Tests

tests/imagebuild_aff4.py is a from-scratch fixture builder, written independently of engine/aff4.py (same discipline as every other format fixture in this repo), covering:

  • Both image shapes (bare ImageStream, and DiskImage → Map → ImageStream)
  • Compressed and stored chunks, multiple bevies, partial/overlapping/boundary-crossing reads
  • Map gaps (default aff4:Zero fill and a custom symbolic fill stream), multiple disjoint ranges
  • Stored-hash verification (MD5/SHA1, matching and mismatched)
  • A genuine Zip64 container (EOCD64 + locator + 64-bit extra fields)
  • Refusals: missing information.turtle, missing volume URI, unsupported compression method, a striped/segmented volume (Map referencing a target not in this volume), malformed Turtle, not a Zip at all
  • Resilience: a corrupted compressed chunk and a malformed bevy index both zero-fill and report a finding rather than failing the read

CodeQL

One new alert (#119, py/path-injection on the new open(path, "rb") call
for AFF4 dispatch in engine/ewf.py) was dismissed as won't-fix: it is the
identical pattern as three already-dismissed alerts in this same file
(#84/#85/#86) — an examiner-chosen evidence path, opened the same way every
other format in open_image() already does, behind the loopback-gated API
(see SECURITY.md and #37).

Test plan

  • python -m unittest discover -s tests -p "test_*.py" — passes (two pre-existing, unrelated failures confirmed present on plain main too: a Windows path-normalization quirk in the RAID feature, and the known VHD memory-measurement flake)
  • python tests/smoke.py — all checks pass
  • No real AFF4 imaging tool's output has been checked against this yet (noted in docs/ROADMAP.md, marked [~] rather than [x])

🤖 Generated with Claude Code

AFF4 (Advanced Forensic Format 4) is a ZIP64 container whose image
content is either a Map (a virtual address space over one or more
ImageStreams) or a bare ImageStream. Both shapes are read, with
zlib/Deflate-compressed or stored chunks; a damaged chunk or bevy index
reads as zeros and is reported, rather than failing the whole image. A
striped or segmented multi-volume set, and any compression method other
than Deflate, are refused with the reason.

Built from the published AFF4 Standard v1.0a (github.com/aff4/Standard),
but its prose does not match real output in two places, found by
building real AFF4 files with pyaff4 (the reference implementation) and
inspecting the bytes directly:

- The ImageStream bevy index is documented as an 8+4-byte (bevy offset,
  chunk size) struct; real files use a 12-byte (offset, reserved,
  stored length) record per chunk.
- The Deflate compression method is documented against the RFC1951 (raw
  DEFLATE) URI; real chunks are zlib-wrapped (RFC1950).

A separate bug was found (and worked around, not relied on) in pyaff4
itself: its AFF4Map.WriteStream() convenience path writes an incorrect
target_offset, producing files its own reader cannot read back. The
test fixture builder computes correct values independently rather than
reproducing that bug. Also found empirically: real pyaff4 output omits
the aff4:dependentStream property the standard says a Map should carry;
the reader discovers ImageStreams from the whole graph instead of
relying on that property alone.

Detection is by .aff4 extension plus interior markers (the Zip comment
or a container.description member naming an aff4:// volume URI, and a
parseable information.turtle) -- never by a generic Zip signature, which
would misdetect (and then refuse) every ordinary Zip archive.

Tested against a from-scratch fixture builder (tests/imagebuild_aff4.py,
written independently of engine/aff4.py) covering both image shapes,
compression, gaps and symbolic fill streams, stored-hash verification,
Zip64, and resilience to a corrupted chunk or bevy index. No real AFF4
imaging tool's output has been checked against this yet.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@switch-nz switch-nz mentioned this pull request Oct 1, 2026
4 tasks
Comment thread engine/ewf.py
# matched by extension only; Aff4Image's own parsing then requires
# the interior AFF4 markers before accepting the file.
try:
return aff4_mod.Aff4Image(path, open(path, "rb"))
No code change. The "CodeQL" PR check is a github-advanced-security
alert-state snapshot tied to the commit SHA; re-running the same
workflow run against the already-analyzed commit doesn't refresh it,
and alert #119 (dismissed as won't-fix, matching #84/#85/#86's
established precedent in this same file) needs a new commit for a
fresh comparison to reflect that.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@switch-nz
switch-nz merged commit a71ffea into main Oct 2, 2026
31 of 32 checks passed
@switch-nz
switch-nz deleted the aff4-reader branch October 2, 2026 07:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants