Repository navigation
Read AFF4 disk images - #180
Merged
Merged
Conversation
AFF4 (Advanced Forensic Format 4) is a ZIP64 container whose image content is either a Map (a virtual address space over one or more ImageStreams) or a bare ImageStream. Both shapes are read, with zlib/Deflate-compressed or stored chunks; a damaged chunk or bevy index reads as zeros and is reported, rather than failing the whole image. A striped or segmented multi-volume set, and any compression method other than Deflate, are refused with the reason. Built from the published AFF4 Standard v1.0a (github.com/aff4/Standard), but its prose does not match real output in two places, found by building real AFF4 files with pyaff4 (the reference implementation) and inspecting the bytes directly: - The ImageStream bevy index is documented as an 8+4-byte (bevy offset, chunk size) struct; real files use a 12-byte (offset, reserved, stored length) record per chunk. - The Deflate compression method is documented against the RFC1951 (raw DEFLATE) URI; real chunks are zlib-wrapped (RFC1950). A separate bug was found (and worked around, not relied on) in pyaff4 itself: its AFF4Map.WriteStream() convenience path writes an incorrect target_offset, producing files its own reader cannot read back. The test fixture builder computes correct values independently rather than reproducing that bug. Also found empirically: real pyaff4 output omits the aff4:dependentStream property the standard says a Map should carry; the reader discovers ImageStreams from the whole graph instead of relying on that property alone. Detection is by .aff4 extension plus interior markers (the Zip comment or a container.description member naming an aff4:// volume URI, and a parseable information.turtle) -- never by a generic Zip signature, which would misdetect (and then refuse) every ordinary Zip archive. Tested against a from-scratch fixture builder (tests/imagebuild_aff4.py, written independently of engine/aff4.py) covering both image shapes, compression, gaps and symbolic fill streams, stored-hash verification, Zip64, and resilience to a corrupted chunk or bevy index. No real AFF4 imaging tool's output has been checked against this yet. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
4 tasks
| # matched by extension only; Aff4Image's own parsing then requires | ||
| # the interior AFF4 markers before accepting the file. | ||
| try: | ||
| return aff4_mod.Aff4Image(path, open(path, "rb")) |
No code change. The "CodeQL" PR check is a github-advanced-security alert-state snapshot tied to the commit SHA; re-running the same workflow run against the already-analyzed commit doesn't refresh it, and alert #119 (dismissed as won't-fix, matching #84/#85/#86's established precedent in this same file) needs a new commit for a fresh comparison to reflect that. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
.aff4extension plus interior markers (the Zip comment or acontainer.descriptionmember naming anaff4://volume URI, and a parseableinformation.turtle) — never a generic Zip signature, which would misdetect (and then refuse) every ordinary Zip archive.Two real discrepancies between the published spec and real output
Built from the AFF4 Standard v1.0a (github.com/aff4/Standard), but its prose doesn't match real files in two places — found by building real AFF4 files with pyaff4 (the reference implementation) and inspecting the bytes directly, not just trusting the text:
(bevy offset, chunk size)struct. Real files use a 12-byte(offset, reserved, stored length)record per chunk.0x78 0x9czlib header on the actual compressed bytes.A bug in pyaff4 itself was also found along the way (and worked around, not relied on): its
AFF4Map.WriteStream()convenience path writes an incorrecttarget_offset, producing files its own reader can't read back. The test fixture builder computes correct values independently rather than reproducing that bug. Separately, real pyaff4 output omits theaff4:dependentStreamproperty the standard says a Map should carry — the reader discovers ImageStreams from the whole graph instead of depending on that property.Tests
tests/imagebuild_aff4.pyis a from-scratch fixture builder, written independently ofengine/aff4.py(same discipline as every other format fixture in this repo), covering:aff4:Zerofill and a custom symbolic fill stream), multiple disjoint rangesinformation.turtle, missing volume URI, unsupported compression method, a striped/segmented volume (Map referencing a target not in this volume), malformed Turtle, not a Zip at allCodeQL
One new alert (#119,
py/path-injectionon the newopen(path, "rb")callfor AFF4 dispatch in
engine/ewf.py) was dismissed as won't-fix: it is theidentical pattern as three already-dismissed alerts in this same file
(#84/#85/#86) — an examiner-chosen evidence path, opened the same way every
other format in
open_image()already does, behind the loopback-gated API(see SECURITY.md and #37).
Test plan
python -m unittest discover -s tests -p "test_*.py"— passes (two pre-existing, unrelated failures confirmed present on plainmaintoo: a Windows path-normalization quirk in the RAID feature, and the known VHD memory-measurement flake)python tests/smoke.py— all checks passdocs/ROADMAP.md, marked[~]rather than[x])🤖 Generated with Claude Code