Repository navigation
Parser bugs that return wrong results without warning (expected-failure tests from #13) #19
Description
The expected-failure tests added in #13 document engine bugs that, until now, were listed only in that PR's description. This issue tracks them so they can be linked from the release notes.
The ones that matter most are in the first section: they give an examiner wrong or missing results with no warning. Crashes and hangs are loud by comparison, and most are already in #15.
Every item has an @unittest.expectedFailure test. When a bug is fixed, its test starts passing and unittest reports an unexpected success, which fails the run — so the fix PR must remove the decorator in the same change, and should tick the box here. All 17 are currently expected failures on main (297 tests, OK).
Silent wrong results
These produce plausible output. Nothing in the interface indicates a problem.
- FAT: deleted long filenames come out scrambled.
fat.pysorts LFN parts by sequence number, but deletion overwrites every sequence byte with0xE5, so a multi-part deleted name is assembled in on-disk (reverse) order, e.g.ent.txtRemoved docum.
tests/test_fs_fat.py::test_deleted_multi_part_lfn_is_in_order - FAT: slack for a deleted multi-cluster file is reported in the wrong place.
slack()walks the freed FAT chain, so it reports slack inside the first cluster, overlapping the contentread_file()has just returned, rather than after it.
tests/test_fs_fat.py::test_deleted_multi_cluster_file_slack_follows_its_content - exFAT: timestamps ignore the UTC offset but are labelled UTC. The offset bytes (22–24) are never applied, yet the result carries
Z. 10:00 recorded at +12:00 is shown as 10:00Z; it is 22:00 UTC the previous day.
tests/test_fs_exfat.py::test_timestamp_honours_utc_offset - exFAT: contiguous (NoFatChain) subdirectories are listed only as far as their first cluster.
listdir()always walks the FAT, whose entries are zero for such a directory, so files in later clusters are silently missing from the listing.
tests/test_fs_exfat.py::test_contiguous_subdirectory_second_cluster_listed - ext4: inline files over 60 bytes return the wrong bytes. Data beyond 60 bytes is read from
raw[128:]instead of thesystem.dataxattr (ext4.py:296).
tests/test_fs_ext4.py::test_inline_data_spilling_into_xattr - ext4: inline directories are misparsed. Parsed as ordinary dirents, ignoring the 4-byte parent-inode prefix (
ext4.py:311).
tests/test_fs_ext4.py::test_inline_directory - ext4: sparse files lose their holes. The extent's logical start (
ee_block) is dropped, so holes vanish and every later extent shifts to a lower offset. The file reads back with the right bytes at the wrong offsets (ext4.py:261-271).
tests/test_fs_ext4.py::test_sparse_extent_hole_reads_as_zeros - EWF: a truncated compressed chunk is returned short, with no finding. A zlib stream that ends early is accepted (
ewf.pywithinflate.py). This undercuts the README's statement that a failing chunk is reported rather than silently returned.
tests/test_ewf.py::test_truncated_compressed_chunk_is_reported - Raw: a split raw set opens as its first segment only, with no finding.
open_imagehands.001toRawImage, which opens that one file. The examiner gets a partial disk and no indication of it. The README claims split raw sets are supported; that claim is corrected separately.
tests/test_ewf.py::test_split_raw_set
Fails on valid evidence
- EWF: any file with a three-letter extension beside the image is treated as a segment. The
[A-Z]{3}alternative in the segment pattern (ewf.py:58-61, case-insensitive) makescase.txtnext tocase.E01part of the set, which breaks opening.
tests/test_ewf.py::test_unrelated_sidecar_file_is_not_a_segment
Crashes and hangs
For completeness, so every expected failure is tracked somewhere.
| Bug | Test | Tracked |
|---|---|---|
exFAT run list unbounded by the cluster heap (MemoryError) |
test_fs_exfat.py::test_contiguous_run_bounded_by_cluster_heap |
#15 (1) |
| NTFS record count unbounded by volume size (hang) | test_fs_ntfs.py::test_record_count_bounded_by_volume |
#15 (3) |
NTFS invalid record cached, then AttributeError on reuse |
test_fs_ntfs.py::test_invalid_record_stays_invalid_on_second_lookup |
#15 (3) |
| ext4 self-referencing extent index: depth is bounded but fan-out is not, so reads can effectively hang | test_fs_ext4.py::test_self_referencing_extent_index_is_bounded |
here |
ext4 reading a deleted file when the journal is unreadable: the stand-in has no read_recovered() (AttributeError) |
test_fs_ext4.py::test_deleted_file_read_with_unreadable_journal |
here — distinct from the jbd2 block_size bug in #15 (2) |
Registry: a regf file under 48 bytes raises struct.error (registry.py:47-54) |
test_registry.py::test_truncated_base_block |
here |
Registry: carve_deleted on a truncated hive raises struct.error (registry.py:268-276) |
test_registry.py::test_truncated_hive_bins_carve |
here |
The fuzz run on #14 also hit the first registry bug, surfacing at registry.py:28 (the _u32 helper it calls).
Suggested priority
The silent-result bugs first, and within them the ones that change what an examiner would conclude: exFAT timestamps, sparse ext4 files at the wrong offsets, split raw sets loading as a partial disk, and the EWF short chunk. A crash tells the examiner something went wrong; these do not.
Activity
Link the wrong-result bugs from the 0.1.0 known issues
switch-nz commentedon Sep 17, 2026
From the Crashes and hangs table above: "ext4 reading a deleted file when the journal is unreadable" is fixed by #25 (
_Absentgainedread_recovered();test_deleted_file_read_with_unreadable_journalis no longer an expected failure). The silent wrong-result bugs in the first section are unchanged.exfat: apply the timestamp UTC offset
switch-nz commentedon Sep 17, 2026
Progress since this was opened:
Unchanged: sparse ext4 offsets, ext4 inline data and inline directories, deleted FAT names and slack, contiguous exFAT subdirectories, truncated EWF chunks, split raw sets, the EWF sidecar-as-segment bug, and the two registry crashes.
Add docs/ROADMAP.md with a roadmap diagram
switch-nz commentedon Sep 17, 2026
Working through the remaining items now, one PR per fix, starting with the ones that give silent wrong results: sparse ext4 extents first, then the truncated EWF chunk, then deleted FAT names and slack. I'll note here as each is picked up, so nothing is done twice.
switch-nz commentedon Sep 17, 2026
Sparse ext4 files: fixed by #30. Holes now read as zeros in place, both between extents and at the end of a file. A corrupt extent root still yields no runs, and directories get no holes; each guard is pinned by a test. Fuzz on the PR: 211,871 cases, zero findings.
Next: truncated compressed EWF chunk returned without a finding.
switch-nz commentedon Sep 17, 2026
Truncated compressed EWF chunk: fixed by #31. A chunk whose stream stops early is now reported; the part that decoded is kept and the rest zero-filled, so reads carry on past it at the right offsets. A damaged stream stays zero-filled rather than trusting output decoded before the error. Fuzz: 125,147 cases, zero findings.
Two follow-ups, same pattern, not yet fixed:
Neither has a test image to prove a change against yet.
Next: deleted FAT long filenames and slack.
switch-nz commentedon Sep 17, 2026
Fixed in #32: deleted FAT long filenames made of several parts are now assembled in order. Deletion overwrites every sequence byte with
0xE5, so the parts are joined in reverse on-disk order instead.In review: #33, slack and runs for deleted multi-cluster FAT files.
Next: exFAT contiguous subdirectory listing.
switch-nz commentedon Sep 17, 2026
Fixed in #33: slack and runs for a deleted FAT file larger than one cluster now follow the clusters its content is read from, not the freed one-cluster chain.
In review: #34, exFAT contiguous (NoFatChain) directories listed past their first cluster.
Found while doing that one: exFAT
entry_offsetpoints to the byte after an entry set instead of its File entry, so the hex view opens past the entry. I'll fix that separately.Next: ext4 inline data spilling into the xattr, and inline directories.
switch-nz commentedon Sep 17, 2026
Fixed in #34: exFAT contiguous (NoFatChain) directories are now listed past their first cluster.
In review: #35, ext4 inline data past 60 bytes and inline directories.
Next: EWF sidecar file mistaken for a segment.
switch-nz commentedon Sep 17, 2026
Fixed in #35: ext4 inline data past 60 bytes is read from the
system.dataxattr, and inline directories are listed.In review: #36, sidecar files such as
case.txtno longer join an EWF segment set.Next: split raw sets (
.001,.002, …).switch-nz commentedon Sep 17, 2026
Fixed in #36: a sidecar file such as
case.txtno longer joins an EWF segment set. Lettered segment names count only after.E99, and only when the file carries an EWF signature.In review: #37, split raw sets opened as one disk, with findings for a missing or wrongly sized piece.
Next: self-referencing ext4 extent index, then the two registry truncated-hive crashes.
switch-nz commentedon Sep 17, 2026
Fixed in #37: split raw sets (
.001,.002, …) now open as one disk, with findings for a missing or wrongly sized piece.Fixed in #38: ext4 extent tree traversal is bounded. Depth must decrease by one per level and no index block is read twice, so a self-referencing index no longer hangs.
In review: #39, the two registry crashes on truncated hives.
After that the only #19 item left is a design decision: exFAT entries that record no valid UTC offset are shown unadjusted but labelled UTC. The VMDK and AD1 truncated-stream follow-ups still need test images.
switch-nz commentedon Sep 17, 2026
Fixed in #39: a registry hive cut short no longer raises. One ending inside its base block opens with a finding and no keys, and deleted-cell carving stops at the end of the data.
That completes every code item in this issue. The unit suite on
mainhas no expected failures left.Still open here:
switch-nz commentedon Sep 17, 2026
Fixed in #40: exFAT times with no valid UTC offset are no longer marked or shown as UTC. They appear exactly as recorded, with no zone label, no conversion and no finding, so the analyst decides which zone they belong to. Plain FAT timestamps and shellbag DOS times, which never record a zone, are handled the same way.
The only item left here is the VMDK and AD1 truncated-stream follow-up, which is waiting on test images.
switch-nz commentedon Sep 17, 2026
Fixed in #87 (built on #41's new VMDK/AD1 test images): the VMDK header and grain-table bugs the new fuzz targets found — five unchecked fields (descriptor offset/size, grain size, entries-per-table, and a stream-optimized grain's declared compressed size) that could crash, exhaust memory, or in one case stall a full read for minutes. Also fixes the grain-truncation report gap from #41 itself. Replayed all 19 original CI findings plus a fresh 75,000-case run afterwards: clean.
Remaining: three AD1 chunk-handling bugs from #41 (a truncated chunk shifts every later chunk instead of staying in place, same for ranged reads, and a chunk that won't inflate at all is served as its own compressed bytes). Working on those next.
switch-nz commentedon Sep 17, 2026
Fixed in #88: AD1 chunks now keep their own place when one is truncated or fails to decompress — previously it shifted every later chunk (or, worse, served raw compressed bytes as content). Same fix applied to hashing.
That closes out every code item raised in this issue: FAT, exFAT, ext4, EWF, VMDK, AD1 and registry are all fixed and merged, and the unit suite on
mainhas no known failures left. The exFAT-UTC item was resolved as a deliberate choice (#40): unlabelled times, left for the analyst.Closing this issue.
Merge pull request #89 from switch-nz/update-docs-post-19
Release 0.2.0 (#114)