D20: find the partitions eight short AKAI images displaced

CHANGELOG.md

@@ -4,6 +4,35 @@ Notable changes to `samplerdisc`. Format-level findings live in [docs/formats/](
 
 ## Unreleased
 
+### Added
+
+- **Eight AKAI images are short of the disc they were made from, and their displaced partitions are now read.** Whole 32 KB blocks are missing from these `.mdx` files — the container decodes every block it holds, and the file is not a complete copy of the disc — so every partition after a gap sits that much nearer the front than the disk's own partition table places it. **39 partitions are found there**, carrying **432 volumes, 17 180 files and 15 808 samples** that nothing could reach before. The collection goes from 89 156 samples to **104 921**, and AKAI from 56 425 to **72 190**. ([docs/formats/akai-fs.md](docs/formats/akai-fs.md), [ADR-0028](docs/adr/0028-a-displaced-partition-is-anchored-quantised-and-floored.md), [#25](https://github.com/bmxcode/samplerdisc/issues/25))
+
+  | Disc | Recovered | Volumes | Files | Samples | Written |
+  |---|---:|---:|---:|---:|---:|
+  | `AMG - Kickin' Lunatic Beats 2 CD1` | 7 | 121 | 7 723 | 7 308 | 7 293 |
+  | `AMG - Kickin' Lunatic Beats 2 CD2` | 7 | 100 | 6 203 | 5 912 | 5 912 |
+  | `AKAI.S3000.Sound.Library.6` | 7 | 84 | 1 054 | 955 | 955 |
+  | `Back In Time Records - Elektra Vox` | 5 | 33 | 661 | 463 | 463 |
+  | `Audio Factory - Classical Wild Takes` | 4 | 18 | 189 | 80 | 80 |
+  | `AKAI.S3000.Sound.Library.5` | 3 | 34 | 473 | 400 | 377 |
+  | `AKAI.S3000.Sound.Library.7` | 3 | 20 | 660 | 546 | 546 |
+  | `AMG - Global Trance Mission 2` | 3 | 22 | 217 | 144 | 139 |
+
+  **The audio verifies, and that is what made this safe to do at all.** 15 765 of the 15 808 recovered samples — 99.7 % — carry a payload header whose id, valid flag and name agree with the directory entry that placed them, checked by a structure the search never consults. A gap removes whole blocks, so everything past one moves by a *constant*: inside a displaced partition the directory and its audio moved together and stay consistent. Five of the eight discs recover thousands of files with **not one refusal between them**.
+
+  **The search is anchored, quantised and floored, which is what separates it from the scan [ADR-0023](docs/adr/0023-partitions-come-from-the-table-the-disc-declares.md) refused.** It walks *backwards from the position the table gives that partition* — which also settles which partition it is, since sizes repeat across a disk; it steps in *the unit the container states it stores the disc in*, new as `SectorImage.granularity`, 32 768 bytes on these images; and it *stops at the end of the partition already accepted*, so nothing is ever found inside a partition already being read. Partition 1 therefore cannot move on any disc — its declared position is 0 — and every count pinned since D15 is unchanged on the other 36 discs.
+
+  **70 declared partitions stay unread and 60 of them are not absent**, which is the deliberate cost. A header is there, inside a partition already being read: 31 would overlap one, 29 land exactly on one. `Best Service - Alpha Dance I` is the whole disc's worth of that and recovers nothing. The remaining 10 have no header at any position the search may look at.
+
+  **ADR-0023 was right on the evidence it had, and one piece of that evidence turns out to be something else.** It refused a signature scan partly because the header's constant field is a sawtooth that audio reproduces — 374 matching blocks on `Global Trance Mission 2`, 153 on `ProSamples vol.14`. Digested, `vol.14`'s 153 are its five real partition headers plus **one 8192-byte block repeated 148 times, byte for byte**; audio does not do that. They are complete stale partition headers sitting in free space, with volume directories that parse. There is no byte test that separates those from a real header, because they are real headers — so the refusal stands and is firmer than it was.
+
+  **The payload name check still has no unique positives, and this was the deliverable expected to give it one.** It fires 104 times across the collection now and never once without the id and valid tests firing too — the 15 808 recovered samples included. A displacement landing exactly on another sample's header is the case it exists for, and 15 808 more displaced samples produced none of them. ([ADR-0027](docs/adr/0027-a-payload-must-be-the-file-its-entry-placed.md))
+
+  **`AKAI.S3000.Sound.Library.6`'s recovered partition 5 brings 18 stereo pairs whose halves declare different rates** — `PROPHE FX1` at 44 597 against 44 100, and the like. The joiner refuses to fuse a pair the disc disagrees with itself about and writes both mono halves, so nothing is lost; it is why the run's "skipped (damage)" line goes from 5 to 23 while no audio went missing.
+
+- **A short image says so.** `list` prints the displacement under each partition heading and in its layout line — *"11 partitions declared, 8 present in this image (7 of them displaced -- this image is short of the disc it was made from)"* — and `batch` records that line per disc in the manifest, as `layout`. Before this, a disc missing ten of its eleven partitions said only how many were present. ([ADR-0028](docs/adr/0028-a-displaced-partition-is-anchored-quantised-and-floored.md))
+
 ### Fixed
 
 - **S3000 AKAI samples had 42 bytes of header at the front of their audio.** The S3000 family writes a **192-byte** header where the S1000 family writes 150, and every sample was read at 150. That does not fail — the frame count comes out right, the WAV opens, the length is within 0.1 % — so what shipped was a file beginning with a burst of roughly ±20 000 lasting 0.24 ms in place of the attack, missing the last 21 frames of the sound, with every loop point 21 frames out of alignment. **13 451 of the collection's 56 490 AKAI samples**, on nine discs: `AKAI.S3000.Sound.Library.1`–`7` (4 455, 3 086, 1 990, 1 010, 601, 168 and 218), `East Connexion Piano` (730) and `AMG - Now CD-Rom for (AKAI)` (1 193). **Anyone who extracted those discs should do it again.** ([docs/formats/akai-fs.md](docs/formats/akai-fs.md), [ADR-0027](docs/adr/0027-a-payload-must-be-the-file-its-entry-placed.md))

README.md

@@ -78,32 +78,36 @@ Compressed `.mdx` is the piece no other open-source tool reads today. The format
 | | |
 |---|---|
 | Discs converted | 72 of 79 |
-| Samples | 89 156 |
-| Stereo pairs rejoined | 15 808 |
+| Samples | 104 921 |
+| Stereo pairs rejoined | 17 309 |
 | Audio CD tracks | 161 |
 | Duplicate audio suppressed | 5 719 |
-| Entries not the file their entry placed | 61 |
-| Entries skipped (damage) | 5 |
-| Time | 48 s |
+| Entries not the file their entry placed | 104 |
+| Entries skipped (damage) | 23 |
+| Time | 72 s |
 
 By filesystem:
 
 | | Discs | Samples | Stereo pairs | Skipped |
 |---|---:|---:|---:|---:|
-| AKAI | 44 | 56 425 | 14 461 | 66 |
+| AKAI | 44 | 72 190 | 15 962 | 127 |
 | E-mu `EMU3` | 7 | 14 738 | 6 | 0 |
 | ISO 9660 | 15 | 11 601 | — | 0 |
 | Roland `S770 MR25A` | 5 | 6 392 | 1 341 | 0 |
 
 "Stereo pairs" counts files joined from an `-L`/`-R` pair by name. E-mu's six are the only ones on those discs, and they are a different and much rarer thing than the 2 656 samples whose record declares two channels.
 
-Every WAV was checked against the disc it came from — **70 of 70 discs match exactly**, comparing multisets of SHA-256 over the PCM per disc rather than going via filenames, so duplicate names cannot mask a mismatch and no path is guessed. 89 156 payloads, zero mismatches. The E-mu stereo samples are compared with their channels put back the way the disc stored them, since their WAV holds the same bytes interleaved; `tests/test_discs.py` asserts that de-interleaving reproduces the disc's two blocks exactly, per sample, on all seven discs. The two audio CDs are not in that count: their tracks are cut from a stream by a cue, so there is no run of bytes on the disc to compare a track against.
+Every WAV was checked against the disc it came from — **70 of 70 discs match exactly**, comparing multisets of SHA-256 over the PCM per disc rather than going via filenames, so duplicate names cannot mask a mismatch and no path is guessed. 104 921 payloads, zero mismatches. The E-mu stereo samples are compared with their channels put back the way the disc stored them, since their WAV holds the same bytes interleaved; `tests/test_discs.py` asserts that de-interleaving reproduces the disc's two blocks exactly, per sample, on all seven discs. The two audio CDs are not in that count: their tracks are cut from a stream by a cue, so there is no run of bytes on the disc to compare a track against.
 
-105 125 WAV files were written in all — the samples, the stereo joins and the audio CD tracks. None is unreadable and none is zero-length. **275 are silent for their whole length, and every one of them matches the disc exactly**: 267 are the blank `15G-KIT…Z` slots on `ProSamples vol.15`, six are on a Proteus library that ships `Dead Air` as a sample, and two are on a Roland disc. That is what the discs hold, not something the decoder did.
+122 391 WAV files were written in all — the samples, the stereo joins and the audio CD tracks. None is unreadable and none is zero-length. **275 are silent for their whole length, and every one of them matches the disc exactly**: 267 are the blank `15G-KIT…Z` slots on `ProSamples vol.15`, six are on a Proteus library that ships `Dead Air` as a sample, and two are on a Roland disc. That is what the discs hold, not something the decoder did.
 
-Sample rates run from 6 000 to 49 999 Hz across 1 047 distinct values. The odd ones are real — E-mu writes rates like 24 444 and 27 778, and AKAI uses 33 075 (¾ of 44 100) and 29 400 (⅔) to trade bandwidth for memory. They are carried through exactly as the disc states them and never rounded.
+Sample rates run from 6 000 to 49 999 Hz across 1 071 distinct values. The odd ones are real — E-mu writes rates like 24 444 and 27 778, and AKAI uses 33 075 (¾ of 44 100) and 29 400 (⅔) to trade bandwidth for memory. They are carried through exactly as the disc states them and never rounded.
 
-The 66 AKAI entries not written are two different faults. **61 are payloads that are not the file the directory placed there** — their header carries another file's id, valid flag or name — and 60 of those 61 are a run to the end of one volume, on ten discs; the other 34 AKAI discs have none. That is what a rip losing a run of blocks looks like from inside a directory, and it does not need a partition to go missing: `Best Service - Alpha Dance II` declares six partitions and holds all six, and still loses 21 of `AC.DRUMLOOPS`'s 22 samples. Each is refused with a line naming every field that disagrees and the entry that placed it, rather than being written out under a name that is not its own ([ADR-0027](docs/adr/0027-a-payload-must-be-the-file-its-entry-placed.md)). The remaining five are damage of a different kind: four samples whose header is otherwise perfect and whose rate field reads 0, 519, 519 or 1280, and one stereo pair whose halves declare 44 033 and 44 100, so the joiner refuses to fuse them and writes both mono halves instead.
+The 127 AKAI entries not written are two different faults. **104 are payloads that are not the file the directory placed there** — their header carries another file's id, valid flag or name — and 103 of those 104 are a run to the end of one volume, on nine discs; the other 35 AKAI discs have none. That is what a rip losing a run of blocks looks like from inside a directory, and it does not need a partition to go missing: `Best Service - Alpha Dance II` declares six partitions and holds all six, and still loses 21 of `AC.DRUMLOOPS`'s 22 samples. Each is refused with a line naming every field that disagrees and the entry that placed it, rather than being written out under a name that is not its own ([ADR-0027](docs/adr/0027-a-payload-must-be-the-file-its-entry-placed.md)). Every one of those 103 is sitting intact, under its own name, a whole number of container blocks earlier in the image — the same fault as the displaced partitions below, one level down, and not yet recovered ([#35](https://github.com/bmxcode/samplerdisc/issues/35)).
+
+The other 23 are damage of a different kind, and 19 of them are not damage at all: four samples whose header is otherwise perfect and whose rate field reads 0, 519, 519 or 1280, and **19 `-L`/`-R` pairs whose two halves declare different sample rates** — 18 of them synth FX on `AKAI.S3000.Sound.Library.6`, at 44 597 against 44 100 and the like. The joiner refuses to fuse a pair the disc disagrees with itself about, and writes both mono halves instead, so no audio is lost.
+
+**Eight of the 44 AKAI images are short of the disc they were made from.** Whole 32 KB blocks are missing from the file, so every partition after a gap sits that much nearer the front than the disk's own table says — and 39 of them are found there and read, carrying 17 180 files and 15 808 samples that nothing could reach before. `list` and the manifest say so per disc: *"11 partitions declared, 8 present in this image (7 of them displaced — this image is short of the disc it was made from)"*. 70 declared partitions are still unread, and 60 of those are refused rather than absent: a header is there, inside a partition already being read, and reading it would put one run of bytes under two partitions at once ([ADR-0028](docs/adr/0028-a-displaced-partition-is-anchored-quantised-and-floored.md)).
 
 The seven that do not convert are accounted for: one S-550 disc present as both `.iso` and `.nrg` — a different format from the S-7xx and not yet read ([ADR-0014](docs/adr/0014-one-backend-per-on-disc-format.md)) — two Digidesign SampleCell discs, one audio CD with no cue sheet present as both `.mdx` and `.cdr`, and one ISO 9660 disc holding E-mu `.EBL` banks rather than audio.