compact_pro

compact_pro is a clean-room, cross-platform Compact Pro (.cpt) implementation.

• Pure Zig core for archive/container/codec logic.
• C ABI (FFI) for embedding.
• C CLI named compact-pro for all filesystem and platform I/O.

CI Coverage

• Mechatron Prime CI is the project's continuous-integration service: a self-hosted Nix build worker that builds every push to yolo. It builds the flake attributes listed in .mechatron-prime/targets:
  • Linux x86_64: full unit and CLI test suite, plus a release build.
  • Linux aarch64, Windows x86_64 and Windows aarch64: cross-compiled release builds.
• GitHub Actions (.github/workflows/ci.yml) runs on native runners:
  • Linux x86_64, Linux aarch64 and macOS aarch64: Nix-based full test suite and release build.
  • Windows x86_64: Zig unit tests, release build and a CLI smoke test.
  • Windows aarch64: cross-compiled build artifact (not executed).

Build

./build

Debug build:

./build --debug

Run full test suite:

./test

Run benchmark suite (speed + compression performance):

./bm

CLI

compact-pro

Usage:
  compact-pro compress [--sidecar|--rsrc <path>|--xattr <name>] [--progress|--no-progress] [--force|-f] [-o <archive.cpt|->] <file...|->
  compact-pro expand [--sidecar|--rsrc <path>|--xattr <name>] [--progress|--no-progress] <archive.cpt> [-d <outdir>] [--path <entry> ...]
  compact-pro add [--sidecar|--rsrc <path>|--xattr <name>] [--progress|--no-progress] <archive.cpt> <file...>
  compact-pro list <archive.cpt>
  compact-pro verify <archive.cpt|->   (bounded-memory integrity check)
  compact-pro --help

Command Details

• compress

  • Creates a new .cpt archive from input files (or stdin via -).
  • Directory inputs are recursively expanded; archive paths are rooted at the input directory basename.
  • Empty directories are captured via metadata trailer records and recreated on extraction.
  • Default output behavior is no-clobber: if target archive exists, command fails.
  • Use --force (-f) to overwrite an existing output archive.
  • Prints completion stats to stderr: input bytes, compressed bytes, compressed percent, MB/s throughput, elapsed seconds.
  • Progress:
    • --progress forces progress output to stderr.
    • --no-progress disables progress output.
    • default is progress on TTY stderr.
    • live output renders phase bars with percent and ETA, e.g. [=====>-----] 42% (ETA: 35s).
  • -o is optional; default output is derived from filename (or cwd for multi-input).
  • If -o is provided without .cpt, .cpt is appended automatically.
  • -o - writes archive bytes to stdout.
  • ~-prefixed paths are expanded to home directory.
  • Preserves relative input paths as archive paths (for directory-aware extraction).
  • Captures hierarchical metadata extension in an appended compatibility trailer outside normal Compact Pro entries.
  • Trailer metadata includes file and directory records with per-field masks (permissions, ownership, timestamps, Apple flags, and NTFS attributes/timestamps when present).

• expand

  • Extracts archive contents into -d <outdir> (default .).
  • Recreates metadata-recorded empty directories in addition to file-parent directories.
  • Prints completion stats to stderr: compressed bytes, expanded bytes (compressed -> expanded), expansion ratio, MB/s throughput, elapsed seconds.
  • Progress:
    • --progress forces progress output to stderr.
    • --no-progress disables progress output.
    • default is progress on TTY stderr.
    • live output renders phase bars with percent and ETA.
  • --path <entry> may be repeated to extract only selected entries.
  • Entries are decoded on all CPU cores (up to 16) through cp_archive_extract_parallel; output and errors are identical to cp_archive_extract.
  • Restores file and directory metadata from the appended trailer when present. If the trailer's CRC-32 does not match, it prints warning: metadata trailer crc mismatch; metadata restore skipped and still extracts the (independently CRC-checked) file data.
  • Unsupported/unrestorable fields emit explicit warnings with the specific field name and do not abort data extraction.

• add

  • Adds files to an existing archive by rebuilding archive content via FFI.
  • Directory inputs are recursively expanded before add.
  • Progress:
    • --progress forces progress output to stderr.
    • --no-progress disables progress output.
    • default is progress on TTY stderr.
    • live output renders phase bars with percent and ETA.
  • Regenerates the appended metadata trailer so metadata remains synchronized after updates.

• list

  • Lists archive entries and fork sizes.
  • Metadata trailer is out-of-band and never appears as a normal archive entry.
  • Prints trailer accounting line (trailer_size, trailer_payload) at the end so on-disk overhead is explicit.

• verify

  • Checks archive integrity (header CRC, every fork's decoding, per-file CRC) without extracting anything or loading the archive into memory.
  • Uses the bounded-memory cp_archive_verify_stream C API, which reads the archive through a range-read callback and returns the same error codes as cp_archive_extract.
  • Prints ok<TAB>entries=N<TAB>bytes=M on success; on failure prints error: verify failed: <reason> (archive offset N) to stderr and exits 1. The offset is where the failure was detected: just past the last byte consumed, or the start of the CRC-covered region for header and file CRC mismatches.
  • LZH bitstream damage is reported as invalid lzh codebook or invalid lzh huffman code (C codes CP_ERR_LZH_INVALID_CODEBOOK, CP_ERR_LZH_INVALID_HUFFMAN_CODE).
  • - or @stdin reads the archive from stdin (buffered in memory, since stdin is not seekable).
  • Entries are verified on all CPU cores (cp_archive_verify_stream_parallel), each worker using fixed 128 KiB of buffers; results and reported offsets are identical to sequential verification.

Resource Fork Policy

• --sidecar: AppleDouble sidecar (._<name>)
• --rsrc <path>: explicit sidecar path
• --xattr <name>: Linux xattr key (Linux only)

Defaults:

• macOS: native resource fork path (..namedfork/rsrc) for default behavior.
• non-macOS: sidecar output on extraction; no implicit resource-fork input unless selected.

Compatibility Notes

• RLE profile implemented for read/write.
• LZH profile implemented for read/write.
• On write, each fork is encoded as RLE first, then optionally LZH-over-RLE when strictly smaller.
• Multi-block LZH writer now follows legacy block termination semantics (block_count >= 0x1fff0 with token-cost overshoot permitted), matching The Unarchiver/unar expectations.
• Container entry-count fields follow legacy Compact Pro subtree-count semantics for root/directory metadata traversal.
• Metadata extension is appended as trailer bytes after canonical Compact Pro payload; legacy tools should ignore trailing bytes while this implementation restores metadata from that trailer.
• Trailer layout (version 2): [payload][CRC-32 of payload, LE u32][magic "CPXTRLR1"][payload length, LE u32][version = 2, LE u32]. Version 1 trailers (no CRC field) from earlier releases are still read. verify fails on a version 2 CRC mismatch.

Benchmarking

• ./bm runs comparative benchmarks against compact-pro, zip, and gzip.
• Captures:
  • compress speed (wall, user, system, throughput MiB/s)
  • extract speed (wall, user, system, throughput MiB/s)
  • compression performance (output size and out/in ratio)
• Persists history to tests/benchmark/history.tsv.
• Fails loudly on sudden drift (default 20%) versus the previous row unless accepted via --accept or BENCH_ACCEPT=1.