rolling_hash is a C++23 command-line tool for generating, applying, and
inspecting binary deltas between two files. It uses content-defined chunking
with Rabin-Karp rolling fingerprints for chunk boundaries and BLAKE2b-512
(RFC 7693, via OpenSSL) for strong chunk identity checks.
The single rolling_hash binary exposes three subcommands:
create: generate a delta from an old file and a new file.apply: reconstruct a new file from an old file and a delta.view: print a human-readable inspection of a delta file.
The project also ships rolling_hash_unit, a GoogleTest-based suite covering
hashing, file I/O, signatures, delta application, and rolling fingerprints.
- Content-defined chunking, so insertions and deletions do not force every later chunk to change.
- Adaptive chunk boundaries with a 512 byte minimum, 16 KiB maximum, and an 8 KiB target average chunk size.
- Dual chunk identity checks using a rolling fingerprint plus BLAKE2b-512.
- Delta entries for original, added, modified, and removed chunks.
- Repeated content is reused rather than re-sent: a reordered or duplicated block costs a ~19-byte reference per occurrence instead of a full copy. Doubling a 40 MB zero-filled file produces a 103 KB delta.
- A byte-level diff is emitted only when it is actually smaller than storing the chunk outright, which bounds a delta at roughly the size of its input.
- Delta application verifies each generated payload against its chunk hash and the whole reconstructed file against a trailer hash, and refuses an output path that aliases either input.
- CMake 3.16 or newer (3.19+ to use the configure presets).
- A C++23 compiler and standard library — see Compiler requirements below, as one plausible-looking combination does not work.
- OpenSSL 1.1.0 or newer, for BLAKE2b-512 via the EVP digest interface. This is a hard dependency: configure fails without it.
- Network access during first configure, because CMake fetches GoogleTest v1.14.0
for the test target. Configure with
-DBUILD_TESTING=OFFto skip both.
The quickest route is a configure preset:
cmake --preset release # or: debug, asan, no-tests
cmake --build build/release -j$(nproc)
ctest --preset release| Preset | Purpose |
|---|---|
release |
Optimised build with tests |
debug |
Unoptimised, full debug info |
asan |
Debug plus AddressSanitizer and UBSan |
no-tests |
Binary only; GoogleTest is never fetched |
Without presets:
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc)On Windows with Visual Studio:
cmake -S . -B build -G "Visual Studio 17 2022"
cmake --build build --config Release| Option | Default | Effect |
|---|---|---|
BUILD_TESTING |
ON |
OFF skips the test target, so GoogleTest is not downloaded and configure needs no network |
RH_BUILD_TESTS |
follows BUILD_TESTING |
Overrides the test target independently |
RH_ENABLE_ASAN |
OFF |
Builds with AddressSanitizer and UBSan |
apply and view parse untrusted input, so the parsers have fuzz targets:
cmake -B build-fuzz -DCMAKE_BUILD_TYPE=Debug -DRH_BUILD_FUZZERS=ON -DBUILD_TESTING=OFF
cmake --build build-fuzz -j$(nproc)
./fuzz/make_corpus.sh build-fuzz/rolling_hash corpus
build-fuzz/fuzz_apply corpus -max_total_time=60 # libFuzzer builds
build-fuzz/fuzz_apply_replay corpus/*.delta # otherwisefuzz_apply drives hostile deltas through both readers; fuzz_roundtrip
generates a delta between two halves of the input and requires an exact
reconstruction. Both check invariants beyond "did not crash": a failed apply
must leave no output file, a successful one must be deterministic, and a
freshly generated delta must always apply.
Where the compiler supports libFuzzer these are coverage-guided fuzzers; elsewhere they build as replay drivers that re-check the corpus, so the targets stay useful as regression tests. Note that Clang 18 cannot build them: libFuzzer's runtime links against libstdc++ while this project needs libc++ there, so use Clang 19+ (which works with libstdc++) for real fuzzing.
The corpus is generated rather than committed — a stored delta is pinned to the format version that produced it, and after a version bump would only exercise the version-rejection path.
cmake --install build --prefix /usr/localstd::expected is required, which rules out one combination that otherwise
looks supported: Clang before 19 reports __cpp_concepts as 201907, and
libstdc++ gates <expected> on 202002, so Clang 18 with libstdc++ compiles
as C++23 but has no std::expected. Use GCC 13 or newer, Clang 19 or newer,
or Clang with -stdlib=libc++. Configure fails with an explanatory message
rather than a wall of template errors.
Generate a delta:
./rolling_hash create oldfile.txt newfile.txt changes.deltaApply a delta:
./rolling_hash apply oldfile.txt changes.delta reconstructed.txtInspect a delta:
./rolling_hash view changes.deltaThe delta file is a versioned binary stream. It opens with a 4-byte magic and a big-endian format version, followed by one entry per chunk of the reconstructed file, and closes with a trailer holding a hash of the whole result.
Entries name their source chunk in the old file by index, so one old chunk
can back any number of new ones — repeated content costs about 19 bytes per
occurrence instead of a copy. Each entry carries a 128-bit digest truncated from
BLAKE2b-512, which both identifies the chunk and verifies it; lengths and indices
are varints. src/DeltaCodec.hpp is the single definition of the layout, and
rolling_hash view prints it.
From the repository root:
printf "hello\nold line\n" > old.txt
printf "hello\nnew line\n" > new.txt
cmake -S . -B build
cmake --build build -j$(nproc)
./build/rolling_hash create old.txt new.txt changes.delta
./build/rolling_hash apply old.txt changes.delta reconstructed.txt
cmp new.txt reconstructed.txtIf cmp exits successfully, the reconstructed file matches the new file.
-
Signaturereads each input file and splits it into variable-sized chunks. -
Every chunk receives a Rabin-Karp rolling fingerprint and a BLAKE2b-512 hash.
-
Deltacompares the old and new signatures and emits one entry per new chunk. A chunk whose content exists anywhere in the old file becomes a reference to it rather than a copy — however many times it recurs, and without needing a matching entry for old chunks that are simply gone. -
Chunks whose content is not in the old file at all are shipped whole; otherwise a modified chunk stores compact byte-level diff opcodes, computed with Myers' O(ND) algorithm (falling back to a greedy diff when the edit distance is large):
D: replace bytes at a position.I: insert bytes at a position.X: delete bytes at a position.
The diff is kept only if it costs less than storing the chunk literally; otherwise the chunk is emitted whole.
-
Applyreads the old file and delta records in target-file order, verifies each generated payload against its recorded hash, and writes the reconstructed output. A trailing whole-file hash is checked at the end, so truncation or reordering is caught even when every individual chunk verifies.
All multi-byte integer fields are big-endian. The stream carries a format version and readers reject anything they do not recognise, but the format is still an internal one: it is not a compatibility promise across versions.
| Status | Meaning |
|---|---|
0 |
Success |
1 |
Bad input: malformed or truncated delta, hash mismatch, or an output path that aliases an input |
2 |
Environment failure: a file could not be opened, read, or written |
Build and run the GoogleTest suite:
cd build
ctest --output-on-failure
./rolling_hash_unitThe current test suite covers:
- File I/O: opening, reading, writing, buffered byte-wise reads interleaved with bulk reads, EOF behaviour, close semantics, and invalid paths.
- BLAKE2b-512 against the RFC 7693 vectors, streaming versus one-shot.
- Rabin-Karp fingerprints checked against a naive reference at every position, for both the Mersenne fast path and the general modulus.
- Signature generation, including sub-window files and resynchronisation after an insertion.
- The delta codec: big-endian round trips, header and version rejection, and truncation in the entry header, opcodes, and trailer.
- The chunk index, including reuse of repeated content.
- Concept conformance, including implementations that inherit from nothing.
- Delta application for identical files, empty inputs, append and truncate cases, in-chunk modifications, malformed and hostile deltas, out-of-range chunk sizes, failure categories, and output alias protection.
src/
main.cpp rolling_hash CLI entry point and subcommand dispatcher
Apply.hpp delta application logic
Delta.hpp delta generation, Myers diff, entry selection
Signature.hpp content-defined chunk signature generation
RK_finger.hpp Rabin-Karp rolling fingerprint implementation
ChunkIndex.hpp content index shared by Delta and Apply
DeltaCodec.hpp the delta wire format: reader, writer, and layout
DeltaFormat.hpp format constants, entry types, and chunk-size bounds
DeltaError.hpp failure categories and success statistics
HashConcepts.hpp requirements on the rolling and strong hash algorithms
DeltaViewer.* delta inspection command implementation
FileIO.* buffered file I/O helper
blake2b.* BLAKE2b-512 via OpenSSL EVP
rh_config.h.in configure-time toolchain probes
tests/
*_tests.cpp GoogleTest unit tests
CMakeLists.txt build and test configuration
CMakePresets.json release / debug / asan / no-tests presets
Keep changes warning-clean under the CMake options in CMakeLists.txt
(-Wall -Wextra for non-MSVC builds, /W4 for MSVC); CI treats warnings as
build failures. Add focused tests under tests/ when changing file I/O,
chunking, hashing, delta generation, or delta application behaviour.
Before proposing a change that touches the delta format or the diff algorithm, check the effect on both correctness and size: a create/apply round trip must reproduce the new file byte for byte, and a refactor that is not meant to change output should produce byte-identical deltas. Run the sanitised build too:
cmake --preset asan && cmake --build build/asan -j$(nproc) && ctest --preset asanThis project is licensed under the MIT License. See LICENSE.
Piotr Olszewski (asmie)