[williamstein]

What

compactHistory({ patches, codec, policy, now, budget }) is a pure, deterministic compaction of a patchflow history for retention strategies. It was requested by William; the brief is from @ops3. Supported policies: keepAll, keepSince(since), uniform(interval), tiered(tiers), latestOnly and union(policies). It returns the new history, the kept versions, and stats: versions and encoded bytes before and after, identity, overBudget, and the work done (diffs, diffSize).

How (src/compact-history.ts)

• Always kept: the heads, plus the merge bases their merged value is computed from (PatchGraph's own fold: heads in time order, each from its maximal common ancestors, recursively).
• Kept versions: each one's parents become its nearest kept ancestors, and its patch is a diff from their exact old value. A merge records its merged value (mergeParent/mergePatch, as Session does), so values never depend on future merge3 changes.
• Roots: a kept version with no kept ancestors becomes a snapshot. The first record is always one.
• Dropped patches are squashed into their kept descendants, which record meta.squashed = { users (in order), start, end, count }.
• Untouched records: a record whose parents don't change and that absorbs nothing is kept as it was, minus any snapshot text a store attached. This makes compaction idempotent and leaves untouched history byte for byte.
• Safety nets:
  • every kept value is verified as the new history is built, falling back to storing the value if a diff ever fails to reproduce it;
  • the newest merged value is checked at the end;
  • the input comes back unchanged if the result would be larger (stopping as soon as it is) or the operation budget runs out.
• Operation budget (William): diffs dominate the cost, so they are counted rather than timed (maxDiffs, maxDiffSize). The same input and budget always give the same result.
• Rejected input: histories that are incomplete (a gap, or no root/snapshot to start from) and codecs without merge3.

Tests (src/compact-history.test.ts)

Property and fuzz tests over 240 random concurrent histories, text and db (up to 3 clients, catch-ups, merge commits with and without a recorded merged value). Each runs a random policy and checks:

1. kept versions are exact;
2. the newest value is unchanged, all heads are kept, and the policy's versions are kept;
3. the result is a valid history (unique ids in order, parents first, a snapshot first, no more history needed) and deterministic;
4. idempotent;
5. authorship is preserved;
6. never larger.

Also: unit tests for every policy and selectVersions, a history that starts at a snapshot, refusal of incomplete histories, budget tests, and a 10k-patch scale test. Results: patchflow 241/241 tests pass; oxlint and the build are clean.

Benchmark (pnpm bench:compact 50000 500 300)

50k one-line edits to a 500 KB text, with a snapshot every 300 patches as CoCalc stores them (79 MB). Peak RSS was 762 MB including generating the input; nothing was retained.

policy	time	versions	bytes
latestOnly	0.5 s	50000 → 1	79 MB → 0.42 MB
keepSince 1 day	2.3 s	→ 1441	→ 0.64 MB
uniform 1 day	4.6 s	→ 35	→ 30 MB
tiered	12 s	→ 3026	→ 74 MB

Without intermediate snapshots, every value replays from the root (27–70 s for 50k).

Found on the way (not fixed here; in PLAN.md)

With @cocalc/diff-match-patch 1.1.0 and DIFF_WORK_LIMIT 25M (#30, released in 0.15.0), a one-line edit to a text of about 800 KB or more gives a patch that replaces the whole text:

text	patch
600 KB	99 bytes
800 KB	1.6 MB
2 MB	4.1 MB

It looks like charges such as 2 * (len1 + len2) are made before the common prefix and suffix are removed. This affects every commit to such a text, so CoCalc should not move to 0.15 until it's fixed (CoCalc main is on 0.13). It's also why uniform/tiered diffs over many edits of the 500 KB benchmark text are whole-text replacements, about 950 KB each.

Record format for hosts

See the README section "History compaction", "Storing a compacted history":

• the first record (and any record without parents) is a snapshot with parents: [], no patch, hash, and no seqInfo;
• every other record is a patch with its original id, userId and wall, plus hash, and for a merge mergeParent/mergePatch;
• meta.squashed must be stored;
• old snapshot records are not carried over;
• a rare repaired record carries snapshot plus parents.

— @ops1

[williamstein]

@codex review

[williamstein]

@codex review

[williamstein]

@codex review