Add verifyTables to backups.verify() so a backup this build can't read fails before restore

[cb1kenobi]

⊙ Problem

backups.verify() checks file sizes, optionally checksums, and the transaction-log snapshot, but none of those catch a table this build cannot read. A checksum-valid SST written with a newer format_version, or compressed with an algorithm this build doesn't link, passes verify and only fails when something reads it. For Harper's restore_backup that is too late: the default purgeAllFiles restore has already deleted the database it was meant to replace (harper#2965: restore_backup purges the destination before anything verifies the backup can be read). harper#3121: Verify a backup can be read before restore_backup replaces the database needs a way to check this in place, before the purge.

Closes #920.

❓ Your call: Is an opt-in option on verify the right shape, or should backups.restore run the same check itself, under the shared lock it already holds, before it purges? I followed the issue: verify gains the option and the caller holds a shared .backup.lock across verify and restore (documented, and the existing shared-lock test shows a restore coexists with that holder). A restore-side preflight would protect callers that skip the recipe, and could be added later without changing this API.

💡 Solution

backups.verify(dir, id, { verifyTables: true }) opens every SST in the backup, one at a time, and rejects with the file's name when this build cannot decode its footer (for example Corrupt or unsupported format_version 99) or properties, or when the table records a compression algorithm this build does not support. It is off by default, as decided on the issue, so existing verify calls do the same work as before.

await backups.verify(backupDir, backupId, { verifyWithChecksum: true, verifyTables: true });

SstFileReader::Open alone is not enough: it never reads a data block, so an LZ4 table opened by a build without LZ4 passes Open and fails on the first cold read. The check therefore also decodes the table's compression_name property (both the format_version >= 7 encoding and the older single-name form) and fails closed on anything it cannot interpret.

No companion HarperFast/documentation PR: rocksdb-js documents its API in this repo (README and docs/backups.md, both updated here). Any user-facing change to Harper's restore_backup belongs with harper#3121.

⚖️ Alternatives

• Read the first data block of each table instead of parsing compression_name, which would avoid our own parser. Rejected: a block that does not compress well enough is stored uncompressed, so the first block proves nothing about later ones.
• Harper opens the backup as a read-only database and scans it. Rejected: a read-only open with max_open_files = -1 opens every table at once, so descriptors grow with table count, and a full scan reads every data block.
• Pin BlockBasedTableOptions::format_version so a newer build never writes a format an older one can't read. It doesn't help backups that already exist or the compression gap, so it would be a follow-up at most.

❓ Your call: Blob files are not checked. RocksDB has no public blob-file reader, so covering them means decoding the fixed blob-file header ourselves (magic, version, compression type). The gap is documented in the README, docs/backups.md and DESIGN.md; adding it later is additive. Worth a follow-up issue?

❓ Your call: The CLI's backups <dir> verify <id> still runs checksum verification only. Turning verifyTables on there is a one-line change, but it changes what an existing command checks, so I left it out. Want it?

🔧 Changes

• API: src/backup.ts rejects a non-boolean verifyTables, null included, with a TypeError, so a value from untyped config can't silently skip the check, then passes it to native. The JSDoc describes the option. src/load-binding.ts adds the sixth native argument.
• N-API: src/binding/database/backup.cpp reads the new argument (a five-argument call leaves it false) and runs the table walk only after VerifyBackup succeeds and only when the option is set.
• Table walk (Node-free, so GoogleTest covers it directly; declared in backup_verify_tables.h): src/binding/database/backup_verify_tables.cpp
  • rejects a metadata name that is absolute or contains ..;
  • opens backupDir + "/" + relative_filename, one reader at a time (BackupFileInfo::directory is a virtual private/<id> path, so it can't be used);
  • checks every recorded compression type against this build;
  • keeps RocksDB's full status (code, subcode, severity) when an open fails, so an EMFILE or EACCES doesn't read as a corrupt backup.

⚠️ Look hardest: the path join. The design originally used directory + relative_filename, which finds no healthy table. The native test covers all three sharing layouts.

• Compression parsing: tableCompressionTypes (declared in compression.h) decodes both encodings of compression_name.

❓ Your call: The parser fails closed on any compression_name it can't interpret, including an empty one. RocksDB reads an empty name as a pre-4.9 table, so such a table would be rejected even though RocksDB can read it. No table this binding writes has an empty name, and relaxing the rule later is local to compression.cpp. Keep it strict?

• Build: binding.gyp adds the source to the addon and to the native test target, along with the new test file.
• Docs:
  • README.md documents the option.
  • docs/backups.md explains it and shows holding the shared lock across verify and restore.
  • src/binding/database/DESIGN.md records the three facts the check depends on, and the root DESIGN.md index points to it.

✅ Verification

End-to-end route: the Vitest cases exercise the shipped N-API path. Harper's own preflight lands in harper#3121.

• test/native/backup_verify_tables_test.cc:
  • healthy backups pass in all three sharing layouts;
  • a table patched to format_version 99 fails with its name, while RocksDB's own size-only VerifyBackup still passes;
  • a real compressed table fails as NotSupported when the build is made to lack its compressor;
  • across 40 tables, at most one table file is open at once, on both the passing and the failing path;
  • an injected IOError/NoSpace open failure keeps its code and subcode;
  • metadata names outside the backup, behind / or \, are rejected.
• test/native/compression_test.cc covers legacy names, v7 encodings, and fail-closed cases (empty, unknown manager, odd-length or non-hex codes, ZSTDNotFinal).
• test/backup.test.ts:
  • a healthy multi-table backup passes with verifyTables and checksums;
  • a patched footer passes default verify but rejects with verifyTables, naming the file and format_version;
  • 'true' and null reject with a TypeError.
• Results:
  • pnpm test:native: all pass; the only skips are the 3 existing madvise tests.
  • Full Node suite at the final head: 80 files, 1183 passed, 10 skipped. One run printed a registry cleanup failed teardown line at worker exit for an unrelated test database. It did not reproduce in 4 repeats of test/backup.test.ts, did not appear in the 2 other full runs, and the run still exited 0.
  • test/backup.test.ts: 33/33 on Node, Bun and Deno.
  • pnpm check is clean.

_{🤖 Generated by Anthropic Claude (Opus); posted via @cb1kenobi.}

Related PRs: none found

Complexity: medium

_{Review-Coverage: authored=claude; ran=cursor-composer,gemini,codex; adjudicated=domain; declined=cursor-grok,cursor-kimi,cursor-muse; rounds=2; full=1 @ 140dbd3}

_{Review-Attention: read ~5m (decisions: restore-preflight-vs-verify-option, default-off, blob-files-out-of-scope, fail-closed-compression-name, do-less-alternative) @ 140dbd3}

[github-actions]

📊 Benchmark Results

get-sync.bench.ts

getSync() > random keys - small key size (100 records)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 lmdb	1	22.93K ops/sec	43.61	40.59	796.347	0.164	114,663
🥈 rocksdb	2	10.74K ops/sec	93.14	87.26	25,147.484	1.02	53,684

getSync() > sequential keys - small key size (100 records)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 lmdb	1	26.60K ops/sec	37.59	35.38	534.489	0.146	133,018
🥈 rocksdb	2	10.36K ops/sec	96.48	90.86	717.077	0.130	51,822

ranges.bench.ts

getRange() > small range (100 records, 50 range)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 lmdb	1	25.35K ops/sec	39.45	35.84	2,121.038	0.324	126,730
🥈 rocksdb	2	14.54K ops/sec	68.76	58.58	1,084.91	0.158	72,714

realistic-load.bench.ts

Realistic write load with workers > write variable records with transaction log

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	420.37 ops/sec	2,378.841	47.56	51,041.403	14.62	841
🥈 lmdb	2	26.13 ops/sec	38,270.911	430.769	1,213,418.358	136.225	64.00

transaction-log.bench.ts

Transaction log > read 100 iterators while write log with 100 byte records

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	34.93K ops/sec	28.63	12.59	21,143.77	0.860	174,664
🥈 lmdb	2	440.49 ops/sec	2,270.208	141.847	13,985.143	1.33	2,203

Transaction log > read one entry from random position from log with 1000 100 byte records

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	601.27K ops/sec	1.66	1.38	5,369.175	0.237	3,006,354
🥈 lmdb	2	429.03K ops/sec	2.33	1.17	7,126.087	0.625	2,145,173

worker-put-sync.bench.ts

putSync() > random keys - small key size (100 records, 10 workers)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	809.86 ops/sec	1,234.781	1,047.726	3,216.577	0.417	1,620
🥈 lmdb	2	1.15 ops/sec	870,300.626	792,741.076	1,120,676.2	8.01	10.00

worker-transaction-log.bench.ts

Transaction log with workers > write log with 100 byte records

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	19.90K ops/sec	50.24	31.50	20,633.752	2.09	39,808
🥈 lmdb	2	826.44 ops/sec	1,210.011	274.009	12,198.394	5.23	1,655

---

Results from commit 0bff6f8

[kriszyp]

Excellent, great feature!

_{🤖 Reviewed with Codex}