Skip to content

Stop creating empty transaction logs named after unmapped node ids - #2787

Merged
kriszyp merged 3 commits into
mainfrom
fix/txnlog-numeric-log-no-create
Sep 25, 2026
Merged

kriszyp merged 3 commits into
mainfrom
fix/txnlog-numeric-log-no-create

Conversation

@kriszyp

@kriszyp kriszyp commented Sep 24, 2026 •

Copy link
Copy Markdown
Member

Reading the audit log by a node id that has no transaction log of its own (a relayed origin, or one whose log was removed) no longer creates an empty RocksDB transaction log named after that id. RocksTransactionLogStore.getRange now treats a numeric log as a lookup into the node-id→log map and returns an empty range when there is no log; a string log name keeps get-or-create.

Closes #2778

Before: every auditStore.get(key, tableId, id, nodeId) for such an id fell through to rootStore.useLog(nodeId), which is get-or-create in rocksdb-js, so a permanently empty log "53" appeared on disk. On the next start loadLogs() treated "53" as a node name and minted a new node id for it. The lookup itself missed either way, because the new log was empty.

For the human reviewer

  1. Requirement. Warranted as filed: log creation is a side effect of a read, it adds one persisted node-id mapping per junk name on the next load, and it adds one iterator per junk log to every all-logs getRange (replication and subscriptions). Not superseded by in-flight work; Report a transaction log's corrupt-frame stop to getRange callers through an onCorruptFrame hook edits getRange for a different reason and may conflict textually, not semantically.
  2. An unmapped node id is a silent empty range. Alternative: log once per id. Not done because this runs on every audit lookup on the replication apply path, and a node id without its own log is a normal state (put() already routes those writes to the via-node or local log). Adding a log line later is cheap.
  3. Existing junk logs are left on disk. Stores already named "53" etc., and the node ids minted for them, stay. Alternative: prune empty numeric-named logs at load. Not done because that cannot tell a junk log from a node whose configured name is all digits, and deleting stores at startup races writers. After every node runs this fix, the number of junk logs stops growing; during a rolling deploy, older nodes still create them. Removing them is an operator action or a separate issue.
  4. Adjacent gap, not changed here. Lookups by origin id still miss entries that put() wrote to the via-node or local log for an origin with no log of its own. This PR does not change that: those lookups already missed (they opened an empty numeric log). Fixing it needs viaNodeId at the 12 get/getSync call sites or an all-logs scan per miss, so it belongs in its own issue.

Planning review: Framing-Verdict: chosen-approach-sound (Codex, --mode plan). Alternatives rejected there: change rocksdb-js useLog (the numeric id is the wrong key whether or not useLog creates), resolve the id to where put() wrote (see 4), clean up junk logs at load (see 3).

Changes

Verification

  • Regression test (unitTests/resources/auditLog.test.js "a numeric log id with no log misses without creating one", RocksDB only; skipped on LMDB, which has no per-node logs): a local write is found by node id 0; a peer write through a source-apply transaction is found by its mapped node id and by nodeId: undefined (all logs); an unmapped id returns undefined from auditStore.get, and getRange for it is empty with no failures and does not add String(id) to rootStore.listLogs(); a string log name still creates its log.
  • Fails on base: on 07b2c9015 the test fails with reading by node id 1001 must not create a transaction log named after it (log "1001" created); with the fix it passes. rocksdb-js 2.10.0 probe: db.useLog(53) makes listLogs() return ['53', 'local'].
  • npm run test:unit:resources — 3030 passing, 48 pending, 0 failing.
  • npm run test:unit:main — 1843 passing, 8 failing, none in transaction-log code. 7 were timeouts in unitTests/components/{packageComponent,prepareApplicationSerialization,RuntimeModuleTracker}.test.js under a load average of ~18 from other worktrees' suites; those three files pass 22/22 when run alone. The 8th, gitCredentials.test.js "grants the credential environment to the spawn that clones", fails locally because the dispatch worker's own shell sets GIT_CONFIG_GLOBAL/GIT_EDITOR/GIT_SSH_COMMAND, independent of this diff.
  • npm run test:integration:all — 2150 pass, 0 fail, 18 skipped, 6 cancelled. All 6 cancelled are OllamaBackend against a real Ollama instance, which needs Ollama models this machine does not have.
  • npx oxlint --deny-warnings and prettier --check on the changed files, npm run check:design-docs — clean.
  • CI: the first Unit Test (Node.js v26) run failed twice. One failure was caching.test.js:365 "Cache invalidation triggers updates", which asserts after fixed 10 ms / 20 ms sleeps around a 5 ms TTL and never reaches a numeric log lookup. The other was the Unit tests: lmdb step's 4-minute timeout, which main also hit in 3 of its last 6 completed unit-test runs. The caching test passed 5/5 locally on Node 26.2 and on this PR's Node 22 and 24 legs. The rerun of the Node 26 job passed, and all 43 checks are green.
  • End to end: not observable in harper core alone. It needs a relayed or removed origin in a multi-node cluster, which is harper-pro's integration surface; no such test was run for this PR. The unit test drives the production entry point (auditStore.get on a real RocksDB-backed table).

Signed: Claude Opus 5.5

🤖 Generated with Claude Code

https://claude.ai/code/session_01Lt1QAheetzHhsS8TFW9cGj

Complexity: medium

Dispatch: task chat-issue-harper-2778-kriszyp · queued by unknown · ran by claude/opus/xhigh · worker kzyp-xps-1

Review-Coverage: authored=claude; ran=codex,gemini; adjudicated=domain; declined=cursor-grok,cursor-composer,cursor-kimi,cursor-muse; rounds=3; full=1 @ f450474

Human-Review-Need: 3 (decisions: numeric-miss-is-empty-not-fallback, no-cleanup-of-stray-numeric-logs) @ f450474

kriszyp and others added 3 commits September 24, 2026 15:08
…ode id

RocksTransactionLogStore.getRange({ log: <number> }) fell through to
rootStore.useLog(number) when the node id had no log, and useLog is
get-or-create: every audit lookup by a relayed or removed origin's id
created a permanently empty log named after the id, which the next
load then mapped to a freshly minted node id. A numeric log is now a
pure lookup into the id→log map and an unmapped id yields an empty
range; string log names keep get-or-create.

Closes #2778

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Lt1QAheetzHhsS8TFW9cGj
…mes a log

Refs #2778

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Lt1QAheetzHhsS8TFW9cGj
…its enforcement

Refs #2778

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Lt1QAheetzHhsS8TFW9cGj
@kriszyp kriszyp added this to the v5.3 milestone Sep 24, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates RocksTransactionLogStore.getRange to ensure that a numeric transaction-log selector (representing a node ID) is treated strictly as a lookup and never as a log name. This prevents the unintended creation of a new transaction log via useLog when an unmapped node ID is queried, returning an empty range instead. The changes are documented in DESIGN.md and verified with a new unit test in auditLog.test.js. There are no review comments to address.

@kriszyp
kriszyp marked this pull request as ready for review September 25, 2026 13:02
@kriszyp
kriszyp merged commit 6584ad0 into main Sep 25, 2026
57 of 58 checks passed
@kriszyp
kriszyp deleted the fix/txnlog-numeric-log-no-create branch September 25, 2026 13:02
@claude

claude Bot commented Sep 25, 2026

Copy link
Copy Markdown
Contributor

Reviewed; no blockers found.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RocksTransactionLogStore.getRange creates empty stores named after unknown numeric origin ids (useLog(number))

1 participant