feat: Shed resident SHAMap subtrees from the validated ledger's state map

[dangell7]

High Level Overview of Change

A SHAMapInnerNode holds a strong pointer to every child that has been read into memory. On a backed immutable map there is no way to release those pointers, so the state map of a validated ledger keeps every node any reader has ever loaded for as long as the ledger is retained. On our perf network that resident tree, not the TreeNodeCache, is the term that grows with account count: about 13 million live nodes at 10 million accounts.

This adds the reverse operation. SHAMap::shedCold(minDepth) walks the resident tree of a backed immutable map and drops the child pointers of clean inner nodes at or below minDepth, keeping the child hashes, so descend reads a dropped node back from the NodeStore the next time a traversal needs it. The sweep calls it on the validated ledger's state map when [shed_cold_subtrees] is on. Default off.

• SHAMapInnerNode::dropChild(branch) resets the child pointer under the branch spinlock and leaves the hash array and isBranch_ untouched, which is the state a freshly faulted inner node starts in.
• compare, belowHelper, visitDifferences, getMissingNodes and getNodeFat walk with bare SHAMapTreeNode pointers. Each now takes a shared lock on a process-wide shared_mutex while shedding is enabled, and shedCold takes it exclusively. With the gate off the guard is one relaxed atomic load and no lock. The mutex is process-wide because immutable snapshots of one ledger are distinct SHAMap objects sharing the same physical nodes, so a per-object lock would not cover a reader on another snapshot.
• Only clean nodes (cowid() == 0) are shed, since a dirty node's subtree may not be on disk yet. The root is never dropped. The sweep only sheds the ledger returned by getValidatedLedger(), so the open and current ledgers are never touched.
• [shed_cold_subtrees] (default false) and [shed_min_depth] (default 3) are documented in cfg/xrpld-example.cfg.
• Admin command shed {enable, run, min_depth} sets the gate at runtime and runs one pass, returning the dropped count and the TreeNodeCache size and track counts before and after. A pass runs only if the gate was already on before the request: a descent that started with the gate off holds no lock, so enabling and shedding in one call could free a node under it.

Context of Change

This is the follow-up to the tree cache work in #7965 and #7966. memory_limit bounds the caches and removes the OOM cliff, but the memory versus accounts slope is the resident state tree, and no cache bound reaches it. This is the mechanism for that slope.

It is a draft so it can be composed into the perf network image and measured before anything about it is called a result. The run should record, at the same account count with the gate on and off: resident memory, treenode_track_size, shedCold pass duration (readers holding the guard are excluded while it runs, and a pass waits for in-flight guarded walks), and read latency after a pass. That run picks the default for shed_min_depth.

API Impact

• Public API: New feature (new methods and/or new fields)
• Public API: Breaking change (in general, breaking changes should only impact the next api_version)
• libxrpl change (any change that may affect libxrpl or dependents of libxrpl)
• Peer protocol change (must be backward compatible or bump the peer protocol version)

The new shed command is admin only. SHAMap gains shedCold, setShedEnabled and shedEnabled; SHAMapInnerNode gains dropChild. No existing signature changes. develop is at 3.4.0-rc1 and this is not a 3.4.0 change, so the API-CHANGELOG entry follows once the target release is set.

Test Plan

SHAMapShed.shed_cold_is_transparent (gtest): 3000 items into a backed map, flushDirty, setImmutable, shedCold(1) drops more than zero pointers, the TreeNodeCache is cleared so re-reads must go to the NodeStore, then every sampled item reads back byte identical and the map hash is unchanged.

On this branch merged with develop, Debug, macOS: xrpl_tests --gtest_filter='SHAMap*' 16 tests passed; xrpl.app.SHAMapStore 12 cases, 965 tests, 0 failures; xrpl.rpc.RPCCall 3 cases, 2085 tests, 0 failures; xrpl.rpc.LedgerData 5548 tests, 0 failures. Pre-commit clean over every changed file.

The behavioral proof is the perf network run described above. No measurement is claimed here.

[codecov]

Codecov Report

❌ Patch coverage is 42.47788% with 65 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/xrpld/rpc/handlers/admin/data/Shed.cpp	0.0%	40 Missing ⚠️
src/xrpld/app/main/Application.cpp	15.4%	11 Missing ⚠️
src/libxrpl/shamap/SHAMap.cpp	77.8%	8 Missing ⚠️
src/libxrpl/shamap/SHAMapInnerNode.cpp	83.3%	2 Missing ⚠️
src/xrpld/core/detail/Config.cpp	50.0%	2 Missing ⚠️
include/xrpl/shamap/SHAMap.h	75.0%	1 Missing ⚠️
src/libxrpl/shamap/SHAMapSync.cpp	66.7%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[github-actions]

This PR has conflicts, please resolve them in order for the PR to be reviewed.