Excessive `clientFeeChange` jobs: data race and no-subscriber update failure in `reportFeeChange()`

[rippleitinnz]

Background

David Schwartz (@JoelKatz) observed elevated clientFeeChange job rates on his hub node running 3.3.0 and applied a partial fix in his 3.3.0-DJS build. He noted: "if anyone else is seeing large numbers of clientFeeChange jobs, then we might need to investigate why I needed that change".

Seeing the same behaviour on a production validator and a test node, I investigated further. This issue documents the root cause analysis. A fix is proposed in the accompanying PR.

Summary

NetworkOPsImp::reportFeeChange() in 3.3.0 contains two related bugs that compound to produce a sustained flood of redundant JtClientFeeChange ("PubFee") jobs:

1. Data race: lastFeeSummary_ is read without holding any lock in reportFeeChange(), but is written under streamLock_ in pubServer(). Under concurrent calls, multiple threads simultaneously pass the f != lastFeeSummary_ deduplication guard and queue redundant jobs.

2. No-subscriber update failure: lastFeeSummary_ is only written inside pubServer() under if (!streamMaps_[SServer].empty()). On nodes with no WebSocket subscribers on the server stream - the typical production configuration for validators and non-transactional nodes pubServer() never updates lastFeeSummary_. This means every genuine fee change queues a job, pubServer() runs, finds no subscribers, skips the update, and the cycle repeats on the next fee change indefinitely.

Affected version

3.3.0 (00a178fb92ca49521b937ae1a99d863765ea8a90)

Root cause

Bug 1: Data race on lastFeeSummary_

// reportFeeChange() — no lock held
void
NetworkOPsImp::reportFeeChange()
{
    ServerFeeSummary const f{...};
    if (f != lastFeeSummary_)   // READ, no lock
    {
        jobQueue_.addJob(JtClientFeeChange, "PubFee", [this]() { pubServer(); });
    }
}
// pubServer() — written under streamLock_

std::scoped_lock const sl(streamLock_);

if (!streamMaps_[SServer].empty())

{

// ...

lastFeeSummary_ = f;   // WRITE, under streamLock_

}

lastFeeSummary_ is a plain member (line 1003) with no associated mutex. Between a PubFee job being queued and executing, any concurrent caller reads the stale value, passes the guard, and queues another redundant job.

Bug 2: No-subscriber update failure

pubServer() only updates lastFeeSummary_ when streamMaps_[SServer] is non-empty. On nodes with no server stream subscribers:

1. Fee changes → reportFeeChange() reads stale lastFeeSummary_, passes guard, queues job
2. pubServer() runs, finds no subscribers, skips the lastFeeSummary_ update
3. Next fee change → stale value still present, guard passes again, another job queued
4. Repeat indefinitely

This is why the issue manifests predominantly on validators and non-public nodes — they typically have no WebSocket clients subscribed to the server stream.

Evidence

1. Sustained production observation - stock 3.3.0

Two nodes (a production validator and a fresh test node) running stock 3.3.0 sampled simultaneously every 10 seconds over 5 minutes. Values are near-identical and spike simultaneously - confirming the churn is network-driven, not node-specific:

# validator (reputation-test)    # test node (chris-xahau1)
04:45:00 — per_second: 19        04:45:00 — per_second: 19
04:45:50 — per_second: 28        04:45:50 — per_second: 28
04:46:40 — per_second: 44        04:46:40 — per_second: 47  ← simultaneous spike
04:49:52 — per_second: 24        04:49:51 — per_second: 17

Rate sustained at 15–47/sec throughout, never recovering to zero.

2. JobQueue debug log - burst pattern

7 PubFee jobs queued within 27ms - consistent with multiple threads simultaneously passing the unguarded check:

2026-Aug-10 07:59:08.612 JobQueue:DBG addRefCountedJob : Adding job : PubFee : 4
2026-Aug-10 07:59:08.616 JobQueue:DBG addRefCountedJob : Adding job : PubFee : 4
2026-Aug-10 07:59:08.622 JobQueue:DBG addRefCountedJob : Adding job : PubFee : 4
2026-Aug-10 07:59:08.625 JobQueue:DBG addRefCountedJob : Adding job : PubFee : 4
2026-Aug-10 07:59:08.629 JobQueue:DBG addRefCountedJob : Adding job : PubFee : 4
2026-Aug-10 07:59:08.636 JobQueue:DBG addRefCountedJob : Adding job : PubFee : 4
2026-Aug-10 07:59:08.639 JobQueue:DBG addRefCountedJob : Adding job : PubFee : 4

3. Reproducible unit test

A new test case in src/test/rpc/Subscribe_test.cpp reliably reproduces the race:

• Subscribe to the server stream
• Raise the fee once and drain the legitimate serverStatus message
• Spawn 16 threads, barrier them, have each raise the fee and call reportFeeChange() simultaneously
• Expect exactly 1 serverStatus message (one fee change event)

Result on stock 3.3.0 - test fails:

xrpl.rpc.Subscribe reportFeeChange race condition on lastFeeSummary_
#3 failed: Subscribe_test.cpp(2034)
failed: xrpl.rpc.Subscribe had 1 failures.

More than 1 message received - multiple threads passed the guard before lastFeeSummary_ was updated.

4. 24-hour monitoring with fix applied

The test node (chris-xahau1) run as a proposing validator with the fix applied was monitored for 24 hours. clientFeeChange was absent from server_info during normal operation and appeared only during genuine network transaction spikes, correlating directly with TxQ metric changes:

Time (UTC)	Ledger	Rate	TxQ event
10:50:23	106220016	2/sec	869 tx ledger, expected=830
14:54:30	106223779	2/sec	New account burst, TxQ depth change
15:26:41	106224275	1/sec	TxQ metric oscillation
19:08:02	106227686	1/sec	775 tx ledger
19:23:16	106227920	7/sec	552 + 755 tx consecutive ledgers
21:11:36	106229593	1/sec	TxQ metric change
23:49:46	106232030	5/sec	682 tx ledger, expected=700

Every trigger correlates with a genuine ServerFeeSummary change. The rate is proportionate to the intensity of the event. The unfixed production validator maintained 15–44/sec continuously throughout the same period.

5. Correctness confirmed

Legitimate fee change notifications are correctly delivered and not suppressed by the fix. The clientFeeChange job fires when and only when the fee summary genuinely changes.

Impact

• Redundant pubServer() executions on every fee change event, proportional to thread concurrency
• Unnecessary WebSocket message fan-out to server stream subscribers on every redundant job
• Sustained job queue overhead of 15–47 jobs/sec on production validators with no WebSocket clients

Related

@nbougalis identified a separate underlying issue with fee escalation in LoadManager::run() that has not yet been resolved.