Repository navigation
Description
Bug report: block download stalls after initial sync (node stays behind the header tip)
Repo: libbitcoin/libbitcoin-node (affects the full node / bs)
Component: chaser_download (block download chaser)
Summary
After a full initial sync from genesis, the node stops downloading and validating
blocks while the header chain continues to advance to the network tip. The
block tip freezes a handful of blocks above the configured milestone, and the
download chaser never issues further work. The stall is reproducible and also
blocks shutdown (the DB drain never completes → forced kill → snapshot
recovery → stall again at the same height).
Environment
-
OS: Debian GNU/Linux 13 (trixie), kernel
6.12.115+deb13.1-amd64, x86_64 -
Hardware: 4 vCPU (QEMU), 21 GiB RAM
-
Build: GNU autotools,
--build-config=release --build-link=static --build-skip-tests --enable-isystem -
Versions (
bs --version):libbitcoin-system.... 4.0.178 7e81230c libbitcoin-database.. 4.0.107 0710236a libbitcoin-network... 4.0.157 cef2ff21 libbitcoin-node...... 4.0.111 a2487f07 libbitcoin-server.... 4.0.283 6b34025a compiled schema...... 4.0.32 -
Network: mainnet, non-pruned store (
pruned:0), store schema 4.0.32 -
Relevant config:
[bitcoin] network = mainnet milestone = 0000000000000000000084caf9eb062a1088955dd60a93de234355d96b41c0ab:976432 [manual] peer = 192.168.2.196:56922 [node] # all defaults (limited_blocks=false, require_blocks=true, require_witness=true, # currency_window_minutes=1440, maximum_concurrency=50000, compact_timeout_seconds=10)
Observed behaviour
-
The store is built from genesis and reaches the then-current tip (~976467)
in ~16 h. Up to this point block download + validation works normally
(Validated full block [976467] …). -
Immediately afterwards the node stops downloading/validating blocks, while
headers keep advancing. The block tip freezes at 976465–976467./v1/top:{"confirmed":{"hash":"000000000000000000000f9b6bbe836e08ebcdd3fe765098c7cc58eb8158574a","height":976465}, "candidate":{"hash":"00000000000000000000bc8d19eb60d51cd75c10c21a74e22210150db600b58e","height":976544}}GET /v1/block/height/976468(and every height up to the candidate tip)
returnsNot Found. -
Exactly one
Advance byline is emitted, then no further download
activity:2026-11-18T19:16:38Z.1 Validated full block [976467] of (3724) txs with (3592) unpooled in (3853844) usecs. 2026-11-18T19:17:38Z.1 Advance by (50000) above (976467) from (976467) stop (1026467) found (1) last (976468). ... (11 hours of nothing) ... -
The process is otherwise idle: ~0 disk I/O, low CPU, many connected peers,
header chaser active. No validation, no download, no error/fault logged. -
The node reports itself current even though it is ~79 blocks behind:
2026-11-19T06:48:13Z.0 Memory is below the amount recommended for initial block download. 2026-11-19T06:48:14Z.1 Node is current at startup block [976465].
Reproduction after a restart
Restarting does not clear the condition:
$ ./stop-bs.sh
Sende SIGTERM an bs (PID 1278).
Warte bis zu 600s auf sauberes Herunterfahren (DB-Drain) ...
WARNUNG: bs hat nach 600s nicht gestoppt. Sende SIGKILL. # shutdown blocked
bs hart beendet.
The forced kill leaves the store flush-locked; on the next start bs performs a
snapshot recovery and comes back at the snapshot height, then stalls again at
exactly the same point:
2026-11-19T06:48:14Z.1 Node is current at startup block [976465].
2026-11-19T06:48:16Z.1 Advance by (50000) above (976465) from (976465) stop (1026465) found (3) last (976468).
Only the count changes (found (1) vs found (3)), always ending at
last (976468).
Analysis
chaser_download::set_unassociated() (libbitcoin-node/src/chasers/chaser_download.cpp)
defers all new work while validation has not caught up to the last request:
// Defer new work issuance until gaps filled.
if (position() < requested_)
return {};
// Defer new work until validation caught up to request.
if (advanced_ < requested_)
return {};At the stall point: position() == 976465, advanced_ == 976465,
requested_ == 976468. The three requested blocks (976466–976468) are never
delivered/validated, so advanced_ never reaches requested_, and the chaser
returns {} forever. The header chaser keeps advancing the candidate to the tip
(976544) but the download chaser is not re-triggered.
Two things look relevant:
-
Compact-top deferral. In the same function:
if (!is_zero(timeout.count()) && is_current_chain(false)) { size_t top{}; const auto link = query.to_top_candidate(); if (link != expired_ && query.is_compact(link) && !query.is_associated(link) && query.get_height(top, link)) { stop = std::min(stop, sub1(top)); ... } }
At the single
Advance, the top candidate was 976469, sostopwas clamped to
976468 and only 3 blocks were found. When headers later advanced to 976544,
no furtherAdvancewas produced — i.e. the header-sidechases::headers/
chases::bumpnotification does not appear to restart the download chaser in
this state. -
get_inventory_size()requiresis_current_chain(false).is_current_chain
is a 24 h currency-window test on the candidate top timestamp, which istrue
here, so this is not the gate — but note the node considers itself "current"
(and starts the mempool/pool machinery) while still ~79 blocks behind the tip.
Suspected regression area
The stall appears right after the node transitions from IBD to "current" and
begins pool/compact-block processing. Recent commits in this area:
9365109 Archive full blocks on the compact block strand while pooling.
e072a88 Pool an archived tx not yet pooled and request one not yet known.
800f46e Fire pool timing and peer drop events.
Impact
- The node never catches up to the tip (permanently stuck N blocks behind).
bscannot be shut down cleanly (drain blocks on the stuck job), so it is
killed and the store is recovered from the last snapshot on every restart.
Question
Is this a known issue? Is there a configuration that avoids it (e.g. disabling
compact blocks / node.compact_timeout_seconds=0, or a different
maximum_concurrency), and is there a supported way to un-stick a running node
without the forced-kill + snapshot-recovery cycle?
Happy to provide the full console log / store layout on request.
Activity
evoskuil commentedon Nov 19, 2026
Thanks for the detailed report. This is a known defect, already fixed on master in 93853e4 (#1271), which merged after your build (node a2487f0).
Your build has 9365109, under which a full block archives through the transaction chaser while the pool is active. If the channel that delivered the block stopped before that completion ran, the completion was dropped. The block was stored, but the download chaser was never told, so it waits forever for a block it already has. Your "Advance by ... last (976468)" followed by silence is that state. The compact deferral is not involved (your stop was not clamped: 1026467). The headers notification does reach
set_unassociated, which returns at the gap check.The restart repeats the stall because the forced kill and snapshot restore discard the stored block, so it is downloaded again through the same race. No configuration avoids it, and
compact_timeout_secondsdoes not affect this path. Please update to current master.The shutdown that did not complete in 600 seconds is a separate matter. We have closed from this stalled state in seconds, so if it recurs on a current build, please attach the console log from the SIGTERM onward.
evoskuil commentedon Nov 19, 2026
I suspect this is an error in the shell script. It's common for clanker scripts to fail to handle the exit prompt.