Skip to content

Block download stalls after initial sync (node stays behind the header tip) #1273

Open

Description

@Juniormind1

Bug report: block download stalls after initial sync (node stays behind the header tip)

Repo: libbitcoin/libbitcoin-node (affects the full node / bs)
Component: chaser_download (block download chaser)


Summary

After a full initial sync from genesis, the node stops downloading and validating
blocks while the header chain continues to advance to the network tip. The
block tip freezes a handful of blocks above the configured milestone, and the
download chaser never issues further work. The stall is reproducible and also
blocks shutdown (the DB drain never completes → forced kill → snapshot
recovery → stall again at the same height).

Environment

  • OS: Debian GNU/Linux 13 (trixie), kernel 6.12.115+deb13.1-amd64, x86_64

  • Hardware: 4 vCPU (QEMU), 21 GiB RAM

  • Build: GNU autotools, --build-config=release --build-link=static --build-skip-tests --enable-isystem

  • Versions (bs --version):

    libbitcoin-system.... 4.0.178 7e81230c
    libbitcoin-database.. 4.0.107 0710236a
    libbitcoin-network... 4.0.157 cef2ff21
    libbitcoin-node...... 4.0.111 a2487f07
    libbitcoin-server.... 4.0.283 6b34025a
    compiled schema...... 4.0.32
    
  • Network: mainnet, non-pruned store (pruned:0), store schema 4.0.32

  • Relevant config:

    [bitcoin]
    network   = mainnet
    milestone = 0000000000000000000084caf9eb062a1088955dd60a93de234355d96b41c0ab:976432
    
    [manual]
    peer = 192.168.2.196:56922
    
    [node]
    # all defaults (limited_blocks=false, require_blocks=true, require_witness=true,
    # currency_window_minutes=1440, maximum_concurrency=50000, compact_timeout_seconds=10)

Observed behaviour

  1. The store is built from genesis and reaches the then-current tip (~976467)
    in ~16 h. Up to this point block download + validation works normally
    (Validated full block [976467] …).

  2. Immediately afterwards the node stops downloading/validating blocks, while
    headers keep advancing. The block tip freezes at 976465–976467.

    /v1/top:

    {"confirmed":{"hash":"000000000000000000000f9b6bbe836e08ebcdd3fe765098c7cc58eb8158574a","height":976465},
     "candidate":{"hash":"00000000000000000000bc8d19eb60d51cd75c10c21a74e22210150db600b58e","height":976544}}

    GET /v1/block/height/976468 (and every height up to the candidate tip)
    returns Not Found.

  3. Exactly one Advance by line is emitted, then no further download
    activity:

    2026-11-18T19:16:38Z.1 Validated full block [976467] of (3724) txs with (3592) unpooled in (3853844) usecs.
    2026-11-18T19:17:38Z.1 Advance by (50000) above (976467) from (976467) stop (1026467) found (1) last (976468).
    ... (11 hours of nothing) ...
    
  4. The process is otherwise idle: ~0 disk I/O, low CPU, many connected peers,
    header chaser active. No validation, no download, no error/fault logged.

  5. The node reports itself current even though it is ~79 blocks behind:

    2026-11-19T06:48:13Z.0 Memory is below the amount recommended for initial block download.
    2026-11-19T06:48:14Z.1 Node is current at startup block [976465].
    

Reproduction after a restart

Restarting does not clear the condition:

$ ./stop-bs.sh
Sende SIGTERM an bs (PID 1278).
Warte bis zu 600s auf sauberes Herunterfahren (DB-Drain) ...
WARNUNG: bs hat nach 600s nicht gestoppt. Sende SIGKILL.      # shutdown blocked
bs hart beendet.

The forced kill leaves the store flush-locked; on the next start bs performs a
snapshot recovery and comes back at the snapshot height, then stalls again at
exactly the same point
:

2026-11-19T06:48:14Z.1 Node is current at startup block [976465].
2026-11-19T06:48:16Z.1 Advance by (50000) above (976465) from (976465) stop (1026465) found (3) last (976468).

Only the count changes (found (1) vs found (3)), always ending at
last (976468).

Analysis

chaser_download::set_unassociated() (libbitcoin-node/src/chasers/chaser_download.cpp)
defers all new work while validation has not caught up to the last request:

// Defer new work issuance until gaps filled.
if (position() < requested_)
    return {};

// Defer new work until validation caught up to request.
if (advanced_ < requested_)
    return {};

At the stall point: position() == 976465, advanced_ == 976465,
requested_ == 976468. The three requested blocks (976466–976468) are never
delivered/validated, so advanced_ never reaches requested_, and the chaser
returns {} forever. The header chaser keeps advancing the candidate to the tip
(976544) but the download chaser is not re-triggered.

Two things look relevant:

  1. Compact-top deferral. In the same function:

    if (!is_zero(timeout.count()) && is_current_chain(false))
    {
        size_t top{};
        const auto link = query.to_top_candidate();
        if (link != expired_ && query.is_compact(link) &&
            !query.is_associated(link) && query.get_height(top, link))
        {
            stop = std::min(stop, sub1(top));
            ...
        }
    }

    At the single Advance, the top candidate was 976469, so stop was clamped to
    976468 and only 3 blocks were found. When headers later advanced to 976544,
    no further Advance was produced — i.e. the header-side chases::headers /
    chases::bump notification does not appear to restart the download chaser in
    this state.

  2. get_inventory_size() requires is_current_chain(false). is_current_chain
    is a 24 h currency-window test on the candidate top timestamp, which is true
    here, so this is not the gate — but note the node considers itself "current"
    (and starts the mempool/pool machinery) while still ~79 blocks behind the tip.

Suspected regression area

The stall appears right after the node transitions from IBD to "current" and
begins pool/compact-block processing. Recent commits in this area:

9365109 Archive full blocks on the compact block strand while pooling.
e072a88 Pool an archived tx not yet pooled and request one not yet known.
800f46e Fire pool timing and peer drop events.

Impact

  • The node never catches up to the tip (permanently stuck N blocks behind).
  • bs cannot be shut down cleanly (drain blocks on the stuck job), so it is
    killed and the store is recovered from the last snapshot on every restart.

Question

Is this a known issue? Is there a configuration that avoids it (e.g. disabling
compact blocks / node.compact_timeout_seconds=0, or a different
maximum_concurrency), and is there a supported way to un-stick a running node
without the forced-kill + snapshot-recovery cycle?

Happy to provide the full console log / store layout on request.

Activity

  1. evoskuil commented on Nov 19, 2026

    @evoskuil
    Member

    Thanks for the detailed report. This is a known defect, already fixed on master in 93853e4 (#1271), which merged after your build (node a2487f0).

    Your build has 9365109, under which a full block archives through the transaction chaser while the pool is active. If the channel that delivered the block stopped before that completion ran, the completion was dropped. The block was stored, but the download chaser was never told, so it waits forever for a block it already has. Your "Advance by ... last (976468)" followed by silence is that state. The compact deferral is not involved (your stop was not clamped: 1026467). The headers notification does reach set_unassociated, which returns at the gap check.

    The restart repeats the stall because the forced kill and snapshot restore discard the stored block, so it is downloaded again through the same race. No configuration avoids it, and compact_timeout_seconds does not affect this path. Please update to current master.

    The shutdown that did not complete in 600 seconds is a separate matter. We have closed from this stalled state in seconds, so if it recurs on a current build, please attach the console log from the SIGTERM onward.

  2. self-assigned this
    on Nov 19, 2026
  3. added this to the 4.0 milestone on Nov 19, 2026
  4. added theissue type on Nov 19, 2026
  5. evoskuil commented on Nov 19, 2026

    @evoskuil
    Member
    $ ./stop-bs.sh
    Sende SIGTERM an bs (PID 1278).
    Warte bis zu 600s auf sauberes Herunterfahren (DB-Drain) ...
    WARNUNG: bs hat nach 600s nicht gestoppt. Sende SIGKILL.      # shutdown blocked
    bs hart beendet.
    

    I suspect this is an error in the shell script. It's common for clanker scripts to fail to handle the exit prompt.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions