feat(storage): say on the wire whether a node still holds an old chunk store

config/production.toml

@@ -49,57 +49,18 @@ verify_on_read = true
 # Maximum size in GiB of the legacy LMDB store, while one still exists
 # (0 = derive it from available disk). Retired along with LMDB itself.
 db_size_gb = 0
-
-# --- Moving off the legacy LMDB chunk store ---
+# Storage migration off the old chunk store
 #
 # Chunks are now one file each, under {root_dir}/chunks/. A node that still has a
 # chunks.mdb copies it into files in the background, then deletes it whole, which is the
 # only moment LMDB's disk comes back.
 #
-# The two release-level switches (whether to delete the old store, and whether audits
-# still penalise) belong to the build, not to this file, so they are deliberately absent.
-[storage.migration]
-# Run the copier. Turning this off leaves both stores in place forever and never
-# returns the old store's disk.
-enabled = true
-
-# Also write new chunks to the legacy store while it exists, so a fleet rollback to an
-# older build cannot lose a chunk uploaded during the migration.
-dual_write_legacy = true
-
-# Allow a node that cannot fit its chunks to give up the ones it is furthest from.
-#
-# Whatever this is set to, a chunk is only ever given up when the node is near the back of
-# its group for it, its close group has received the node's reduced commitment, AND all but
-# one of that group has cryptographically proven it holds a copy. A node that cannot show
-# all three keeps both stores and asks for more disk. Turn this off if you would rather add
-# disk than have the node give anything up at all.
-allow_shed = true
-
-# Hours after this build first starts before a node may give anything up, so peers on
-# older builds have upgraded and stopped penalising it for doing so.
-shed_hold_hours = 72
-
-# Hours between one migration wave opening and the next.
-#
-# A close group is split into waves so only two of its members give chunks up at a time.
-# If all seven went together none could prove to the others that a copy survived, and the
-# group would deadlock waiting on each other. A node with room to copy everything does not
-# wait for a wave: it is never unable to serve, so it is not part of that problem.
-wave_hours = 24
-
-# Hours between a node committing to what it will keep and deleting the old store.
-# Never shorter than 4: that is what the answerability window needs.
-retire_delay_hours = 4
-
-# Free space, in MiB, the copier leaves untouched on top of disk_reserve_mb.
-copier_slack_mb = 2048
-
-# Copy rate ceiling, in MiB/s. Keep it modest: an unthrottled copier competing with the
-# audit responder for disk turns a storage migration into an audit incident.
-copier_throttle_mib_per_sec = 32
+# There is deliberately no [storage.migration] section. Every one of its settings belongs
+# to the build rather than to this file: the release that later deletes the old store has
+# to be able to assume every node ran the same schedule, and it cannot assume that while
+# the schedule belongs to whoever last edited a config. A [storage.migration] section left
+# over from an earlier release still parses, it is simply ignored.
 
-# --- Upgrade ---
 [upgrade]
 enabled = false
 channel = "stable"

deploy/terraform/cloud-init/worker.yml

@@ -79,7 +79,12 @@ write_files:
       Type=simple
       User=ant
       Group=ant
-      ExecStart=$${BINARY_PATH} --root-dir $${NODE_DIR} --port 0 --metrics-port $${METRICS_PORT} $${BOOTSTRAP_ARGS}
+      # --enable-logging is not optional here, whatever it looks like. Without it the
+      # binary installs no tracing subscriber and emits nothing at all, so a node on this
+      # unit is silent: no migration progress, no warnings, no way to tell a fleet that has
+      # finished moving off the old chunk store from one that has not. JSON because what
+      # reads these lines is a query, and the fields that matter are structured.
+      ExecStart=$${BINARY_PATH} --root-dir $${NODE_DIR} --port 0 --metrics-port $${METRICS_PORT} --enable-logging --log-format json $${BOOTSTRAP_ARGS}
       Restart=always
       RestartSec=10
       MemoryMax=350M

docs/adr/ADR-0014-file-based-chunk-store-and-lmdb-retirement.md

@@ -370,6 +370,49 @@ belief carry its own expiry — the directory carries its mark, the proof carrie
 saw, the write carries its note — rather than to check again and hope the check is close
 enough to the act.
 
+## Amendment: the schedule belongs to the build
+
+Shipped after this record's release, as a patch on top of it.
+
+`MigrationConfig` is now `serde(skip)` on every field. A `[storage.migration]` section left in
+an operator's file still parses and is ignored. The release that later deletes the old store
+has to be able to assume every node ran the same schedule, and it cannot assume that while the
+schedule belongs to whoever last edited a config. `wave_hours` goes from 24 to 12 so the whole
+schedule fits inside the window before that release.
+
+Two behaviour changes follow from it and are intended, not accidents:
+
+- A node whose file says `enabled = false` migrates anyway. Nobody may opt out.
+- A node an operator had paused resumes with its hold and its waves already spent, because the
+  marker is stamped when the store opens rather than when the copier starts. Nothing
+  distinguishes it from a node whose disk filled on day one, and restamping both would add
+  three days of waiting to the nodes with the least room. What is lost is the stagger, not the
+  safety: a chunk is still only given up once all but one of the close group has proven it
+  holds a copy. Pinned by a test.
+
+Halving the wave spacing only ever moves a node earlier. Both schedules share the same base and
+the same wave index and only the multiplier shrinks, so no wave that had opened closes again.
+While the fleet is mixed the two schedules can put different waves in the same slot, which
+costs churn and not data.
+
+The one path that deleted a chunk without a possession check is gone. When the file store
+refused a legacy value for exceeding the per-chunk ceiling, the copier deleted it outright.
+That case cannot occur — `MAX_CHUNK_SIZE` has been 4 MiB since ant-protocol's first commit,
+the chunk store and its size check arrived together in v0.4.0, replication arrived already
+checking on receive and fetch, and `FileStore::put` checks the address before the size — but a
+branch that destroys a chunk it cannot replace does not get to rely on being unreachable. The
+generic error path now handles it: refuse, name the key and the size, release the volume lock,
+retry. Pinned by a test, because an impossible case is the one a later reader tidies away.
+
+`storage_empty` in the commitment rotation now means no bytes anywhere, not "nothing left to
+commit to". Two nodes reported an empty set while holding data: one that could not copy
+anything before its disk filled, whose keys are all still in the legacy store, and one whose
+last readable file has gone transiently bad, since `all_keys` drops anything marked suspect or
+known-wrong. Both took the `clear_all` branch and dropped every retained root, and an auditor
+holding a valid pin then got `UnknownCommitment`, graded as a confirmed failure, on a node that
+could have answered. Emptiness is now asked of `current_chunks`, the union of the raw file
+index and the legacy-only set, and a read error reads as not-empty.
+
 ## Consequences
 
 ### Positive

src/config.rs

@@ -613,25 +613,50 @@ mod tests {
 
     #[test]
     fn the_shipped_storage_config_parses() {
-        // The migration section is operator-facing, so a typo in it would only surface on
-        // a node that had already shipped. Only `[storage]` is checked: the rest of
-        // `production.toml` does not currently deserialize as a `NodeConfig` (its
-        // `evm_network` is a bare string where an internally tagged enum is expected),
-        // which is a separate, pre-existing problem.
+        // Only `[storage]` is checked: the rest of `production.toml` does not currently
+        // deserialize as a `NodeConfig` (its `evm_network` is a bare string where an
+        // internally tagged enum is expected), which is a separate, pre-existing problem.
         let raw = include_str!("../config/production.toml");
         let doc: toml::Value = toml::from_str(raw).expect("production.toml must be valid TOML");
         let storage = doc.get("storage").expect("a [storage] section").clone();
         let config: StorageConfig = storage.try_into().expect("[storage] must deserialize");
 
-        assert!(config.migration.enabled);
-        assert!(config.migration.dual_write_legacy);
-        assert_eq!(config.migration.shed_hold_hours, 72);
-        assert_eq!(config.migration.copier_throttle_mib_per_sec, 32);
-        assert_eq!(config.migration.copier_slack_mb, 2048);
-        // The release switches are absent from the file on purpose, so they come from the
-        // build rather than from whatever an operator's config last recorded.
-        let build = MigrationConfig::default();
-        assert_eq!(config.migration.retire_legacy, build.retire_legacy);
+        // Every migration setting comes from the build. The shipped file no longer carries
+        // a `[storage.migration]` section at all, and would be ignored if it did.
+        assert_eq!(config.migration, MigrationConfig::default());
+    }
+
+    /// A config file cannot put a node on a schedule of its own.
+    ///
+    /// This is the whole guarantee the release that deletes the old chunk store rests on:
+    /// it may assume every node ran the same migration schedule, and it may only assume
+    /// that while no file can change one. The obvious way to break this is to give one of
+    /// these fields a `serde(default)` again, which looks like a tidy-up and is not, so it
+    /// is worth a test rather than a comment. Every value below is deliberately different
+    /// from the build's.
+    #[test]
+    fn a_config_file_cannot_change_the_migration_schedule() {
+        let hostile = r"
+enabled = true
+[migration]
+enabled = false
+dual_write_legacy = false
+allow_shed = false
+shed_hold_hours = 9999
+retire_delay_hours = 9999
+wave_hours = 9999
+copier_slack_mb = 1
+copier_throttle_mib_per_sec = 1
+tick_secs = 9999
+batch_chunks = 1
+";
+        let config: StorageConfig =
+            toml::from_str(hostile).expect("an old or hostile file must still parse");
+        assert_eq!(
+            config.migration,
+            MigrationConfig::default(),
+            "the file changed the migration schedule, which the final release cannot survive"
+        );
     }
 
     #[test]

src/node.rs

@@ -317,6 +317,23 @@ impl NodeBuilder {
             }
         }
 
+        // Say on the wire whether this node still has an old chunk store. It costs no new
+        // message and no new field: saorsa-core already sends a user agent with every signed
+        // message and keeps each peer's, so this is a different value in a string that was
+        // already there. It is how the release that deletes the old store finds out whether
+        // the fleet has finished, including the nodes we do not run and have no logs from.
+        //
+        // Read from the filesystem here rather than from the store, because the store is
+        // built later and a node with storage switched off never builds one at all, while
+        // the directory on its disk is just as real either way.
+        //
+        // Fixed for the life of the process: saorsa-core copies the string when it builds
+        // the transport. A node that finishes migrating goes on saying `legacy` until it
+        // restarts, which overstates how much is left rather than understating it, and is
+        // the direction a release gate should err in.
+        let signal = crate::storage::MigrationSignal::from_disk(&config.root_dir);
+        core_config.custom_user_agent = Some(crate::storage::migration_signal::user_agent(signal));
+
         // Persist close group peers + trust scores across restarts.
         // Default to root_dir (alongside node_identity.key) when not explicitly set.
         core_config.close_group_cache_dir = Some(
@@ -442,8 +459,12 @@ impl NodeBuilder {
         }
 
         if config.upgrade.staged_rollout_hours > 0 {
-            monitor =
-                monitor.with_staged_rollout(node_id_seed, config.upgrade.staged_rollout_hours);
+            monitor = monitor
+                .with_staged_rollout(node_id_seed, config.upgrade.staged_rollout_hours)
+                // Under the node's own root, not the machine-wide upgrade cache: this is one
+                // node's place in one window, and the cache is shared by every node on the
+                // host. Without it the window restarts whenever the node does.
+                .with_rollout_state(&config.root_dir);
         }
 
         monitor
@@ -655,6 +676,24 @@ impl RunningNode {
             info!("Replication engine started");
         }
 
+        // Say where this node is in the storage migration, and what it can see around it.
+        //
+        // The release that deletes the old chunk store may only be published once the fleet
+        // has moved, and no calendar establishes that. Our own logs cover the nodes we run;
+        // this covers the ones every node can see, which includes the ones we do not run and
+        // would otherwise have no view of at all.
+        {
+            // Weak on purpose: see `report_until_shutdown`. A reporter that kept the node
+            // alive would keep its port bound after the node was dropped.
+            let p2p = Arc::downgrade(&self.p2p_node);
+            let root_dir = self.config.root_dir.clone();
+            let shutdown = self.shutdown.clone();
+            tokio::spawn(async move {
+                crate::storage::migration_signal::report_until_shutdown(p2p, root_dir, shutdown)
+                    .await;
+            });
+        }
+
         // Start upgrade monitor if enabled
         if let Some(monitor) = self.upgrade_monitor.take() {
             let events_tx = self.events_tx.clone();

src/replication/mod.rs

@@ -9980,7 +9980,29 @@ async fn rebuild_and_rotate_commitment(
     // this filter the pruner's reprieve would keep re-committing stale keys
     // forever (the rebuild reads all_keys, so a retained-on-disk key would be
     // re-committed and re-gossiped every rotation — a permanent pin).
-    let storage_empty = stored_keys.is_empty();
+    // "Empty" has to mean this node holds no bytes at all, not that it has nothing left to
+    // COMMIT to. Once the migration has settled, `committable_keys` narrows to the file-backed
+    // set, so a node that could not copy anything before its disk filled reports an empty set
+    // while a full legacy environment sits beside it. Reading that as "the bytes are gone"
+    // takes the `clear_all` branch below and drops every retained root, and an auditor still
+    // holding a valid pin then gets `UnknownCommitment`, which is graded as a confirmed
+    // failure. That node can answer the challenge: the bytes are right there in the legacy
+    // store. It is the `retire_current` branch it belongs in.
+    // Asked of what is physically on disk, not of what this node will currently answer for.
+    // The two differ: `all_keys` drops a file the store has marked suspect or known-wrong, so
+    // a node whose last readable file has gone transiently bad has an empty committable set
+    // while the bytes are still there and may be readable again in a moment. So does a node
+    // that could not copy anything before its disk filled, whose keys are all still in the
+    // legacy store. Reading either as "the bytes are gone" takes the `clear_all` branch below
+    // and drops every retained root, and an auditor holding a valid pin then gets
+    // `UnknownCommitment`, graded as a confirmed failure, on a node that could have answered.
+    //
+    // `current_chunks` is the union of the raw file index, suspect entries included, and the
+    // legacy-only set, which is exactly the question. An error reads as not-empty on purpose:
+    // being wrong that way costs a `retire_current` where `clear_all` would have done, which
+    // stops advertising the root and stays answerable until the gossip TTL lapses. Being wrong
+    // the other way costs a trust penalty on a node that did nothing.
+    let storage_empty = stored_keys.is_empty() && storage.current_chunks().unwrap_or(1) == 0;
     let self_id = *p2p.peer_id();
     let mut keys = Vec::with_capacity(stored_keys.len());
     for k in stored_keys {