influxdb3 node lifecycle docs - #7625
Open
jstirnaman wants to merge 15 commits into
Open
Conversation
Add a Node lifecycle page for InfluxDB 3 Core and Enterprise that documents how a node moves through the catalog states it registers itself in. Shared content (content/shared/influxdb3-admin/node-lifecycle.md): - Node states (running, stopping, stopped, removing) and node identity (node ID compared to instance ID) - Lifecycle state diagrams, scoped per product - Register, stop, remove, and re-register a node, including the two paths to stopped and why only a graceful stop node records a final snapshot - Conditions under which removal is refused (409) and how to resolve each - Restart compared to removal, the most common way to lose data during routine maintenance - Deploy with an orchestrator: Kubernetes/Helm and Ansible/systemd - Verify node state and troubleshoot lifecycle issues Orchestrated upgrades (content/shared/influxdb3-admin/upgrade.md): - Add Helm and Ansible tabs to the multi-node rolling upgrade procedure. The Helm chart uses a single chart-wide image.tag and a StatefulSet per node mode, so a plain helm upgrade rolls every mode at once and does not follow the recommended node upgrade order. Document sequencing rollouts with updateStrategy.rollingUpdate.partition. - Add troubleshooting entries for out-of-order Helm rollouts, nodes that do not return to running after a rollout, and catalog entries that multiply when a deployment assigns unstable node IDs. Cross-link the lifecycle page from clustering, recover-node, the Kubernetes install page, and the remove/show/stop node CLI references, and link the lifecycle and upgrade troubleshooting sections to each other. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WUNvDmPaYvcFX5rchCiYhF
Two failure modes an orchestrated deployment hits that the lifecycle and upgrade guidance didn't yet address: - helm rollback reverts the image tag but can't revert the v2-to-v3 catalog migration that 3.10 performs on first start, so a rollback leaves pods crash-looping on a catalog the older binary can't read. Restoring the backed-up catalog objects is the only way back. - Setting a node mode's enabled: false deletes its StatefulSet but leaves its nodes in the catalog, and a compactor node can't be removed at all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WUNvDmPaYvcFX5rchCiYhF
Use the target section heading as link text instead of the page name, matching the other cross-references into the node lifecycle page. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WUNvDmPaYvcFX5rchCiYhF
…open questions Validated the node lifecycle guidance against the InfluxDB catalog source and support history, then corrected what didn't hold up. Corrections: - Remove the claim that a repeat stop always returns 200 OK. Two catalog ops exist with opposite responses for a non-running node (StopNodeOp returns 400 NodeAlreadyStopped; RequestStopNodeOp returns an idempotent 200), and the 400 is what users actually hit. The page previously asserted both. Additions, each traced to the source or to a recurring support pattern: - Removal completes on the compactor's schedule: the compactor waits until its per-node compaction floor passes the node's final snapshot sequence before deleting object-store prefixes and purging the catalog entry, so removing can persist for hours behind a compaction backlog, by design. - A node stays in removing: check compactor progress before assuming the removal is stuck, and don't scale a replacement into a removing node's ID. - Stop a node whose process is already gone by sending the request to a running node with --host. - Strengthen the --force-finalize guidance. Confirmed accurate and left unchanged: the four state names and their semantics, the re-registration matrix, all four removal-refusal conditions and their error strings, remove idempotency, and that removing is terminal for a node ID. Add VERIFY comments marking six claims that need a live instance to settle, including whether stopping a node frees its licensed cores and the database that serves system.pt_compaction_nodes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WUNvDmPaYvcFX5rchCiYhF
Sweep the page for behavioral claims that source validation didn't settle and mark each for testing against a live instance. - Instance ID reuse on restart. The page's promise that an ordinary restart always re-registers depends on a restarting process reusing its existing instance ID, because the catalog only accepts a running or stopping node when the incoming instance ID matches. If a restart mints a new UUID, the re-registration section, the crashed-node troubleshooting entry, and the rolling-restart guidance are all wrong together. - The four-step Core shutdown sequence and its order, which was written from the Enterprise stop cascade rather than Core-specific behavior. - Whether a Core node can ever surface stopping or removing, given the state enum is shared catalog code. - The compactor replacement procedure and whether the compaction lease TTL forces a wait between stopping the old node and starting its replacement. - The system.nodes database and column names, and the catalog sync interval default carried over from the CLI reference. Comments are stripped from rendered output, so they stay author-only. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WUNvDmPaYvcFX5rchCiYhF
Map every VERIFY comment in the node lifecycle page to a runnable test, the outcome that confirms the claim, and the fix to apply if it doesn't. Split by what a license gates: the four Core-testable items need only a throwaway Core node, including the highest-priority question of whether a restarting process reuses its instance ID. The remaining Enterprise items need a multi-node cluster, and two are flagged destructive. Remove before merge; a required PR check blocks PLAN.md on the default branch. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WUNvDmPaYvcFX5rchCiYhF
Readers enter a long page at arbitrary points, so linking a term only on its first use leaves later sections undefined for anyone who scans or deep-links into them. Link all four WAL tail mentions to the data durability reference and all five node mode mentions to the specialized cluster nodes reference. Link body text rather than bold lead-ins where the body repeats the term. Both targets resolve: the WAL tail anchor exists in the Core and Enterprise builds of the durability page, and node mode links are confined to Enterprise-gated blocks because the clustering page is Enterprise-only. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WUNvDmPaYvcFX5rchCiYhF
Apply the same treatment as e1c2538 to the pages that describe the same node lifecycle operations: link every "WAL tail" mention, not just the first one on the page. - stop/node.md: 1 of 4 mentions linked -> 4 of 4 - remove/node.md: 0 of 3 mentions linked -> 3 of 3 (the page never defined the term, despite the WAL tail being the reason removal is guarded) - recover-node.md: 1 of 5 mentions linked -> 5 of 5 Both shared CLI files are sourced by Enterprise only, so /influxdb3/version/ resolves to Enterprise; recover-node.md is Enterprise-specific and keeps the /influxdb3/enterprise/ form already used on that page. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WUNvDmPaYvcFX5rchCiYhF
jstirnaman
added
product:v3-monolith
jstirnaman
marked this pull request as ready for review
August 6, 2026 22:34
jstirnaman
marked this pull request as draft
August 6, 2026 22:34
jstirnaman
marked this pull request as ready for review
August 6, 2026 22:35
jstirnaman
marked this pull request as draft
August 6, 2026 22:36
jstirnaman
changed the title
…rade state Add a "For more information about the upgraded storage engine" pointer where the node lifecycle docs say the unsnapshotted-WAL safeguard requires the upgraded engine. Also answer the question those passages raise but never address: admins are told Parquet and mid-upgrade clusters aren't guarded, with no way to tell which state their cluster is in. Point to the documented signals -- new clusters on 3.11+ default to the upgraded engine, clusters from 3.10 or earlier keep Parquet until restarted with --upgrade-pacha-tree, and system.upgrade_parquet_node reports per-node completion. Covers all three pages that make the "not guarded" claim: remove/node.md, node-lifecycle.md, and recover-node.md. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WUNvDmPaYvcFX5rchCiYhF
Both places that describe what a graceful stop drains said "Parquet: WAL flush". WAL flush is a different operation -- it writes the WAL buffer to WAL files every --wal-flush-interval (default 1s) and runs constantly, so it is not the step a graceful stop has to force. Data durability says buffered writes stay in the WAL "until the next Parquet persistence captures it". Reword both spots to "Parquet engine: Parquet persistence; upgraded engine: WAL snapshot" and flag the claim for engineering and product review, since the docs alone can't settle it. Add a VERIFY comment at each location and a PLAN.md entry noting the two spots must stay in sync if it's reverted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WUNvDmPaYvcFX5rchCiYhF
jstirnaman
commented
Aug 6, 2026
…hand The parenthetical "(Parquet engine: Parquet persistence; upgraded engine: WAL snapshot)" was too compressed to read. Replace it in both spots with a full sentence that says what each engine actually does: on the Parquet engine the node persists buffered writes to Parquet files; on the upgraded storage engine it captures them in a WAL snapshot. Update the VERIFY comments and the PLAN.md entry to match the new form. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WUNvDmPaYvcFX5rchCiYhF
…docs-m9pbew' into claude/influxdb3-node-lifecycle-docs-m9pbew
Follow-ups to merging the WAL tail link removal in recover-node.md: - Rewrap the sentence the unlink left behind; "WAL tail" was stranded alone on its own line from the removed link's line breaks. - Move the VERIFY comment inside step 2 of the recovery procedure. At column 0 between list items it terminated the ordered list, producing two <ol> elements. Goldmark's start=2 kept the visible numbering correct, but the list is now a single <ol> again. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WUNvDmPaYvcFX5rchCiYhF
Contributor
Vale Style Check Results
Warnings (4)
✅ Check passed |
Contributor
🔗 Link Check Results — Link Check Bot✅ All links are valid
|
| Source File | URL | Issue |
|---|---|---|
content/influxdb3/core/admin/node-lifecycle/_index.md |
https://support.influxdata.com/ | Network error: SSL certificate not trusted. Use --insecure if site is trusted (e… |
content/influxdb3/core/admin/upgrade/_index.md |
https://support.influxdata.com/ | Network error: SSL certificate not trusted. Use --insecure if site is trusted (e… |
content/influxdb3/enterprise/admin/clustering/_index.md |
https://support.influxdata.com/ | Network error: SSL certificate not trusted. Use --insecure if site is trusted (e… |
content/influxdb3/enterprise/admin/node-lifecycle/_index.md |
https://support.influxdata.com/ | Network error: SSL certificate not trusted. Use --insecure if site is trusted (e… |
content/influxdb3/enterprise/admin/recover-node/_index.md |
https://support.influxdata.com/ | Network error: SSL certificate not trusted. Use --insecure if site is trusted (e… |
content/influxdb3/enterprise/admin/upgrade/_index.md |
https://support.influxdata.com/ | Network error: SSL certificate not trusted. Use --insecure if site is trusted (e… |
content/influxdb3/enterprise/install/kubernetes/_index.md |
https://reddit.com/r/influxdb | Error (cached) |
content/influxdb3/enterprise/install/kubernetes/_index.md |
https://support.influxdata.com/ | Network error: SSL certificate not trusted. Use --insecure if site is trusted (e… |
content/influxdb3/enterprise/install/kubernetes/_index.md |
https://support.influxdata.com/ | Network error: SSL certificate not trusted. Use --insecure if site is trusted (e… |
content/influxdb3/enterprise/reference/cli/influxdb3/remove/node/_index.md |
https://reddit.com/r/influxdb | Error (cached) |
content/influxdb3/enterprise/reference/cli/influxdb3/remove/node/_index.md |
https://support.influxdata.com/ | Network error: SSL certificate not trusted. Use --insecure if site is trusted (e… |
content/influxdb3/enterprise/reference/cli/influxdb3/show/nodes/_index.md |
https://support.influxdata.com/ | Network error: SSL certificate not trusted. Use --insecure if site is trusted (e… |
content/influxdb3/enterprise/reference/cli/influxdb3/stop/node/_index.md |
https://support.influxdata.com/ | Network error: SSL certificate not trusted. Use --insecure if site is trusted (e… |
Full details: workflow run summary and artifact. Last updated: 2026-08-07 00:07:32 UTC
jstirnaman
marked this pull request as ready for review
August 6, 2026 23:36
Contributor
📦 PR Preview — Preview Bot
Changed pages (10)
Preview auto-deploys on push. Will be cleaned up when PR closes. |
Contributor
Author
|
Review findings that need verification:
|
…nding Two issues with the diagram shortcode's styling: - .edgeLabel set font-size: .85em, but Mermaid nests .edgeLabel inside .edgeLabel, so the multiplier applied twice and nested arrow labels rendered at 11.56px against 16px node labels. Use rem so it applies once. - Mermaid pins an inline max-width to the diagram's natural pixel size, so diagrams filled only ~60% of the article column. The svg carries a viewBox, so lifting the cap scales the whole diagram proportionally; ceiling it at 680px keeps the scale-up from dwarfing body text. Measured in Chromium on the node lifecycle page: arrow labels 11.56/13.6px -> 16px, diagram 504x688 -> 680x928. Avoid CSS min() in the ceiling -- Sass evaluates it and fails the build on mixed px/% units. docs(influxdb3): correct the Helm rollout sequence and partition ceiling - Step 3's comment promised queriers, then compactor, then processing engine, but the command dropped both partitions at once. Process nodes have no ordering requirement per the recommended node upgrade order, so document them rolling together rather than serializing them. - partition=99 left ordinal 99 and above free to update, contradicting the surrounding text. Use a ceiling no deployment reaches and state the rule. - Add the missing rollout wait before `influxdb3 show nodes`, which could otherwise report on pods that had not rolled yet. Flag the chart-specific values keys and component labels for verification. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WUNvDmPaYvcFX5rchCiYhF
jstirnaman
requested review from
jeffreyssmith2nd,
peterbarnett03 and
reidkaufmann
August 7, 2026 00:13
reidkaufmann
approved these changes
Aug 7, 2026
reidkaufmann
left a comment
reidkaufmann
left a comment
Contributor
There was a problem hiding this comment.
Skimmed the helm stuff. Perhaps some places we could say "lose the most recent writes" instead of "WAL tail", but it's a bit of judgment call how much to tailor the language to external effect vs. internal concept (some times the latter is more succinct). I didn't see anything I objected to; approving.
sanderson
approved these changes
Aug 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
content/influxdb3/core/admin/node-lifecycle.mdWhy
Existing Enterprise docs covered
stopandremovenode commands and upgrade troubleshooting, but lacked a clear explanation of the entire node lifecycle.Customers using Helm or Ansible to deploy and manage clusters need to know how to properly bring down/up nodes, WAL and compaction timing, how to check node status.
Mermaid diagram font size was too small:
Example questions:
Impact
Provides guidance and self-help for customers managing nodes and clusters.
PLAN.md file won't be published: it is ephemeral and delete-squashed on PR merge.
Verification
Checklist
npx hugo --quiet)