Skip to content

Fix fresh cluster bootstrap deadlock and stale peer discovery - #13

Merged
alihm merged 1 commit into
masterfrom
development
Aug 13, 2026
Merged

alihm merged 1 commit into
masterfrom
development

Conversation

@alihm

@alihm alihm commented Aug 13, 2026

Copy link
Copy Markdown
Member

Summary

  • treat the agent-owned .etcd3_api breadcrumb as fresh state while continuing to reject real or unknown etcd data
  • use the marker-aware freshness check for both bootstrap eligibility and peer identity responses
  • reload /etc/cluster_env on every etcd join attempt, including retries that began with an empty peer list
  • reload identity credentials and membership per request so a long-running proxy does not serve a stale HMAC or topology
  • add regression tests for marker-only directories, preserved etcd state, topology reload retries, reload failures, and refreshed identity authentication/membership

Validation

  • go test ./...
  • git diff --check

@alihm
alihm merged commit da76a82 into master Aug 13, 2026
3 checks passed
@alihm

alihm commented Aug 14, 2026

Copy link
Copy Markdown
Member Author

Added automatic recovery for complete loss of ephemeral etcd state in commit c111a8c.

Recovery now requires repeated authenticated agreement from every expected node. With multiple PGDATA copies it proceeds only when all etcd directories are empty, every PostgreSQL system ID matches, and exactly one node retains the durable primary role while all others identify as replicas. PGDATA is never deleted by this path; non-authority nodes reset only local etcd and rejoin as learners. Ambiguous roles, multiple primaries, mismatched IDs, residual etcd state, or unreachable peers continue to block.

Also documented persistent Flux storage for /var/lib/etcd and replaced public IP fixtures with RFC 5737 test addresses. Validation: go test ./..., go test -race ./cmd/flux-agent, and git diff --check.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant