Skip to content

fix(connectivity): accept the ledger v3 preview as satisfying the v3 gate - #532

Open
sylr wants to merge 4 commits into
mainfrom
fix/connectivity-ledger-v3-preview-gate
Open

fix(connectivity): accept the ledger v3 preview as satisfying the v3 gate#532
sylr wants to merge 4 commits into
mainfrom
fix/connectivity-ledger-v3-preview-gate

Conversation

@sylr

@sylr sylr commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Summary

Enabling the Connectivity module on a stack running the Ledger v3 preview (ledger.v3.preview-version Setting) was impossible: the connectivity reconciler gated on the ledger's resolved module version alone, which stays on v2 in preview mode by design. The gate therefore always reported connectivity requires a Ledger v3 and — being a hard gate — tore down the delegated Connectivity, the GatewayHTTPAPI route, and the god-mode ledger Credentials on every reconcile, even though the v3 gRPC endpoint connectivity binds to is exactly what the preview provisions. Observed on stack jdxmvkvwlyiy-pepj (eks-acme-dev-euw1-01).

  • Add ledgers.HasV3(ctx, stack, ledgerVersion): true for a v3 module version or an active v3 preview. It reuses the ledger reconciler's own preview resolution, so both modules always agree — the Setting is ignored when the Ledger Operator CRD is unavailable, and values at or below v3.0.0-alpha are rejected.
  • Use it in the connectivity v3 gate and in ledgerGateClosed (the teardown decision on the capability-unavailable path).
  • An error resolving the preview Setting is transient: new LedgerV3PreviewUnresolved condition, pending without teardown, mirroring LedgerVersionUnresolved.
  • The readiness gate needs no change: a preview stack's Ledger only reports Ready once the preview Cluster is ready, and the gRPC backend (ledgers.V3GRPCBackendRef) already resolves identically for preview and full v3.
  • The LedgerNotV3 condition message now reads connectivity requires a Ledger v3 (found "v2.x" and no v3 preview).
  • Documented the interaction in the Ledger module docs' preview section.

Test plan

  • TestHasV3 (ledgers): v3 module version; v2 + preview Setting; v2 without Setting; Setting ignored when the Ledger Operator CRD is unavailable; invalid Setting surfaces an error
  • TestConnectivityReconcilePassesGateWhenLedgerV3PreviewActive: gate passes on a v2 ledger with the preview active, proceeds to credentials provisioning, no teardown
  • TestConnectivityReconcileKeepsDelegatedWhenPreviewGateUnresolved: preview resolution error stays pending without flapping the delegated resources
  • Existing gate tests unchanged and green (hard teardown on plain v2, transient not-ready handling, capability-unavailable paths)
  • go build ./..., make lint, go test ./internal/... (incl. envtest suite) all green
  • End-to-end on jdxmvkvwlyiy-pepj once released

@sylr
sylr requested a review from a team as a code owner September 1, 2026 08:21
@NumaryBot

NumaryBot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

✅ Approve — automated review

No actionable defects were identified. The current code addresses the previously raised gate reachability, readiness, version binding, teardown, and retry concerns.

No findings.

@NumaryBot NumaryBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NumaryBot posted 1 new inline finding.

Summary: #532 (comment)

Comment thread internal/resources/ledgers/exports.go

@flemzord flemzord left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inline review focused on compatibility with the current main branch and reconciliation behavior.

Comment thread internal/resources/connectivities/init.go Outdated
Comment thread internal/resources/connectivities/init.go Outdated
sylr added a commit that referenced this pull request Sep 2, 2026
Address the review findings on #532, on top of the module version
requirements introduced by #527:

- Declare the Ledger requirement presence-only. The previous
  VersionAtLeast(LedgerV3Version) constraint was evaluated by ForModule
  before Reconcile, so a v2 ledger running the v3 preview failed with
  DependencyVersionMismatch and the preview gate was unreachable. The
  effective v3 capability (module version or active preview, including
  the hard teardown) is decided by the stackLedgerHasV3 gate inside
  Reconcile.

- Require reconciled preview readiness before provisioning. The Ledger's
  aggregate status.ready can be stale-true from a v2-only reconcile that
  predates the preview Setting, and the god-mode Credentials turns Ready
  from additionalNamespaces alone (no matched Cluster needed), so neither
  gate stops initial provisioning against a v3 service that does not
  exist yet. When the gate is opened by the preview, additionally require
  the Ledger's LedgerV3PreviewReady condition to be True: pending without
  teardown otherwise, so initial provisioning is blocked while existing
  resources survive transient unavailability.

- Bound the unresolved-preview retry. A Settings read failure produces no
  watch event on recovery, and a pending error without WithRequeueAfter
  is returned as a success with no requeue, leaving Connectivity pending
  indefinitely; poll with a 5s delay instead.

Constraint: the requirements framework only supports static version bounds, so preview awareness cannot be expressed in the Require declaration
Rejected: extending core with a custom requirement predicate | invasive on a freshly merged framework; presence-only + module gate achieves the same with less machinery
Rejected: trusting the credentials-Ready gate to block early provisioning | the ledger operator marks Credentials Ready from additionalNamespaces alone, without a matched Cluster
Confidence: high
Scope-risk: narrow
Not-tested: end-to-end preview + connectivity on a live cluster
@sylr
sylr force-pushed the fix/connectivity-ledger-v3-preview-gate branch from aa3f73e to 6be6ad5 Compare September 2, 2026 09:25
Comment thread internal/resources/ledgers/exports.go Outdated
Comment thread internal/resources/connectivities/init.go
sylr added a commit that referenced this pull request Sep 2, 2026
…d retry unresolved gates

Address the second review round on #532:

- ledgers.V3PreviewReady no longer reads the Ledger's LedgerV3PreviewReady
  condition, which survives a rapid Setting remove/re-add until the ledger
  reconcile persists its cleanup and does not identify which preview version
  it reflects. It now reads the preview Cluster itself: it must exist, carry
  the preview label, not be deleting, be annotated with the currently
  resolved ledger.v3.preview-version (a new annotation stamped by
  createOrUpdateV3Cluster), and be Running at its observed generation.

- ledgerGateClosed surfaces lookup errors instead of collapsing them into
  "gate open". On the capability-unavailable path a possibly-due teardown
  was otherwise skipped with a pending error carrying no requeue — and
  recovery from a Settings read failure emits no watch event — so that path
  now polls with ledgerGateRetryDelay; the unsatisfied-requirements handler
  returns the error so the framework retries.

Constraint: Ledger CR status (ready flag or conditions) is stale across Setting changes; only the Cluster itself reflects the currently reconciled preview
Rejected: stamping the preview version into the condition message | free-text parsing; the Cluster annotation is structured and updated atomically with the spec
Rejected: comparing the Setting version to the Cluster's spec.image.tag | image override Settings can legitimately pin a different tag
Confidence: high
Scope-risk: narrow
Not-tested: end-to-end preview remove/re-add on a live cluster
sylr added 4 commits September 7, 2026 15:11
…gate

Enabling connectivity on a stack running the ledger.v3.preview-version
preview was impossible: the connectivity module gated on the ledger's
resolved module version alone, which stays on v2 in preview mode by
design, so the reconciler reported "connectivity requires a Ledger v3"
and tore down the delegated resources on every reconcile — even though
the v3 gRPC endpoint connectivity binds to is exactly what the preview
provisions (observed on stack jdxmvkvwlyiy-pepj, eks-acme-dev-euw1-01).

Add ledgers.HasV3, which accepts a v3 module version or an active v3
preview, mirroring the ledger reconciler's own decision (the Setting is
ignored when the Ledger Operator CRD is unavailable, and values at or
below v3.0.0-alpha are rejected), and use it for the connectivity gate
and the capability-unavailable teardown decision. An error resolving
the preview Setting is transient (LedgerV3PreviewUnresolved): pending
without teardown, like an unresolvable module version. The readiness
gate needs no change: a preview stack's Ledger only reports Ready once
the preview Cluster is ready.

Constraint: preview mode keeps the ledger module version on v2 by design
Rejected: gating on the LedgerV3PreviewReady condition | status-derived and racy; the Setting is the source of truth the ledger reconciler itself uses
Rejected: reading the Setting directly in the connectivity module | would disagree with the ledger reconciler when the Ledger Operator CRD is absent
Confidence: high
Scope-risk: narrow
Directive: ledgerHasV3 is a package var only so connectivity tests can stub the preview branch, which depends on the ledger controller's startup capability discovery — keep production code assigning it exactly once
Not-tested: end-to-end preview + connectivity on a live cluster
Address the review findings on #532, on top of the module version
requirements introduced by #527:

- Declare the Ledger requirement presence-only. The previous
  VersionAtLeast(LedgerV3Version) constraint was evaluated by ForModule
  before Reconcile, so a v2 ledger running the v3 preview failed with
  DependencyVersionMismatch and the preview gate was unreachable. The
  effective v3 capability (module version or active preview, including
  the hard teardown) is decided by the stackLedgerHasV3 gate inside
  Reconcile.

- Require reconciled preview readiness before provisioning. The Ledger's
  aggregate status.ready can be stale-true from a v2-only reconcile that
  predates the preview Setting, and the god-mode Credentials turns Ready
  from additionalNamespaces alone (no matched Cluster needed), so neither
  gate stops initial provisioning against a v3 service that does not
  exist yet. When the gate is opened by the preview, additionally require
  the Ledger's LedgerV3PreviewReady condition to be True: pending without
  teardown otherwise, so initial provisioning is blocked while existing
  resources survive transient unavailability.

- Bound the unresolved-preview retry. A Settings read failure produces no
  watch event on recovery, and a pending error without WithRequeueAfter
  is returned as a success with no requeue, leaving Connectivity pending
  indefinitely; poll with a 5s delay instead.

Constraint: the requirements framework only supports static version bounds, so preview awareness cannot be expressed in the Require declaration
Rejected: extending core with a custom requirement predicate | invasive on a freshly merged framework; presence-only + module gate achieves the same with less machinery
Rejected: trusting the credentials-Ready gate to block early provisioning | the ledger operator marks Credentials Ready from additionalNamespaces alone, without a matched Cluster
Confidence: high
Scope-risk: narrow
Not-tested: end-to-end preview + connectivity on a live cluster
…d retry unresolved gates

Address the second review round on #532:

- ledgers.V3PreviewReady no longer reads the Ledger's LedgerV3PreviewReady
  condition, which survives a rapid Setting remove/re-add until the ledger
  reconcile persists its cleanup and does not identify which preview version
  it reflects. It now reads the preview Cluster itself: it must exist, carry
  the preview label, not be deleting, be annotated with the currently
  resolved ledger.v3.preview-version (a new annotation stamped by
  createOrUpdateV3Cluster), and be Running at its observed generation.

- ledgerGateClosed surfaces lookup errors instead of collapsing them into
  "gate open". On the capability-unavailable path a possibly-due teardown
  was otherwise skipped with a pending error carrying no requeue — and
  recovery from a Settings read failure emits no watch event — so that path
  now polls with ledgerGateRetryDelay; the unsatisfied-requirements handler
  returns the error so the framework retries.

Constraint: Ledger CR status (ready flag or conditions) is stale across Setting changes; only the Cluster itself reflects the currently reconciled preview
Rejected: stamping the preview version into the condition message | free-text parsing; the Cluster annotation is structured and updated atomically with the spec
Rejected: comparing the Setting version to the Cluster's spec.image.tag | image override Settings can legitimately pin a different tag
Confidence: high
Scope-risk: narrow
Not-tested: end-to-end preview remove/re-add on a live cluster
@sylr
sylr force-pushed the fix/connectivity-ledger-v3-preview-gate branch from 92804a5 to 69026af Compare September 7, 2026 13:11
@shipfox-ai

shipfox-ai Bot commented Sep 7, 2026

Copy link
Copy Markdown

This PR correctly implements the core ask: a stack running Ledger v2 with the ledger.v3.preview-version Setting now satisfies Connectivity's v3 gate, with the preview Setting resolution mirroring the ledger reconciler's own decision (including ignoring it when the Ledger Operator CRD is unavailable), a new LedgerV3PreviewNotReady gate preventing initial provisioning against a not-yet-running (or stale-versioned) preview Cluster, transient-resolution errors handled with pending-without-teardown plus a bounded requeue, and the hard teardown preserved for genuine downgrades. I verified the trigger/retry semantics of the new no-requeue pending states (recovery is event-driven via the ledger reconciler's Cluster watch and the module framework's dependency watches — no stall) and found no functional defect. The remaining findings are documentation drift between the PR description, the design docs, and the shipped code. Recommendation: approve with comments — please amend the PR description and the module-dependencies doc before merge.

Standards

1. docs/10-Development/02-Module dependencies.md:17-22 is now stale and contradicts the registration change. The doc records the confirmed declaration "Connectivity requires Ledger >= v3.0.0-0" and gives the Ledger-v3-consumer pattern as Require(&v1beta1.Ledger{}, VersionAtLeast("v3.0.0-0"), Ready()), listing "checks inside each reconciler" as "rejected as the default". This PR replaces that with presence-only Require(&v1beta1.Ledger{}) (internal/resources/connectivities/init.go:926-931) plus in-reconcile gating via stackLedgerHasV3 (init.go:118). Consequence: DependenciesSatisfied now reports RequirementsSatisfied for a plain v2 Ledger with no preview, so the framework condition no longer surfaces the v3 incompatibility (the teardown itself still happens via the in-reconcile hard gate at init.go:210-222). A future contributor following the documented pattern would reintroduce VersionAtLeast and break the preview path — exactly the failure the in-code comment at init.go:921-925 warns about. Please update the doc to describe the presence-only registration and the in-reconcile preview exception.

No other confirmed material standards finding: the duplication/stackLedgerHasV3-bools/test-helper observations raised by the two reviewers are maintainability judgement calls with no correctness, security, compatibility, or test-risk impact, and are not retained.

Spec

1. PR-description bullet 5 ("The readiness gate needs no change") is contradicted by the shipped code. Commits 4d2f693 and 466600e add an entirely new readiness gate: ledgers.V3PreviewReady (internal/resources/ledgers/exports.go:57) reads the preview Cluster directly (deletion timestamp, formance.com/ledger-v3-preview-version annotation vs the currently resolved Setting, observed-generation check), and Reconcile blocks with a new LedgerV3PreviewNotReady condition (internal/resources/connectivities/init.go:240-247) until it passes. The behaviour is defensible and well-tested (TestConnectivityReconcileBlocksProvisioningUntilPreviewReady — the Ledger's aggregate readiness can be stale-true on a v2 ledger, and the god-mode Credentials turn Ready from additionalNamespaces alone), but the stated rationale no longer matches the implementation. The PR body should be amended so reviewers can tell the divergence is intentional.

2. The promised ledgers.HasV3 API and TestHasV3 test do not exist. PR bullet 2 promises "Add ledgers.HasV3(ctx, stack, ledgerVersion)" and the test plan names TestHasV3 as checked. The capability shipped instead as ledgers.V3PreviewActive (exports.go:39) + ledgers.V3PreviewReady (exports.go:57) plus the package-local stackLedgerHasV3 (init.go:118), tested as TestV3PreviewActive/TestV3PreviewReady (which do cover all five listed cases). The behavioural intent is fully met and the module-requirements change from VersionAtLeast(LedgerV3Version) to presence-only is necessary and justified (a version constraint would reject a v2+preview stack before Reconcile could consult the Setting), but the named public API and test in the description were never realised. Update the PR description to the final shape.

No other confirmed material spec finding: the implementation matches the authoritative commit messages, the LedgerNotV3 message and Ledger docs section match the spec verbatim, and the generated settings.catalog.json line refresh is required repo hygiene, not scope creep.

Reviewed independently by GLM (glm-5.3-flash) and DeepSeek (deepseek-v4-pro-0813) via Shipfox; verified and synthesized by GLM.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Development

Successfully merging this pull request may close these issues.

4 participants