Skip to content

fix: retire control-plane pods before Trino rollout assertions - #1181

Open
bill-ph wants to merge 1 commit into
mainfrom
codex/investigate-trino-rollout
Open

fix: retire control-plane pods before Trino rollout assertions#1181
bill-ph wants to merge 1 commit into
mainfrom
codex/investigate-trino-rollout

Conversation

@bill-ph

@bill-ph bill-ph commented Sep 12, 2026

Copy link
Copy Markdown
Collaborator

An idle control-plane pod could exit before Service routing stopped selecting it. The first API request after a successful Deployment rollout then received a TCP reset and the Trino harness mislabeled it as an unexpected legacy cell.

Add a five-second preStop routing-withdrawal allowance to the isolated deployment, preserving its previous 30-second drain budget with a 35-second termination grace period. Each Trino configuration transition now captures the previous pod names and waits for those exact pods to terminate before making the unchanged one-shot assertion. Transport and content failures remain fatal and receive distinct diagnostics. The README and rollout runbook document the lifecycle and recovery procedure.

This changes the isolated E2E deployment and harness; it does not change the application binary or production chart.

Validation:

  • TDD: new retirement and actual-shell-phase regressions failed before the fix and pass afterward; just test-mw-fixtures passes.
  • just lint passes with CI's pinned golangci-lint 2.11.4 and Go 1.25.4.
  • Instrumented live baseline reproduced the post-rollout request reaching the exited old pod and receiving TCP RST.
  • The same reduced fixture with this deployment and the actual shell helpers passed: all 876 continuous Service health probes succeeded, and the first post-retirement authenticated cells request returned only cell-test.
  • Live validation used an empty config store and configured cells without Trino servers; full Trino behavior remains covered by the PR's E2E lane. Diagnostic namespaces were deleted.

@github-actions

Copy link
Copy Markdown

Test Impact Plan

Deterministic summary of how this PR changes tests, CI runners, and coverage-risk signals.

Summary

Area Added Changed Deleted
Test files 2 3 0
E2E/journey files 0 0 0
Workflow files 0 0 0

Signals

  • Test cases: +2 / -0
  • Assertions: +22 / -0
  • Skips or known failures added: 0
  • Workflow continue-on-error added: 0
  • Workflow path filters added: 0
  • Test commands removed from justfile: 0
  • E2E/journey retry lines added: 0

Coverage risk: neutral or increased

No coverage-reduction warnings detected.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant