Fix Elasticsearch finalization after ILM cleanup

[viliakov]

Handle ILM retention without weakening restore-start detection: ordinary STS indices anchor completion, while missing data-stream generations may be an oldest prefix or the entire expired set. Lifecycle-only snapshots are rejected as unsafe to monitor.

Also updates golang.org/x/crypto to v0.56.0 for the current HIGH CVE fixes.

Validation: full tests, vet, build, lint, module verification, source and five-platform binary govulncheck, and GoReleaser artifact verification.

Fixes #39

[ai-collaboration-app]

Approach looks right: separating ILM-removable generations from mandatory indices, requiring the missing set to be an oldest contiguous prefix, and keeping every present index's primaries required. Four points inline — 1 and 2 are the ones I'd want resolved.

Checked by building the branch and probing complete() directly with the real backing-index naming; existing tests pass.

Nits: %d oldest data-stream backing indices reads wrong at 1, and make([]string, 0) differs from the var style in restore.go.

[viliakov]

Latest commit a3224b9 addresses the review and security findings:

• Uses ordinary STS indices as the restore-completion anchor while allowing all ILM-managed backing indices to have expired.
• Rejects lifecycle-only snapshots because they cannot be monitored safely without a positive restore anchor.
• Resets the stall timer only when active primaries increase.
• Adds realistic data-stream names and coverage for full ILM expiration, lifecycle-only snapshots, and decreasing primary counts.
• Updates golang.org/x/crypto from v0.55.0 to v0.56.0 for the reported HIGH CVE fixes.

All CI checks now pass, including tests, lint, govulncheck, Trivy, Grype, and VEX.

[ai-collaboration-app]

Re-reviewed at a3224b9. All four earlier findings are addressed and the anchor reasoning now holds up: deleteAllSTSIndices deletes and verifies removal of every STS index before the restore, so a present required index really does prove the restore's cluster-state update applied — which is what makes the new lifecyclePresent == 0 path sound. Realistic backing-index names in the fixtures also lock in the ordering assumption.

Three things left, one of them a regression from the fix to my earlier stall-timer point.

The > comparison turns an ILM deletion into a permanent ceiling: I reproduced a stall failure while primaries were still climbing. That is the highest-value item. The lifecycle-only rejection and the retention wording are judgement calls worth a second look before merge.

[viliakov]

Implemented the safer ILM handling in d816ff3:

• Rebase shard-progress tracking when ILM removes restored backing indices, without treating deletion as progress or resetting the inactivity timeout.
• Reject lifecycle-only snapshots before confirmation, scale-down, index deletion, or restore, because there is no stable required-index completion anchor.
• Keep the current write index out of completion checks: the restore workflow creates it during pre-restore rollover, so it can exist before snapshot restoration starts.
• Use one exact data-stream backing-index predicate for filtering and classification, excluding sibling streams.
• Added regression tests for ILM deletion followed by real shard progress, timeout behavior, lifecycle-only preflight, and exact-prefix matching.

Local tests, vet, build, lint, GoReleaser validation, module verification, and vulnerability checks pass. The review threads were answered and resolved.

[ai-collaboration-app]

Re-reviewed at d816ff3. The stall fix is confirmed working: my original repro — sts_topology at 3/10 primaries alongside a restored 8-shard backing index, ILM deleting it while topology climbs 3 -> 5 -> 7 -> 10, with a 1ns no-progress timeout — now runs through to SUCCESS instead of failing. Two consecutive deletions also survive. Rebaselining and skipping the stall check on the incomparable poll is the right shape, and the two new tests pin both halves of it.

Moving the lifecycle-only rejection ahead of confirmation, scale-down and index deletion resolves my concern properly. The objection I was raising was the blast radius — a snapshot rejected after the destructive steps left workloads down with only --finalize-only as an exit. Failing before anything is touched removes that entirely.

Your point about the write index is correct and I was wrong to suggest it: deleteAllSTSIndices computes its deletion list before calling RolloverDatastream, so the rollover-created write index is not in that list and survives into the restore. It would have been a false anchor. Worth keeping that reasoning in the commit message — it is the kind of thing that gets re-proposed later.

I also accept the retention-wording reply. The required-index anchor does establish that the metadata was applied, and naming every expired generation would be noise against the count you already log.

One new thing, from narrowing the predicate — low likelihood but it turns a working restore into a hard failure. Full suite passes locally; CI on d816ff3 was still running when I looked.

[viliakov]

Addressed the remaining index-selection review in 34db3b3. The restore request now uses the exact snapshot indices selected by the same predicate used for deletion and completion monitoring, so sibling data streams are excluded even when an existing ConfigMap contains the broader .ds-sts_k8s_logs* value. Added regression coverage for that deployed configuration case.

All review threads are resolved and CI passes.

[ai-collaboration-app]

Re-reviewed at 34db3b3. This is a better fix than the one I suggested: deriving the restore list from the snapshot with the same predicate means deletion, restore and completion monitoring cannot disagree by construction, and it fixes already-deployed ConfigMaps without a migration — which narrowing the glob would not have done. Verified the sibling stream is excluded and the joined string is what RestoreSnapshot receives; build and the cmd/elasticsearch + config suites pass locally, CI green on the commit.

That closes every finding from all three earlier rounds. One loose end left behind by this change, config hygiene rather than a defect.

[ai-collaboration-app]

Re-reviewed at 93e8ca7. All findings from the earlier rounds are resolved and I have no blockers.

Verified locally: build and the full test suite pass, and the sibling data-stream exclusion still holds through the new pattern intersection. CI green on the commit.