fix(secrets): the ninety-day key nag had no series to read, and its fallback was already satisfied (#400) - #403
Merged
Merged
Conversation
…allback was already satisfied (#400) SecretsKeyBackupUnproven has read homelab_key_recipient_last_proof_ timestamp_seconds since ADR-0024 landed (e511590), and that series is written only by scripts/key-recipients.sh --record — which was called by exactly two things: a human proof from removable media, and add-recipient.sh. On the monitoring host the last proof ran at 04:41Z on 2026-09-04, nineteen hours before the emitter existed, so it wrote the old job series and nothing else. The file was never written, the rule had no left side, and it went silent the day it was deployed. THE FALLBACK IT NAMED COULD NOT SPEAK. The rule comment said ScheduledJobNeverRan covered the time before any verification. It covers the time before any verification EVER — it looks for the job series — and this host had one. So the handoff landed on nothing, the tests fed the rule a populated series and could not see it either, and the estate read cleaner than it had before the ADR: the old single- series rule would have fired on 2026-12-03, the new one would not. WHAT MAKES THE FILE EXIST. A sixteenth job, recipient-state, runs key-recipients.sh --record daily through the wrapper with no --proved, so it carries every existing proof forward and sets none — it cannot make a key look proved, only make an unproved one visible. It needs no root, no network and no private key, because the recipient list is the plaintext sops: metadata of the encrypted file. The installer's priming loop writes the file at install time, so a fresh install has the series before its first night. The job is named recipient-state and not key-recipients on purpose: the wrapper writes <job>.prom and the collector writes key-recipients.prom, and #360 is what happens when those are the same file. THE FIRST WRITE INHERITS THE OLD PROOF, with one recipient only. Before the ADR the job series was the recipient's proof — the ADR says so — so a host with one recipient and an old proof gets that timestamp rather than 0. Seeding 0 would turn a proof that happened into an alert saying it never did. With two or more the old series cannot say which key it was, and every recipient starts at 0, which is the ADR's own reading. Exercised against a scratch textfile directory: inherit on first write, carry on second, 0 with no old proof, and two wrapper runs without tripping the #360 guard. WHAT NOTICES IF THE FILE IS MISSING ANYWAY. SecretsKeyRecipientsUnrecorded: the verify-key-backup declaration row `unless on(instance)` the recipient series, for 1h. An unless against the row and not an absent(), for the reason the file header gives — it carries homelab_job and instance like every other rule there. It is the second rule in the file that names a job, and the header now says both are the one job that is not a timer. The existing "not its business" test case is exactly the #400 state (job series present, no recipient series); it now asserts the new rule fires there, paired with the two-recipient case asserting it stays silent. Counted claims move: 78 Prometheus rules, 95 in total, 57 tested, all six in backup.test.yaml; check_docs.py agrees. schedule-maintenance.md also gains the gateway-state row it was missing since #353. NOT DONE HERE: the host. converge runs report-only and install-timers needs sudo there, so until `make install-timers` is run on prometheus the new unit is not installed and the series still does not exist — and SecretsKeyRecipientsUnrecorded will say so from the first deploy, which is the point. The Lemmiwinks page that records this estate's jobs (infrastructure/scheduled_jobs) counts fifteen and should count sixteen once the unit is live. Refs #400, #106, #294, ADR-0024. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
A sixteenth scheduled job,
recipient-state, writes the per-recipient age-key proof series daily soSecretsKeyBackupUnprovenhas something to read; a new rule,SecretsKeyRecipientsUnrecorded, fires when the ninety-day deadline is declared and that series does not exist; and the first write of the file inherits the pre-ADR-0024 proof when there is exactly one recipient.Why
#400. Since ADR-0024 landed the nag reads
homelab_key_recipient_last_proof_timestamp_seconds, which only a human proof run oradd-recipient.shwrote. The monitoring host proved its key nineteen hours before that series existed, so the file was never written, the rule had no left side, and the fallback it named (ScheduledJobNeverRan) was satisfied by the old proof. The proof would have aged out on 2026-12-03 with nothing to say so. Read fromoracleagainst the Prometheus API on 2026-09-08: host checkout atmain,behind_commits0, recipient series absent.The rule's own comment claimed the fallback covered this; it covers the never-verified-ever case only. The test suite fed the rule a populated series and could not see the gap; the existing "not its business" case turns out to be exactly the #400 state and now asserts the new rule fires there.
The job is
recipient-state, notkey-recipients, because the wrapper writes<job>.promand the collector writeskey-recipients.prom— the #360 collision. It runs with no root, no network and no private key: the recipient list is the plaintextsops:metadata.Blast radius
The monitoring host only: one new timer as
robo, one new.promfile, one new alert rule. Nothing changes on the network.secrets/*.sops.yamlVerification
make validatepasses from a worktree onoracle, with one caveat stated rather than hidden: the dashboard round-trip check timed out at its 90-second default because Grafana 13's fresh-database migrations take about 2½ minutes on this laptop's disk; re-run withBOOT_SECONDS=360it passes all three assertions. Dashboards are untouched by this branch. The two on-host checks skip as designed off the deployment checkoutmake lintpassespromtool test rules: six rules inbackup.rules.yaml, all cases pass, including the paired firing/silent cases for the new rulescripts/install-timers.sh --check:recipient-state: *-*-* 09:15:00 every 86400s, alerts at 172800skey-recipients.sh --recordagainst a scratch textfile directory: inherits1788497914fromverify-key-backup.promon first write, carries it on the second, writes0with no old proof; two runs throughrun-scheduled.sh --job recipient-statewriterecipient-state.promandkey-recipients.promside by side without tripping the Patch state is collected on the monitoring host only — oracle, Saruman and morpheus have none #360 guardconvergeis report-only andinstall-timersneeds sudo on the host. After merge:make install-timersonprometheus(it primes the new unit), then confirmhomelab_key_recipient_last_proof_timestamp_secondsreads1788497914andSecretsKeyRecipientsUnrecordedis not firing. Until then the new rule fires on the first deploy, which is correct.schedule-maintenance.md(job table incl. the missinggateway-staterow, the nag paragraph, two troubleshooting rows),back-up-the-age-key.md,successor-handover.md,observability.md, and the rule counts inREADME.md,security.md,stacks/observability/README.md,deploy-stack.md—check_docs.pyreports 78 + 17Refs #400, #106, #294, ADR-0024.
🤖 Generated with Claude Code