SecretsKeyBackupUnproven is the only thing that says the age-key proof has gone stale, and since #106's implementation landed it has had nothing to read. Found from oracle against the Prometheus HTTP API on 2026-09-08, not from the host.
What the monitoring host reads
homelab_key_recipient_last_proof_timestamp_seconds (no series)
homelab_job_last_success_timestamp_seconds{homelab_job="verify-key-backup"}
1788497914 = 2026-09-04 04:41:54Z
homelab_job_max_age_seconds{homelab_job="verify-key-backup"} 7776000
homelab_deploy_revision_info{host="prometheus"} revision="17ce90aa03ac" (= main)
homelab_deploy_behind_commits{host="prometheus"} 0
ALERTS{alertname=~"SecretsKeyBackupUnproven|ScheduledJobNeverRan"} (empty)
The host's checkout is at main, so this is not a stale tree. The series the rule joins on has never been written.
Why
e5115901 (2026-09-04 23:15Z) moved SecretsKeyBackupUnproven from the job series to homelab_key_recipient_last_proof_timestamp_seconds, per ADR-0024. That series is written by scripts/key-recipients.sh --record, which is called from exactly two places:
scripts/verify-key-backup.sh §7 — a human run with removable media mounted;scripts/add-recipient.sh — adding a recipient.
Neither has run since e5115901 merged. The last proof was 2026-09-04 04:41Z, nineteen hours before the emitter existed, so it wrote the old job series and nothing else. install-timers.sh names the script only in a comment; no timer calls it.
The rule's own comment anticipates the empty-file case and hands it off:
ScheduledJobNeverRan still speaks for the case before ANY verification has happened, because key-recipients.sh has not written its file yet and there are no per-recipient series to fire on.
That holds only if no verification has ever happened. Here one had — under the old series — so homelab_job_last_success_timestamp_seconds{homelab_job="verify-key-backup"} exists, ScheduledJobNeverRan (max_age unless last_success) is satisfied, and the handoff lands on nothing. ADR-0024's "a recipient that has never been verified is recorded as 0, not omitted" is true of the file once it exists; the file itself is omitted until a human runs the proof.
stacks/observability/prometheus/tests/backup.test.yaml feeds the rule a populated series, so the tests cannot see this either.
What it costs
The 2026-09-04 proof ages past ninety days on 2026-12-03, and nothing will say so — the old series is excluded from ScheduledJobStale on purpose, and the new rule has no input. This is the .env failure ADR-0024 used as its own example, one layer up: the proof exists, and nothing here can see whether it is going stale.
It also means the estate currently reads cleaner than before e5115901: the previous rule would have fired on the old series on 2026-12-03; the new one will not.
How it could resolve
Two shapes, not exclusive:
- Seed the file from a timer.
key-recipients.sh --record with no --proved writes every current recipient with its prior timestamp, or 0 — it is safe to run unattended and carries forward existing proofs. Running it from install-timers.sh --install, or from the nightly verify-backups unit, means the series exists from the first deploy. Note the seed would write 0 for the one recipient that was proved on 2026-09-04, because that proof predates the file; the alert would then fire as "never proved" until the next mounted-media run. Loud is the right side to fail on here, and the description already tells the operator what to do, but it is worth knowing before the first run.
- Guard the absence. An
absent_over_time(homelab_key_recipient_last_proof_timestamp_seconds[6h]) rule, on the same footing as ScheduledJobMetricsAbsent — with the caveat backup.rules.yaml's header gives about absent() and label synthesis. That catches this case and the case where the .prom file is deleted later.
Either way, the ScheduledJobNeverRan sentence in the rule comment should say what it actually covers.
Refs #106, #294, ADR-0024. Recorded on the Lemmiwinks wiki at infrastructure/scheduled_jobs §3, which will be corrected to point here — it currently attributes the gap to converge running report-only, which the deploy metrics above rule out.
🤖 Generated with Claude Code
SecretsKeyBackupUnprovenis the only thing that says the age-key proof has gone stale, and since #106's implementation landed it has had nothing to read. Found fromoracleagainst the Prometheus HTTP API on 2026-09-08, not from the host.What the monitoring host reads
The host's checkout is at
main, so this is not a stale tree. The series the rule joins on has never been written.Why
e5115901(2026-09-04 23:15Z) movedSecretsKeyBackupUnprovenfrom the job series tohomelab_key_recipient_last_proof_timestamp_seconds, per ADR-0024. That series is written byscripts/key-recipients.sh --record, which is called from exactly two places:scripts/verify-key-backup.sh§7 — a human run with removable media mounted;scripts/add-recipient.sh— adding a recipient.Neither has run since
e5115901merged. The last proof was 2026-09-04 04:41Z, nineteen hours before the emitter existed, so it wrote the old job series and nothing else.install-timers.shnames the script only in a comment; no timer calls it.The rule's own comment anticipates the empty-file case and hands it off:
That holds only if no verification has ever happened. Here one had — under the old series — so
homelab_job_last_success_timestamp_seconds{homelab_job="verify-key-backup"}exists,ScheduledJobNeverRan(max_age unless last_success) is satisfied, and the handoff lands on nothing. ADR-0024's "a recipient that has never been verified is recorded as 0, not omitted" is true of the file once it exists; the file itself is omitted until a human runs the proof.stacks/observability/prometheus/tests/backup.test.yamlfeeds the rule a populated series, so the tests cannot see this either.What it costs
The 2026-09-04 proof ages past ninety days on 2026-12-03, and nothing will say so — the old series is excluded from
ScheduledJobStaleon purpose, and the new rule has no input. This is the.envfailure ADR-0024 used as its own example, one layer up: the proof exists, and nothing here can see whether it is going stale.It also means the estate currently reads cleaner than before
e5115901: the previous rule would have fired on the old series on 2026-12-03; the new one will not.How it could resolve
Two shapes, not exclusive:
key-recipients.sh --recordwith no--provedwrites every current recipient with its prior timestamp, or0— it is safe to run unattended and carries forward existing proofs. Running it frominstall-timers.sh --install, or from the nightlyverify-backupsunit, means the series exists from the first deploy. Note the seed would write0for the one recipient that was proved on 2026-09-04, because that proof predates the file; the alert would then fire as "never proved" until the next mounted-media run. Loud is the right side to fail on here, and the description already tells the operator what to do, but it is worth knowing before the first run.absent_over_time(homelab_key_recipient_last_proof_timestamp_seconds[6h])rule, on the same footing asScheduledJobMetricsAbsent— with the caveatbackup.rules.yaml's header gives aboutabsent()and label synthesis. That catches this case and the case where the.promfile is deleted later.Either way, the
ScheduledJobNeverRansentence in the rule comment should say what it actually covers.Refs #106, #294, ADR-0024. Recorded on the Lemmiwinks wiki at
infrastructure/scheduled_jobs§3, which will be corrected to point here — it currently attributes the gap to converge running report-only, which the deploy metrics above rule out.🤖 Generated with Claude Code