fix(secrets): the ninety-day key nag had no series to read, and its fallback was already satisfied (#400)

Makefile

@@ -310,6 +310,16 @@ pkg-state: ## Collect package state from morpheus over SSH (FreeBSD, runs as rob
 	@# `pfSense-upgrade -c`, whose output has no format contract.
 	./scripts/collect-pkg-state.sh --ssh $(FW_USER)@$(FW_HOST) --host morpheus
 
+.PHONY: recipient-state
+recipient-state: ## Record which age recipients can open the secrets, and when each was proved (#400)
+	@# Reads the recipient list out of the encrypted file's own metadata and
+	@# writes one series per recipient into the textfile dir, carrying every
+	@# existing proof timestamp forward and setting none. Needs no key: the
+	@# `sops:` block is plaintext. This is what SecretsKeyBackupUnproven reads;
+	@# without it a host that proved its key before ADR-0024 has no series and
+	@# the ninety-day nag is silent. secrets-verify-backup is what sets a proof.
+	./scripts/key-recipients.sh --record --stack $(STACK)
+
 .PHONY: smart-state-remote
 smart-state-remote: ## Collect SMART health from morpheus over SSH (runs as robo)
 	@# morpheus is FreeBSD with no node_exporter and no textfile directory, but

README.md

@@ -58,7 +58,7 @@ documents for different readers.
   can't run an agent (firewall, switch, UPS, iLO). One agent config, deployed
   identically everywhere. [How](docs/architecture.md#observability-data-flow)
 - **Dashboards and alerting as code.** 7 provisioned dashboards, 141 panels, and
-  94 alert rules — 77 metric-based in Prometheus, 17 log-based in Loki — sharing
+  95 alert rules — 78 metric-based in Prometheus, 17 log-based in Loki — sharing
   one Alertmanager routing tree. No dashboard exists only in a database.
 - **Secrets encrypted in-repo with SOPS + age.** Per-device credentials,
   decrypted at deploy time into gitignored paths, with `git log` showing which
@@ -175,7 +175,7 @@ Full topology and data flow in [`docs/architecture.md`](docs/architecture.md).
 .
 ├── stacks/observability/     # the deployed stack — one compose file, eight services
 │   ├── compose.yaml
-│   ├── prometheus/           # config, file_sd targets, 77 alert rules
+│   ├── prometheus/           # config, file_sd targets, 78 alert rules
 │   ├── alertmanager/         # routing and inhibition
 │   ├── loki/                 # single-binary config + 17 LogQL rules
 │   ├── alloy/                # the agent config directory, shipped to every host

docs/observability.md

@@ -304,7 +304,7 @@ separates a quiet stream from a stopped one.
 
 ## Alerting
 
-94 rules in total: 77 metric-based in `prometheus/rules/`, and 17 log-based in
+95 rules in total: 78 metric-based in `prometheus/rules/`, and 17 log-based in
 `loki/rules/`.
 
 ### Log-based (Loki ruler)
@@ -411,7 +411,7 @@ argument and for what to do when it exits 1.
 
 ### Metric-based (Prometheus)
 
-77 rules across eleven files in `prometheus/rules/`:
+78 rules across eleven files in `prometheus/rules/`:
 
 | File | Covers |
 | --- | --- |
@@ -423,7 +423,7 @@ argument and for what to do when it exits 1.
 | `watchdog.rules.yaml` | One rule that always fires, so that its absence is detectable |
 | `blackbox.rules.yaml` | Whether an endpoint can actually be reached, from outside the service, and how many days its certificate has left — Grafana verified against the lab CA, the APC card's self-signed one read but not trusted, the wiki, Prometheus, Loki, Alertmanager and the switch UI over plain http. The iLO and pfSense UIs are written into `targets/blackbox.yaml` and left disabled: each needs a firewall pass from `10.0.99.20` that is a segmentation decision, not a monitoring one ([#91](https://github.com/Gerrrt/HomeLab/issues/91)) |
 | `dns.rules.yaml` | Whether the house is still filtering DNS, asked directly at AdGuard Home on port 53 rather than through pfSense — a probe sent down the normal resolver path always passes, because Unbound's fallback is doing its job. [ADR-0010](adr/0010-keep-the-resolver-on-the-gateway.md) made losing the filter silent on purpose, and these two rules are what distinguishes "this site was never on a list" from "AdGuard has been dead for three weeks". Warning, not critical: nothing is down and nobody is blocked. The targets are written into `targets/blackbox-dns.yaml` and left disabled until [#102](https://github.com/Gerrrt/HomeLab/issues/102) builds the mini PC ([#126](https://github.com/Gerrrt/HomeLab/issues/126)) |
-| `backup.rules.yaml` | Whether the scheduled maintenance jobs are still being run at all — staleness, failure, and never-ran |
+| `backup.rules.yaml` | Whether the scheduled maintenance jobs are still being run at all — staleness, failure, never-ran, and whether the age-key proof record exists to be held to its deadline |
 | `deploy.rules.yaml` | Whether this host is running what the repository says — an uncommitted edit made on the host, a revision that did not verify, and how far behind `main` the host is. Reads the record `scripts/converge.sh` writes hourly ([#99](https://github.com/Gerrrt/HomeLab/issues/99), [ADR-0021](adr/0021-converge-on-a-timer-instead-of-deploying-over-ssh.md)) |
 | `ids.rules.yaml` | Whether Suricata is running on each interface it is declared for, read from the firewall's process table over SNMP — the process metric `security.rules.yaml` says a log rule cannot be ([#90](https://github.com/Gerrrt/HomeLab/issues/90)) |
 
@@ -434,11 +434,11 @@ as loaded and healthy and could not fire for any input ([#63](https://github.com
 `prometheus/tests/*.test.yaml` holds `promtool test rules` unit tests, which
 feed a rule synthetic series and assert it fires — paired with a case asserting
 it stays quiet, because a test that only ever expects silence would have passed
-against the broken rule too. Coverage is fifty-six rules of 77 so far — the five
+against the broken rule too. Coverage is fifty-seven rules of 78 so far — the five
 in `blackbox.rules.yaml`, both in `dns.rules.yaml`, `ContainerHighMemory`,
 `ContainerNearMemoryLimit`, `ContainerRestartLoop`, `ContainerCpuThrottled` and
 `PrometheusSizeRetentionActive`, `Watchdog`, the three iLO rules from
-[#76](https://github.com/Gerrrt/HomeLab/issues/76), all five in
+[#76](https://github.com/Gerrrt/HomeLab/issues/76), all six in
 `backup.test.yaml`, all five in `deploy.test.yaml`, `RemoteWriteJobStale`,
 `SuricataStopped`, the two gateway rules from
 [#353](https://github.com/Gerrrt/HomeLab/issues/353), and all eighteen in
@@ -593,10 +593,11 @@ a textfile the node exporter already scrapes:
 
 The threshold each job is held to is a *fifth* series,
 `homelab_job_max_age_seconds`, written by `scripts/install-timers.sh` from the
-same table that decides the cadence. That is what lets the five rules in
-`prometheus/rules/backup.rules.yaml` cover every job without naming any of them,
-and what makes `make check-timers` able to assert that a threshold is at least
-twice its timer's real period.
+same table that decides the cadence. That is what lets the rules in
+`prometheus/rules/backup.rules.yaml` cover every job without naming any of them
+— the two that do name one both concern `verify-key-backup`, the human proof,
+below — and what makes `make check-timers` able to assert that a threshold is
+at least twice its timer's real period.
 
 Two things are deliberate and easy to undo by accident:
 
@@ -629,6 +630,17 @@ to `oracle` and fails if it cannot, so its failure alert doubles as "the config
 has stopped leaving this host". The volume sets do not leave; that is
 [#92](https://github.com/Gerrrt/HomeLab/issues/92).
 
+That series has to exist for the nag to mean anything, and for four days it did
+not ([#400](https://github.com/Gerrrt/HomeLab/issues/400)): it was written only
+by a proof run or by adding a recipient, and this host had proved its key before
+the series was invented, so the rule went quiet the day it was deployed while
+the fallback it named — `ScheduledJobNeverRan` — was satisfied by the old proof.
+The `recipient-state` timer now writes the recipient list daily, carrying proofs
+forward and setting none, and `SecretsKeyRecipientsUnrecorded` fires when the
+deadline row exists and the recipient series does not — an `unless` against the
+declaration row, not an `absent()`, so it carries labels like every other rule
+in the file.
+
 Installing, tuning and troubleshooting all of it:
 [`runbooks/schedule-maintenance.md`](runbooks/schedule-maintenance.md).
 

docs/runbooks/back-up-the-age-key.md

@@ -205,9 +205,23 @@ removable media, and no timer can mount that.
 So it is enforced from the other end. `make secrets-verify-backup` records the
 timestamp of a successful run **against the recipient it just proved**, and
 `SecretsKeyBackupUnproven` fires when any recipient's proof passes ninety days
-old — routed to the normal alert channel like any other warning. Until the first
-verification there is no timestamp at all and `ScheduledJobNeverRan` says so
-instead, which is the honest reading of a key backup nobody has ever tested.
+old — routed to the normal alert channel like any other warning. A recipient
+that has never been proved is recorded as `0`, so it fires with an absurd age
+rather than being invisible, which is the honest reading of a key backup nobody
+has ever tested.
+
+That record has to exist for any of it to work, and until
+[#400](https://github.com/Gerrrt/HomeLab/issues/400) nothing guaranteed it did:
+the per-recipient series was written only by a proof run or by adding a
+recipient, so a host that had proved its key *before* the series was invented
+never wrote it, and the nag was silent from the day it was deployed. The
+`recipient-state` timer now writes the recipient list every day — carrying every
+existing proof forward and setting none, since it never touches a key — and
+`SecretsKeyRecipientsUnrecorded` fires if the deadline is declared and the file
+is missing anyway. On a host with exactly one recipient, the first write
+inherits the pre-ADR-0024 proof from the old `verify-key-backup` series rather
+than starting from never; with more than one, the old series cannot say which
+key it was, and every recipient starts at `0`.
 
 The per-recipient part is [ADR-0024](../adr/0024-hold-a-second-age-recipient-and-prove-each-one-separately.md)
 and it only starts to matter once there is more than one. With one recipient

docs/runbooks/deploy-stack.md

@@ -118,7 +118,7 @@ Then in the UI:
 
 1. **Prometheus → Status → Targets.** Every job `UP`. The four `snmp` targets
    take up to 45 seconds on their first scrape.
-2. **Prometheus → Status → Rules.** 77 rules loaded, none in error. The
+2. **Prometheus → Status → Rules.** 78 rules loaded, none in error. The
    page counts 57: the extra one is `homelab_suricata_expected_interface`,
    the stack's only recording rule. Everything counted in this repository is
    alert rules, so the two numbers differ by one and always have.

docs/runbooks/schedule-maintenance.md

@@ -23,8 +23,14 @@ The one job that genuinely proves off-host recoverability is
 a timer: [`verify-key-backup.sh`](../../scripts/verify-key-backup.sh) refuses the
 live key by device and inode, precisely so that what gets tested is a copy on
 removable media. No timer can mount that. So it is enforced from the other end —
-a successful run records its timestamp, and `SecretsKeyBackupUnproven` fires when
-that proof passes ninety days old.
+a successful run records its timestamp against the recipient it proved, and
+`SecretsKeyBackupUnproven` fires when that proof passes ninety days old. The
+`recipient-state` timer is what makes that series exist at all: it writes one
+row per recipient every day, carrying proofs forward and setting none, because
+a host that had proved its key before the per-recipient series existed had
+nothing else that would ever write it, and the nag was silent for as long as
+that lasted ([#400](https://github.com/Gerrrt/HomeLab/issues/400)).
+`SecretsKeyRecipientsUnrecorded` fires if the file is missing anyway.
 
 One job's output leaves this host: `backup-firewall` copies every export to
 `oracle` and **fails if it cannot**, so its `ScheduledJobFailed` also means "the
@@ -65,6 +71,8 @@ the host.
 | `smart-state` | `make smart-state` | daily 08:30 | 2 days |
 | `smart-state-remote` | `make smart-state-remote` | daily 08:45 | 2 days |
 | `pkg-state` | `make pkg-state` | daily 09:00 | 2 days |
+| `recipient-state` | `make recipient-state` | daily 09:15 | 2 days |
+| `gateway-state` | `make gateway-state` | every 15 minutes | 90 minutes |
 | `verify-key-backup` | **you**, `make secrets-verify-backup KEY=…` | no timer | 90 days |
 
 Thresholds are roughly twice the period, never once: a threshold equal to the
@@ -517,6 +525,7 @@ expected rather than a second fault.
 | Symptom | Cause | Fix |
 | --- | --- | --- |
 | `ScheduledJobNeverRan` right after install | The job has a threshold declared and has never reported a result | Expected for `verify-key-backup` until you first verify the key. For anything else, `systemctl start homelab-<job>.service` and read the journal |
+| `SecretsKeyRecipientsUnrecorded` | The ninety-day deadline is declared and no recipient has a proof series, so `SecretsKeyBackupUnproven` cannot fire however stale the proof is | `systemctl start homelab-recipient-state.service`. If that unit does not exist the timers predate [#400](https://github.com/Gerrrt/HomeLab/issues/400): `make install-timers` adds it and primes it. On a host with one recipient the first write inherits the old `verify-key-backup` proof rather than starting from never |
 | `ScheduledJobMetricsAbsent` | Nothing from the textfile directory has reached Prometheus in six hours | This is the whole directory, not one file — check Alloy is up and the directory still exists. A single malformed file shows as `node_textfile_scrape_error 1` and costs only that file |
 | One job's series missing, `node_textfile_scrape_error` is 1 | That job's `.prom` failed to parse — a truncated write, or something wrote it without the temp-then-rename | The other files are unaffected. Re-run the job; if it recurs, something is writing the file directly instead of through `run-scheduled.sh` |
 | Every `homelab_job_*` series missing, no scrape error | The `textfile` block in `config.alloy` is not reading the right path | It must carry the `ALLOY_ROOTFS` prefix (`/rootfs` in the container). `rootfs_path` does **not** apply to that argument, and a wrong path reports an empty directory rather than an error |
@@ -532,7 +541,7 @@ expected rather than a second fault.
 | `check-versions` exits 1 | A document names an OS version the host is not running | Not an outage — nothing is broken. Read the FAIL lines: each names the document, the cell and what the host reports. Correct the document; the box is the source of truth. A `SKIP` for `morpheus` instead means `sysDescr` is not reaching Prometheus, which is a collection fault rather than a clean bill of health |
 | `docker info` fails only under systemd | The unit is missing `SupplementaryGroups=docker` | A login shell picks the group up from `/etc/group` and a unit does not, which is why this never reproduces by hand |
 | Timers exist but never fire | `WantedBy=timers.target` missing, or the timers were never enabled | `systemctl list-timers 'homelab-*'` shows nothing; re-run `make install-timers` |
-| `ScheduledJobMetricsAbsent` fires and nothing else in `backup.rules.yaml` ever has | This step was never run at all | `systemctl list-unit-files 'homelab*'` reports *0 unit files* and `/var/lib/node_exporter/textfile_collector` does not exist. The four other rules here join against a series `--install` writes, so none of them can fire — that alert is the only one that can, and it is doing its job ([#215](https://github.com/Gerrrt/HomeLab/issues/215)). Run `make install-timers` |
+| `ScheduledJobMetricsAbsent` fires and nothing else in `backup.rules.yaml` ever has | This step was never run at all | `systemctl list-unit-files 'homelab*'` reports *0 unit files* and `/var/lib/node_exporter/textfile_collector` does not exist. The other rules here join against a series `--install` writes, so none of them can fire — that alert is the only one that can, and it is doing its job ([#215](https://github.com/Gerrrt/HomeLab/issues/215)). Run `make install-timers` |
 | `converge` fails every hour with a signature error | GitHub's signing key was never imported into `robo`'s keyring, so nothing on this host can verify | The one-time import in [`converge-the-host.md`](converge-the-host.md) §Set it up. Every other job here is unaffected |
 | `refusing to install from …` | You are in a worktree or a second clone | The units hardcode the deployment path. Install from `/home/robo/code/Gerrrt/HomeLab` |
 | `make validate` fails on the schedule | A cadence and its threshold disagree | `make check-timers` names the job and both numbers. Fix the `JOBS` table or the `.timer`, not the alert |

docs/runbooks/successor-handover.md

@@ -193,7 +193,7 @@ inherits without knowing.
 | --- | --- | --- |
 | **Alert delivery to a destination you do not own** | Immediately, and silently | You do not. This is step 2 above, and it is the reason it is step 2 |
 | **The external heartbeat watcher** — a free-tier cron-monitor on somebody else's account | Whenever that account lapses | Nothing here can tell you. A watcher on this host would fail with the thing it watches, which is why it is off-host and therefore outside anything this repository can check |
-| **The age key backup goes unproven** | 90 days after the last verification, *per recipient* | `SecretsKeyBackupUnproven`, routed to the normal alert channel, naming the recipient — proving one copy does not clear another ([ADR-0024](../adr/0024-hold-a-second-age-recipient-and-prove-each-one-separately.md)). Until the first verification there is no timestamp at all and `ScheduledJobNeverRan` says so instead |
+| **The age key backup goes unproven** | 90 days after the last verification, *per recipient* | `SecretsKeyBackupUnproven`, routed to the normal alert channel, naming the recipient — proving one copy does not clear another ([ADR-0024](../adr/0024-hold-a-second-age-recipient-and-prove-each-one-separately.md)). A recipient never proved is recorded as never, and fires; `SecretsKeyRecipientsUnrecorded` fires instead if the record itself is missing ([#400](https://github.com/Gerrrt/HomeLab/issues/400)) |
 | **Grafana's leaf certificate** | 825 days from issue; the APC card's own certificate expires on its own clock | `TlsCertificateExpiringSoon` at 30 days, `TlsCertificateExpiryImminent` at 7 — read off the served handshake by `blackbox-exporter`, not off a file. Let it lapse and `up{job="grafana"}` goes to 0 as well |
 | **The UPS battery pack** | A pack was fitted 2026-08-28 and passed its self-test; packs are consumables and this one is on a biweekly test schedule | `UpsSelfTestFailed` and `UpsBatteryUnproven` key on the self-test result, which is the single honest signal this card emits — every charge, runtime and alarm value it reports was fabricated while the bay was empty. Two things remain open: the card's test *schedule* is unwatched ([#249](https://github.com/Gerrrt/HomeLab/issues/249)), and `upsBasicBatteryLastReplaceDate` still reads a pre-fit date, so it is not a usable record of the pack's age |
 | **Mains power to the monitoring path** | Any cut | The rack is on the UPS; the switch carrying `prometheus` and `oracle` is not, so both laptops keep running and go deaf. Stated in [`security.md`](../security.md#threat-model) |