From 6056f57e3feea732ba1d2f60f771e77554b81839 Mon Sep 17 00:00:00 2001 From: Your Name Date: Tue, 8 Sep 2026 15:42:14 +0000 Subject: [PATCH] fix(secrets): the ninety-day key nag had no series to read, and its fallback was already satisfied (#400) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit SecretsKeyBackupUnproven has read homelab_key_recipient_last_proof_ timestamp_seconds since ADR-0024 landed (e5115901), and that series is written only by scripts/key-recipients.sh --record — which was called by exactly two things: a human proof from removable media, and add-recipient.sh. On the monitoring host the last proof ran at 04:41Z on 2026-09-04, nineteen hours before the emitter existed, so it wrote the old job series and nothing else. The file was never written, the rule had no left side, and it went silent the day it was deployed. THE FALLBACK IT NAMED COULD NOT SPEAK. The rule comment said ScheduledJobNeverRan covered the time before any verification. It covers the time before any verification EVER — it looks for the job series — and this host had one. So the handoff landed on nothing, the tests fed the rule a populated series and could not see it either, and the estate read cleaner than it had before the ADR: the old single- series rule would have fired on 2026-12-03, the new one would not. WHAT MAKES THE FILE EXIST. A sixteenth job, recipient-state, runs key-recipients.sh --record daily through the wrapper with no --proved, so it carries every existing proof forward and sets none — it cannot make a key look proved, only make an unproved one visible. It needs no root, no network and no private key, because the recipient list is the plaintext sops: metadata of the encrypted file. The installer's priming loop writes the file at install time, so a fresh install has the series before its first night. The job is named recipient-state and not key-recipients on purpose: the wrapper writes .prom and the collector writes key-recipients.prom, and #360 is what happens when those are the same file. THE FIRST WRITE INHERITS THE OLD PROOF, with one recipient only. Before the ADR the job series was the recipient's proof — the ADR says so — so a host with one recipient and an old proof gets that timestamp rather than 0. Seeding 0 would turn a proof that happened into an alert saying it never did. With two or more the old series cannot say which key it was, and every recipient starts at 0, which is the ADR's own reading. Exercised against a scratch textfile directory: inherit on first write, carry on second, 0 with no old proof, and two wrapper runs without tripping the #360 guard. WHAT NOTICES IF THE FILE IS MISSING ANYWAY. SecretsKeyRecipientsUnrecorded: the verify-key-backup declaration row `unless on(instance)` the recipient series, for 1h. An unless against the row and not an absent(), for the reason the file header gives — it carries homelab_job and instance like every other rule there. It is the second rule in the file that names a job, and the header now says both are the one job that is not a timer. The existing "not its business" test case is exactly the #400 state (job series present, no recipient series); it now asserts the new rule fires there, paired with the two-recipient case asserting it stays silent. Counted claims move: 78 Prometheus rules, 95 in total, 57 tested, all six in backup.test.yaml; check_docs.py agrees. schedule-maintenance.md also gains the gateway-state row it was missing since #353. NOT DONE HERE: the host. converge runs report-only and install-timers needs sudo there, so until `make install-timers` is run on prometheus the new unit is not installed and the series still does not exist — and SecretsKeyRecipientsUnrecorded will say so from the first deploy, which is the point. The Lemmiwinks page that records this estate's jobs (infrastructure/scheduled_jobs) counts fifteen and should count sixteen once the unit is live. Refs #400, #106, #294, ADR-0024. Co-Authored-By: Claude Fable 5.1 --- Makefile | 10 +++ README.md | 4 +- docs/observability.md | 30 ++++++--- docs/runbooks/back-up-the-age-key.md | 20 +++++- docs/runbooks/deploy-stack.md | 2 +- docs/runbooks/schedule-maintenance.md | 15 ++++- docs/runbooks/successor-handover.md | 2 +- docs/security.md | 2 +- scripts/install-timers.sh | 11 ++++ scripts/key-recipients.sh | 35 +++++++++++ stacks/observability/README.md | 2 +- .../prometheus/rules/backup.rules.yaml | 61 +++++++++++++++++-- .../prometheus/tests/backup.test.yaml | 45 +++++++++++++- systemd/homelab-recipient-state.service | 52 ++++++++++++++++ systemd/homelab-recipient-state.timer | 16 +++++ 15 files changed, 278 insertions(+), 29 deletions(-) create mode 100644 systemd/homelab-recipient-state.service create mode 100644 systemd/homelab-recipient-state.timer diff --git a/Makefile b/Makefile index 214def3..d26cc8a 100644 --- a/Makefile +++ b/Makefile @@ -310,6 +310,16 @@ pkg-state: ## Collect package state from morpheus over SSH (FreeBSD, runs as rob @# `pfSense-upgrade -c`, whose output has no format contract. ./scripts/collect-pkg-state.sh --ssh $(FW_USER)@$(FW_HOST) --host morpheus +.PHONY: recipient-state +recipient-state: ## Record which age recipients can open the secrets, and when each was proved (#400) + @# Reads the recipient list out of the encrypted file's own metadata and + @# writes one series per recipient into the textfile dir, carrying every + @# existing proof timestamp forward and setting none. Needs no key: the + @# `sops:` block is plaintext. This is what SecretsKeyBackupUnproven reads; + @# without it a host that proved its key before ADR-0024 has no series and + @# the ninety-day nag is silent. secrets-verify-backup is what sets a proof. + ./scripts/key-recipients.sh --record --stack $(STACK) + .PHONY: smart-state-remote smart-state-remote: ## Collect SMART health from morpheus over SSH (runs as robo) @# morpheus is FreeBSD with no node_exporter and no textfile directory, but diff --git a/README.md b/README.md index bcaa542..554476a 100644 --- a/README.md +++ b/README.md @@ -58,7 +58,7 @@ documents for different readers. can't run an agent (firewall, switch, UPS, iLO). One agent config, deployed identically everywhere. [How](docs/architecture.md#observability-data-flow) - **Dashboards and alerting as code.** 7 provisioned dashboards, 141 panels, and - 94 alert rules — 77 metric-based in Prometheus, 17 log-based in Loki — sharing + 95 alert rules — 78 metric-based in Prometheus, 17 log-based in Loki — sharing one Alertmanager routing tree. No dashboard exists only in a database. - **Secrets encrypted in-repo with SOPS + age.** Per-device credentials, decrypted at deploy time into gitignored paths, with `git log` showing which @@ -175,7 +175,7 @@ Full topology and data flow in [`docs/architecture.md`](docs/architecture.md). . ├── stacks/observability/ # the deployed stack — one compose file, eight services │ ├── compose.yaml -│ ├── prometheus/ # config, file_sd targets, 77 alert rules +│ ├── prometheus/ # config, file_sd targets, 78 alert rules │ ├── alertmanager/ # routing and inhibition │ ├── loki/ # single-binary config + 17 LogQL rules │ ├── alloy/ # the agent config directory, shipped to every host diff --git a/docs/observability.md b/docs/observability.md index c9507d5..0e01e6e 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -304,7 +304,7 @@ separates a quiet stream from a stopped one. ## Alerting -94 rules in total: 77 metric-based in `prometheus/rules/`, and 17 log-based in +95 rules in total: 78 metric-based in `prometheus/rules/`, and 17 log-based in `loki/rules/`. ### Log-based (Loki ruler) @@ -411,7 +411,7 @@ argument and for what to do when it exits 1. ### Metric-based (Prometheus) -77 rules across eleven files in `prometheus/rules/`: +78 rules across eleven files in `prometheus/rules/`: | File | Covers | | --- | --- | @@ -423,7 +423,7 @@ argument and for what to do when it exits 1. | `watchdog.rules.yaml` | One rule that always fires, so that its absence is detectable | | `blackbox.rules.yaml` | Whether an endpoint can actually be reached, from outside the service, and how many days its certificate has left — Grafana verified against the lab CA, the APC card's self-signed one read but not trusted, the wiki, Prometheus, Loki, Alertmanager and the switch UI over plain http. The iLO and pfSense UIs are written into `targets/blackbox.yaml` and left disabled: each needs a firewall pass from `10.0.99.20` that is a segmentation decision, not a monitoring one ([#91](https://github.com/Gerrrt/HomeLab/issues/91)) | | `dns.rules.yaml` | Whether the house is still filtering DNS, asked directly at AdGuard Home on port 53 rather than through pfSense — a probe sent down the normal resolver path always passes, because Unbound's fallback is doing its job. [ADR-0010](adr/0010-keep-the-resolver-on-the-gateway.md) made losing the filter silent on purpose, and these two rules are what distinguishes "this site was never on a list" from "AdGuard has been dead for three weeks". Warning, not critical: nothing is down and nobody is blocked. The targets are written into `targets/blackbox-dns.yaml` and left disabled until [#102](https://github.com/Gerrrt/HomeLab/issues/102) builds the mini PC ([#126](https://github.com/Gerrrt/HomeLab/issues/126)) | -| `backup.rules.yaml` | Whether the scheduled maintenance jobs are still being run at all — staleness, failure, and never-ran | +| `backup.rules.yaml` | Whether the scheduled maintenance jobs are still being run at all — staleness, failure, never-ran, and whether the age-key proof record exists to be held to its deadline | | `deploy.rules.yaml` | Whether this host is running what the repository says — an uncommitted edit made on the host, a revision that did not verify, and how far behind `main` the host is. Reads the record `scripts/converge.sh` writes hourly ([#99](https://github.com/Gerrrt/HomeLab/issues/99), [ADR-0021](adr/0021-converge-on-a-timer-instead-of-deploying-over-ssh.md)) | | `ids.rules.yaml` | Whether Suricata is running on each interface it is declared for, read from the firewall's process table over SNMP — the process metric `security.rules.yaml` says a log rule cannot be ([#90](https://github.com/Gerrrt/HomeLab/issues/90)) | @@ -434,11 +434,11 @@ as loaded and healthy and could not fire for any input ([#63](https://github.com `prometheus/tests/*.test.yaml` holds `promtool test rules` unit tests, which feed a rule synthetic series and assert it fires — paired with a case asserting it stays quiet, because a test that only ever expects silence would have passed -against the broken rule too. Coverage is fifty-six rules of 77 so far — the five +against the broken rule too. Coverage is fifty-seven rules of 78 so far — the five in `blackbox.rules.yaml`, both in `dns.rules.yaml`, `ContainerHighMemory`, `ContainerNearMemoryLimit`, `ContainerRestartLoop`, `ContainerCpuThrottled` and `PrometheusSizeRetentionActive`, `Watchdog`, the three iLO rules from -[#76](https://github.com/Gerrrt/HomeLab/issues/76), all five in +[#76](https://github.com/Gerrrt/HomeLab/issues/76), all six in `backup.test.yaml`, all five in `deploy.test.yaml`, `RemoteWriteJobStale`, `SuricataStopped`, the two gateway rules from [#353](https://github.com/Gerrrt/HomeLab/issues/353), and all eighteen in @@ -593,10 +593,11 @@ a textfile the node exporter already scrapes: The threshold each job is held to is a *fifth* series, `homelab_job_max_age_seconds`, written by `scripts/install-timers.sh` from the -same table that decides the cadence. That is what lets the five rules in -`prometheus/rules/backup.rules.yaml` cover every job without naming any of them, -and what makes `make check-timers` able to assert that a threshold is at least -twice its timer's real period. +same table that decides the cadence. That is what lets the rules in +`prometheus/rules/backup.rules.yaml` cover every job without naming any of them +— the two that do name one both concern `verify-key-backup`, the human proof, +below — and what makes `make check-timers` able to assert that a threshold is +at least twice its timer's real period. Two things are deliberate and easy to undo by accident: @@ -629,6 +630,17 @@ to `oracle` and fails if it cannot, so its failure alert doubles as "the config has stopped leaving this host". The volume sets do not leave; that is [#92](https://github.com/Gerrrt/HomeLab/issues/92). +That series has to exist for the nag to mean anything, and for four days it did +not ([#400](https://github.com/Gerrrt/HomeLab/issues/400)): it was written only +by a proof run or by adding a recipient, and this host had proved its key before +the series was invented, so the rule went quiet the day it was deployed while +the fallback it named — `ScheduledJobNeverRan` — was satisfied by the old proof. +The `recipient-state` timer now writes the recipient list daily, carrying proofs +forward and setting none, and `SecretsKeyRecipientsUnrecorded` fires when the +deadline row exists and the recipient series does not — an `unless` against the +declaration row, not an `absent()`, so it carries labels like every other rule +in the file. + Installing, tuning and troubleshooting all of it: [`runbooks/schedule-maintenance.md`](runbooks/schedule-maintenance.md). diff --git a/docs/runbooks/back-up-the-age-key.md b/docs/runbooks/back-up-the-age-key.md index 540b435..e8fa1c8 100644 --- a/docs/runbooks/back-up-the-age-key.md +++ b/docs/runbooks/back-up-the-age-key.md @@ -205,9 +205,23 @@ removable media, and no timer can mount that. So it is enforced from the other end. `make secrets-verify-backup` records the timestamp of a successful run **against the recipient it just proved**, and `SecretsKeyBackupUnproven` fires when any recipient's proof passes ninety days -old — routed to the normal alert channel like any other warning. Until the first -verification there is no timestamp at all and `ScheduledJobNeverRan` says so -instead, which is the honest reading of a key backup nobody has ever tested. +old — routed to the normal alert channel like any other warning. A recipient +that has never been proved is recorded as `0`, so it fires with an absurd age +rather than being invisible, which is the honest reading of a key backup nobody +has ever tested. + +That record has to exist for any of it to work, and until +[#400](https://github.com/Gerrrt/HomeLab/issues/400) nothing guaranteed it did: +the per-recipient series was written only by a proof run or by adding a +recipient, so a host that had proved its key *before* the series was invented +never wrote it, and the nag was silent from the day it was deployed. The +`recipient-state` timer now writes the recipient list every day — carrying every +existing proof forward and setting none, since it never touches a key — and +`SecretsKeyRecipientsUnrecorded` fires if the deadline is declared and the file +is missing anyway. On a host with exactly one recipient, the first write +inherits the pre-ADR-0024 proof from the old `verify-key-backup` series rather +than starting from never; with more than one, the old series cannot say which +key it was, and every recipient starts at `0`. The per-recipient part is [ADR-0024](../adr/0024-hold-a-second-age-recipient-and-prove-each-one-separately.md) and it only starts to matter once there is more than one. With one recipient diff --git a/docs/runbooks/deploy-stack.md b/docs/runbooks/deploy-stack.md index 5ed095d..4ec1eb8 100644 --- a/docs/runbooks/deploy-stack.md +++ b/docs/runbooks/deploy-stack.md @@ -118,7 +118,7 @@ Then in the UI: 1. **Prometheus → Status → Targets.** Every job `UP`. The four `snmp` targets take up to 45 seconds on their first scrape. -2. **Prometheus → Status → Rules.** 77 rules loaded, none in error. The +2. **Prometheus → Status → Rules.** 78 rules loaded, none in error. The page counts 57: the extra one is `homelab_suricata_expected_interface`, the stack's only recording rule. Everything counted in this repository is alert rules, so the two numbers differ by one and always have. diff --git a/docs/runbooks/schedule-maintenance.md b/docs/runbooks/schedule-maintenance.md index 8fc30ab..12e9082 100644 --- a/docs/runbooks/schedule-maintenance.md +++ b/docs/runbooks/schedule-maintenance.md @@ -23,8 +23,14 @@ The one job that genuinely proves off-host recoverability is a timer: [`verify-key-backup.sh`](../../scripts/verify-key-backup.sh) refuses the live key by device and inode, precisely so that what gets tested is a copy on removable media. No timer can mount that. So it is enforced from the other end — -a successful run records its timestamp, and `SecretsKeyBackupUnproven` fires when -that proof passes ninety days old. +a successful run records its timestamp against the recipient it proved, and +`SecretsKeyBackupUnproven` fires when that proof passes ninety days old. The +`recipient-state` timer is what makes that series exist at all: it writes one +row per recipient every day, carrying proofs forward and setting none, because +a host that had proved its key before the per-recipient series existed had +nothing else that would ever write it, and the nag was silent for as long as +that lasted ([#400](https://github.com/Gerrrt/HomeLab/issues/400)). +`SecretsKeyRecipientsUnrecorded` fires if the file is missing anyway. One job's output leaves this host: `backup-firewall` copies every export to `oracle` and **fails if it cannot**, so its `ScheduledJobFailed` also means "the @@ -65,6 +71,8 @@ the host. | `smart-state` | `make smart-state` | daily 08:30 | 2 days | | `smart-state-remote` | `make smart-state-remote` | daily 08:45 | 2 days | | `pkg-state` | `make pkg-state` | daily 09:00 | 2 days | +| `recipient-state` | `make recipient-state` | daily 09:15 | 2 days | +| `gateway-state` | `make gateway-state` | every 15 minutes | 90 minutes | | `verify-key-backup` | **you**, `make secrets-verify-backup KEY=…` | no timer | 90 days | Thresholds are roughly twice the period, never once: a threshold equal to the @@ -517,6 +525,7 @@ expected rather than a second fault. | Symptom | Cause | Fix | | --- | --- | --- | | `ScheduledJobNeverRan` right after install | The job has a threshold declared and has never reported a result | Expected for `verify-key-backup` until you first verify the key. For anything else, `systemctl start homelab-.service` and read the journal | +| `SecretsKeyRecipientsUnrecorded` | The ninety-day deadline is declared and no recipient has a proof series, so `SecretsKeyBackupUnproven` cannot fire however stale the proof is | `systemctl start homelab-recipient-state.service`. If that unit does not exist the timers predate [#400](https://github.com/Gerrrt/HomeLab/issues/400): `make install-timers` adds it and primes it. On a host with one recipient the first write inherits the old `verify-key-backup` proof rather than starting from never | | `ScheduledJobMetricsAbsent` | Nothing from the textfile directory has reached Prometheus in six hours | This is the whole directory, not one file — check Alloy is up and the directory still exists. A single malformed file shows as `node_textfile_scrape_error 1` and costs only that file | | One job's series missing, `node_textfile_scrape_error` is 1 | That job's `.prom` failed to parse — a truncated write, or something wrote it without the temp-then-rename | The other files are unaffected. Re-run the job; if it recurs, something is writing the file directly instead of through `run-scheduled.sh` | | Every `homelab_job_*` series missing, no scrape error | The `textfile` block in `config.alloy` is not reading the right path | It must carry the `ALLOY_ROOTFS` prefix (`/rootfs` in the container). `rootfs_path` does **not** apply to that argument, and a wrong path reports an empty directory rather than an error | @@ -532,7 +541,7 @@ expected rather than a second fault. | `check-versions` exits 1 | A document names an OS version the host is not running | Not an outage — nothing is broken. Read the FAIL lines: each names the document, the cell and what the host reports. Correct the document; the box is the source of truth. A `SKIP` for `morpheus` instead means `sysDescr` is not reaching Prometheus, which is a collection fault rather than a clean bill of health | | `docker info` fails only under systemd | The unit is missing `SupplementaryGroups=docker` | A login shell picks the group up from `/etc/group` and a unit does not, which is why this never reproduces by hand | | Timers exist but never fire | `WantedBy=timers.target` missing, or the timers were never enabled | `systemctl list-timers 'homelab-*'` shows nothing; re-run `make install-timers` | -| `ScheduledJobMetricsAbsent` fires and nothing else in `backup.rules.yaml` ever has | This step was never run at all | `systemctl list-unit-files 'homelab*'` reports *0 unit files* and `/var/lib/node_exporter/textfile_collector` does not exist. The four other rules here join against a series `--install` writes, so none of them can fire — that alert is the only one that can, and it is doing its job ([#215](https://github.com/Gerrrt/HomeLab/issues/215)). Run `make install-timers` | +| `ScheduledJobMetricsAbsent` fires and nothing else in `backup.rules.yaml` ever has | This step was never run at all | `systemctl list-unit-files 'homelab*'` reports *0 unit files* and `/var/lib/node_exporter/textfile_collector` does not exist. The other rules here join against a series `--install` writes, so none of them can fire — that alert is the only one that can, and it is doing its job ([#215](https://github.com/Gerrrt/HomeLab/issues/215)). Run `make install-timers` | | `converge` fails every hour with a signature error | GitHub's signing key was never imported into `robo`'s keyring, so nothing on this host can verify | The one-time import in [`converge-the-host.md`](converge-the-host.md) §Set it up. Every other job here is unaffected | | `refusing to install from …` | You are in a worktree or a second clone | The units hardcode the deployment path. Install from `/home/robo/code/Gerrrt/HomeLab` | | `make validate` fails on the schedule | A cadence and its threshold disagree | `make check-timers` names the job and both numbers. Fix the `JOBS` table or the `.timer`, not the alert | diff --git a/docs/runbooks/successor-handover.md b/docs/runbooks/successor-handover.md index b331721..d03ced1 100644 --- a/docs/runbooks/successor-handover.md +++ b/docs/runbooks/successor-handover.md @@ -193,7 +193,7 @@ inherits without knowing. | --- | --- | --- | | **Alert delivery to a destination you do not own** | Immediately, and silently | You do not. This is step 2 above, and it is the reason it is step 2 | | **The external heartbeat watcher** — a free-tier cron-monitor on somebody else's account | Whenever that account lapses | Nothing here can tell you. A watcher on this host would fail with the thing it watches, which is why it is off-host and therefore outside anything this repository can check | -| **The age key backup goes unproven** | 90 days after the last verification, *per recipient* | `SecretsKeyBackupUnproven`, routed to the normal alert channel, naming the recipient — proving one copy does not clear another ([ADR-0024](../adr/0024-hold-a-second-age-recipient-and-prove-each-one-separately.md)). Until the first verification there is no timestamp at all and `ScheduledJobNeverRan` says so instead | +| **The age key backup goes unproven** | 90 days after the last verification, *per recipient* | `SecretsKeyBackupUnproven`, routed to the normal alert channel, naming the recipient — proving one copy does not clear another ([ADR-0024](../adr/0024-hold-a-second-age-recipient-and-prove-each-one-separately.md)). A recipient never proved is recorded as never, and fires; `SecretsKeyRecipientsUnrecorded` fires instead if the record itself is missing ([#400](https://github.com/Gerrrt/HomeLab/issues/400)) | | **Grafana's leaf certificate** | 825 days from issue; the APC card's own certificate expires on its own clock | `TlsCertificateExpiringSoon` at 30 days, `TlsCertificateExpiryImminent` at 7 — read off the served handshake by `blackbox-exporter`, not off a file. Let it lapse and `up{job="grafana"}` goes to 0 as well | | **The UPS battery pack** | A pack was fitted 2026-08-28 and passed its self-test; packs are consumables and this one is on a biweekly test schedule | `UpsSelfTestFailed` and `UpsBatteryUnproven` key on the self-test result, which is the single honest signal this card emits — every charge, runtime and alarm value it reports was fabricated while the bay was empty. Two things remain open: the card's test *schedule* is unwatched ([#249](https://github.com/Gerrrt/HomeLab/issues/249)), and `upsBasicBatteryLastReplaceDate` still reads a pre-fit date, so it is not a usable record of the pack's age | | **Mains power to the monitoring path** | Any cut | The rack is on the UPS; the switch carrying `prometheus` and `oracle` is not, so both laptops keep running and go deaf. Stated in [`security.md`](../security.md#threat-model) | diff --git a/docs/security.md b/docs/security.md index f85a7f5..cf200ef 100644 --- a/docs/security.md +++ b/docs/security.md @@ -17,7 +17,7 @@ What this network is actually built to survive: | An attacker on the lab segment reaching the hypervisor's BMC | **Accepted.** `shiva` stays on VLAN 30 by decision ([ADR-0033](adr/0033-keep-the-ilo-on-the-lab-segment.md)); a BMC compromise in the lab costs the lab, and the tripwire watches what it initiates | | A range target with a path out | It has none — `ifrit`'s targets sit on a bridge with no physical port, on `172.30.30.0/24`, which the firewall does not route and on which nothing has a default route at all ([ADR-0014](adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md), [ADR-0017](adr/0017-buy-ifrit-for-iops-and-keep-the-range-disposable.md)) | | Someone with the trusted Wi-Fi key quietly joining | Kea's lease log reaches Loki; `UnknownDeviceOnTrustedSegment` fires the first time a MAC appears on VLAN 50 in seven days ([ADR-0019](adr/0019-read-device-joins-from-the-dhcp-server.md)) | -| Losing visibility of a failure | 94 alert rules, 30 days of metrics and logs | +| Losing visibility of a failure | 95 alert rules, 30 days of metrics and logs | | Someone on a reachable VLAN silencing an alert to hide a failure | Alertmanager binds to `127.0.0.1`; silences go through authenticated Grafana | | Mains power loss | **The rack, yes; the monitoring path, no.** A pack fitted to `mjolnir` on 2026-08-28 passed its self-test; the switch carrying `prometheus` and `oracle` still has no battery — see below | | The estate being down while the person who runs it is unavailable | **Documentation, yes; data, not yet.** ADR-0011 puts the emergency tier on paper; [ADR-0023](adr/0023-keep-the-household-recovery-path-outside-the-estate.md) extends the same reasoning to the sensitive tier's data before that tier exists — see below | diff --git a/scripts/install-timers.sh b/scripts/install-timers.sh index c3e1efc..0d74d5c 100755 --- a/scripts/install-timers.sh +++ b/scripts/install-timers.sh @@ -102,6 +102,15 @@ DEPLOY_ROOT="/home/robo/code/Gerrrt/HomeLab" # touching this table, which is the same property every other rule in # backup.rules.yaml has and the reason none of them name a job. # +# recipient-state is what makes that series exist (#400). key-recipients.sh +# used to be run only by a human proof and by add-recipient.sh, so a host that +# had proved its key before ADR-0024 landed never wrote the per-recipient file +# at all — and SecretsKeyBackupUnproven, which reads only that file, went quiet +# the day it was deployed. This row writes the file daily, carrying every +# existing proof forward and setting none; the priming loop below writes it at +# install time. SecretsKeyRecipientsUnrecorded is the rule that notices when +# the file is missing anyway. +# # converge is the only hourly row, and the only one whose threshold is three # times its period rather than two. It shares the `backups` lock with the two # backup jobs, so a run that collides with the weekly archive can legitimately @@ -123,6 +132,7 @@ JOBS=( "smart-state homelab-smart-state 172800 smart-state" "smart-state-remote homelab-smart-state-remote 172800 smart-state-remote" "pkg-state homelab-pkg-state 172800 pkg-state" + "recipient-state homelab-recipient-state 172800 recipient-state" "gateway-state homelab-gateway-state 5400 gateway-state" "verify-key-backup - 7776000 secrets-verify-backup" ) @@ -468,3 +478,4 @@ green "installed — systemctl list-timers 'homelab-*'" info "backup-volumes was NOT primed: it stops the stack. Run it when you can watch:" info " sudo systemctl start homelab-backup-volumes.service" info "verify-key-backup has no timer and never will — docs/runbooks/back-up-the-age-key.md" +info "recipient-state was primed, so SecretsKeyBackupUnproven has a series per recipient to read (#400)" diff --git a/scripts/key-recipients.sh b/scripts/key-recipients.sh index d511308..a365d92 100755 --- a/scripts/key-recipients.sh +++ b/scripts/key-recipients.sh @@ -30,6 +30,23 @@ # "never proved" and "not proved lately" are one alert — the same choice # run-scheduled.sh makes and for the same reason. # +# WHO RUNS --record. verify-key-backup.sh after a proof (with --proved), +# add-recipient.sh after a re-key, and — since #400 — the recipient-state timer +# every day with neither. That last caller is what guarantees the file exists: +# the first two only run when a human does something, and a host that proved +# its key before this script existed had nothing that would ever write it, so +# the alert that reads it was silent from the day it was deployed. +# +# THE FIRST WRITE CARRIES THE OLD PROOF OVER. Before ADR-0024 the only record of +# a proof was homelab_job_last_success_timestamp_seconds{homelab_job= +# "verify-key-backup"}, one series whichever key was mounted. With exactly one +# recipient that series IS that recipient's proof — the ADR says so — so when +# this file is written for the first time on a host with one recipient and an +# old proof, the recipient inherits that timestamp rather than starting at 0. +# Anything else would turn a proof that happened into an alert saying it never +# did. With two or more recipients the old series cannot say which key it was, +# and every recipient starts at 0, which is the ADR's own reading. +# # Usage: # scripts/key-recipients.sh --list [--stack ] # scripts/key-recipients.sh --record [--stack ] [--proved ] @@ -140,6 +157,20 @@ prior_for() { ' "${PROM}" } +# The pre-ADR-0024 proof, read from the job series run-scheduled.sh wrote for +# the human run. Only consulted on the first write of this file, and only when +# there is exactly one recipient; see the header. +legacy_proof() { + local f="${TEXTFILE_DIR}/verify-key-backup.prom" + [[ -r "${f}" ]] || { printf '0'; return; } + awk ' + $1 ~ /^homelab_job_last_success_timestamp_seconds\{/ { value = $NF } + END { print (value ~ /^[0-9]+$/) ? value : "0" } + ' "${f}" +} +first_write=1 +[[ -r "${PROM}" ]] && first_write=0 + # Every recipient of every stack shares one file, because node_exporter merges # the directory and a metric name may carry only one HELP string across it. A # second stack writing its own file would collide on that, not on the series. @@ -160,6 +191,10 @@ tmp="${PROM}.$$" ts="${NOW}" else ts="$(prior_for "${recipient}")" + if ((first_write)) && ((${#RECIPIENTS[@]} == 1)) && [[ "${ts}" == "0" ]]; then + ts="$(legacy_proof)" + [[ "${ts}" != "0" ]] && warn "first write: ${recipient} inherits the pre-ADR-0024 proof at ${ts} from verify-key-backup.prom" + fi fi printf 'homelab_key_recipient_last_proof_timestamp_seconds{stack="%s",recipient="%s"} %s\n' \ "${STACK}" "${recipient}" "${ts}" diff --git a/stacks/observability/README.md b/stacks/observability/README.md index 540343d..b20440c 100644 --- a/stacks/observability/README.md +++ b/stacks/observability/README.md @@ -32,7 +32,7 @@ prometheus/ prometheus.yaml scrape config; SNMP via file_sd targets/snmp.yaml SNMP targets — hot-reloaded, no restart needed targets/blackbox*.yaml probe targets, http and dns — hot-reloaded, no restart - rules/*.rules.yaml 77 alert rules across host/network/ups/containers/blackbox/dns/backup/ids/deploy + rules/*.rules.yaml 78 alert rules across host/network/ups/containers/blackbox/dns/backup/ids/deploy tests/*.test.yaml promtool unit tests — assert the rules can fire blackbox/blackbox.yaml probe modules — reachability, and what a resolver said alertmanager/ diff --git a/stacks/observability/prometheus/rules/backup.rules.yaml b/stacks/observability/prometheus/rules/backup.rules.yaml index 6e00c30..c7556a6 100644 --- a/stacks/observability/prometheus/rules/backup.rules.yaml +++ b/stacks/observability/prometheus/rules/backup.rules.yaml @@ -22,7 +22,9 @@ # JOBS table that also decides the cadence, so a threshold and a schedule cannot # drift apart. It also means NO RULE HERE NAMES A JOB: adding a timer is a row in # that table, not an edit to this file. That is deliberate — a rule file listing -# job names is a list that silently stops growing. +# job names is a list that silently stops growing. The two rules at the bottom +# are the one exception, and they name the one job that is not a timer: +# verify-key-backup, the human proof, whose deadline is applied per recipient. # # That claim has since been tested rather than merely asserted. #99 added # `converge`, an hourly job on a different cadence to everything above, and @@ -37,8 +39,10 @@ # "declared but never ran" would need one rule per job with the name hardcoded — # exactly the list this file is avoiding — and it reacts to a single missed # scrape. `unless on(homelab_job)` against the declaration series says the same -# thing generically. absent_over_time appears once, at the bottom, for the one -# question that genuinely has no series to join against. +# thing generically. absent_over_time appears once, in ScheduledJobMetricsAbsent, +# for the one question that genuinely has no series to join against. Even the +# per-recipient file being missing (#400) is asked as an `unless` against the +# verify-key-backup declaration row rather than as an absent(). groups: - name: backup interval: 60s @@ -195,9 +199,14 @@ groups: # # A recipient that has never been verified is recorded as 0, not omitted, # so it fires here with an absurd duration rather than being invisible. - # ScheduledJobNeverRan still speaks for the case before ANY verification - # has happened, because key-recipients.sh has not written its file yet - # and there are no per-recipient series to fire on. + # + # That only holds once key-recipients.sh has written its file. This rule + # used to claim ScheduledJobNeverRan covered the time before that, and it + # does not: NeverRan looks for the JOB series, and a host that proved its + # key before ADR-0024 has one — so the fallback was satisfied, this rule + # had no left side, and the deadline went unwatched from the day it was + # deployed (#400). SecretsKeyRecipientsUnrecorded, below, is the rule + # for that state, and the recipient-state timer is what ends it. expr: | (time() - homelab_key_recipient_last_proof_timestamp_seconds) > on(instance) group_left() homelab_job_max_age_seconds{homelab_job="verify-key-backup"} @@ -215,3 +224,43 @@ groups: DIFFERENT recipient does not clear this one. Get that copy out and run `make secrets-verify-backup KEY=`: docs/runbooks/back-up-the-age-key.md. + + - alert: SecretsKeyRecipientsUnrecorded + # The deadline is declared and nothing can be held to it. install-timers + # writes the verify-key-backup threshold row; key-recipients.sh writes + # the per-recipient series SecretsKeyBackupUnproven compares against it. + # If the first exists and the second does not, the ninety-day nag has + # no left side and is silent — which is exactly what #400 found: the + # host had proved its key under the old single series, ADR-0024 moved + # the rule onto a file nothing had yet written, and the fallback that + # rule named was satisfied by the old proof. + # + # An `unless` against the declaration row and not absent(), for the + # reason the header gives: absent() would have to invent the labels, + # and this way the alert carries homelab_job and instance like every + # other rule here. It is keyed on one job by name, which the header + # also explains — verify-key-backup is the one job that is not a timer. + # + # Since #400 the recipient-state timer writes the file daily and the + # installer primes it, so in steady state this cannot fire. It fires + # for a host installed before that timer existed, for a file removed + # from the textfile directory, and for a recipient-state job that has + # never run — the last of which ScheduledJobNeverRan also names. + expr: | + homelab_job_max_age_seconds{homelab_job="verify-key-backup"} + unless on(instance) homelab_key_recipient_last_proof_timestamp_seconds + for: 1h + labels: + component: backup + severity: warning + category: correctness + annotations: + summary: "The age-key deadline is declared but no recipient has a proof series to hold to it" + description: >- + scripts/install-timers.sh declared the ninety-day deadline for + verify-key-backup and nothing has written + homelab_key_recipient_last_proof_timestamp_seconds, so + SecretsKeyBackupUnproven has nothing to read and cannot fire however + stale the proof is. `systemctl start homelab-recipient-state.service` + writes it; if that unit does not exist, the timers predate #400 and + `make install-timers` adds it. docs/runbooks/back-up-the-age-key.md. diff --git a/stacks/observability/prometheus/tests/backup.test.yaml b/stacks/observability/prometheus/tests/backup.test.yaml index db55774..e753a5e 100644 --- a/stacks/observability/prometheus/tests/backup.test.yaml +++ b/stacks/observability/prometheus/tests/backup.test.yaml @@ -92,12 +92,44 @@ tests: # And SecretsKeyBackupUnproven does not fire off the JOB series either. # It reads the per-recipient series and nothing else, so a host where # verify-key-backup has run but key-recipients.sh has not written its file - # is covered by ScheduledJobNeverRan, not by a rule that would have to - # invent a recipient label it does not have. + # gets nothing from it. This case used to say ScheduledJobNeverRan covered + # that state; it cannot, because the job series exists — that is #400, and + # the next two assertions are the rule that does cover it. - eval_time: 2h30m alertname: SecretsKeyBackupUnproven exp_alerts: [] + # --- SecretsKeyRecipientsUnrecorded: silent inside its `for` ------------ + - eval_time: 30m + alertname: SecretsKeyRecipientsUnrecorded + exp_alerts: [] + + # --- SecretsKeyRecipientsUnrecorded: the #400 state ------------------- + # + # The deadline row is declared, the job series says a proof happened, and + # no per-recipient series exists. Before this rule, nothing fired here. + - eval_time: 2h30m + alertname: SecretsKeyRecipientsUnrecorded + exp_alerts: + - exp_labels: + alertname: SecretsKeyRecipientsUnrecorded + homelab_job: verify-key-backup + instance: prometheus + component: backup + severity: warning + category: correctness + exp_annotations: + summary: "The age-key deadline is declared but no recipient has a proof series to hold to it" + description: >- + scripts/install-timers.sh declared the ninety-day deadline for + verify-key-backup and nothing has written + homelab_key_recipient_last_proof_timestamp_seconds, so + SecretsKeyBackupUnproven has nothing to read and cannot fire + however stale the proof is. `systemctl start + homelab-recipient-state.service` writes it; if that unit does + not exist, the timers predate #400 and `make install-timers` + adds it. docs/runbooks/back-up-the-age-key.md. + # --- SecretsKeyBackupUnproven: one alert per recipient -------------------- # # The assertion ADR-0024 turns on. Two recipients, one of them verified @@ -139,6 +171,15 @@ tests: copy out and run `make secrets-verify-backup KEY=`: docs/runbooks/back-up-the-age-key.md. + # --- SecretsKeyRecipientsUnrecorded: silent once the file exists -------- + # + # The paired case for the #400 assertion above: the same declaration row, + # and this time the per-recipient series is present, so the "no series" + # rule has nothing to say even though one recipient is unproved. + - eval_time: 2h30m + alertname: SecretsKeyRecipientsUnrecorded + exp_alerts: [] + # --- ScheduledJobFailed: fires on non-zero, silent on zero --------------- - interval: 1m input_series: diff --git a/systemd/homelab-recipient-state.service b/systemd/homelab-recipient-state.service new file mode 100644 index 0000000..a80ff48 --- /dev/null +++ b/systemd/homelab-recipient-state.service @@ -0,0 +1,52 @@ +# Daily record of which age recipients can open the secrets, and when each was +# last proved to (#400). +# +# WHY A TIMER WRITES A FILE THAT A HUMAN RUN ALREADY WRITES. ADR-0024 moved +# SecretsKeyBackupUnproven from the verify-key-backup job series onto one series +# per recipient, and scripts/key-recipients.sh writes those — but only when +# called, and until this unit existed it was called by exactly two things: a +# human proving a key from removable media, and add-recipient.sh. On a host that +# had proved its key BEFORE the per-recipient series existed, neither ran again, +# the file was never written, and the ninety-day alert had nothing to read while +# the fallback it named (ScheduledJobNeverRan) was satisfied by the old proof. +# Silent, and wrong in the direction that looks clean. This unit is the +# unconditional statement: the recipient list exists on disk every day, whether +# or not anyone has mounted a key lately. +# +# Needs no root, no network and no private key. key-recipients.sh reads the +# recipient list out of the plaintext `sops:` metadata of the encrypted file and +# carries every existing proof timestamp forward; it never decrypts anything and +# never sets a timestamp of its own except through --proved, which this unit does +# not pass. So a run here cannot make a recipient look proved — it can only make +# an unproved one visible. +# +# The job is `recipient-state`, not `key-recipients`, on purpose: the wrapper +# writes .prom and the collector writes key-recipients.prom, and #360 is +# what happens when those are the same file. +# +# See homelab-backup-volumes.service for why each directive below is set. +[Unit] +Description=HomeLab: record which age recipients can open the secrets +Documentation=file:///home/robo/code/Gerrrt/HomeLab/docs/runbooks/back-up-the-age-key.md + +[Service] +Type=oneshot +User=robo +Group=robo +WorkingDirectory=/home/robo/code/Gerrrt/HomeLab +Environment=STACK=observability +EnvironmentFile=-/etc/default/homelab-timers +ExecStart=/home/robo/code/Gerrrt/HomeLab/scripts/run-scheduled.sh --job recipient-state -- /usr/bin/make -C /home/robo/code/Gerrrt/HomeLab recipient-state +TimeoutStartSec=120 +Restart=no +Nice=10 +ProtectHome=false +ProtectSystem=false +PrivateTmp=true +PrivateNetwork=true +NoNewPrivileges=true +RestrictSUIDSGID=true +LockPersonality=true +StandardOutput=journal +StandardError=journal +SyslogIdentifier=homelab-recipient-state diff --git a/systemd/homelab-recipient-state.timer b/systemd/homelab-recipient-state.timer new file mode 100644 index 0000000..b43ce2e --- /dev/null +++ b/systemd/homelab-recipient-state.timer @@ -0,0 +1,16 @@ +[Unit] +Description=HomeLab: daily age recipient record +Documentation=file:///home/robo/code/Gerrrt/HomeLab/docs/runbooks/back-up-the-age-key.md + +[Timer] +Unit=homelab-recipient-state.service +# 09:15 daily, last of the morning collectors, a quarter-hour after pkg-state. +# It reads a committed file and writes one small .prom; the slot only has to +# not coincide with anything that stops the stack. +OnCalendar=*-*-* 09:15:00 +RandomizedDelaySec=900 +Persistent=true +AccuracySec=1m + +[Install] +WantedBy=timers.target