diff --git a/Makefile b/Makefile index 214def3..d26cc8a 100644 --- a/Makefile +++ b/Makefile @@ -310,6 +310,16 @@ pkg-state: ## Collect package state from morpheus over SSH (FreeBSD, runs as rob @# `pfSense-upgrade -c`, whose output has no format contract. ./scripts/collect-pkg-state.sh --ssh $(FW_USER)@$(FW_HOST) --host morpheus +.PHONY: recipient-state +recipient-state: ## Record which age recipients can open the secrets, and when each was proved (#400) + @# Reads the recipient list out of the encrypted file's own metadata and + @# writes one series per recipient into the textfile dir, carrying every + @# existing proof timestamp forward and setting none. Needs no key: the + @# `sops:` block is plaintext. This is what SecretsKeyBackupUnproven reads; + @# without it a host that proved its key before ADR-0024 has no series and + @# the ninety-day nag is silent. secrets-verify-backup is what sets a proof. + ./scripts/key-recipients.sh --record --stack $(STACK) + .PHONY: smart-state-remote smart-state-remote: ## Collect SMART health from morpheus over SSH (runs as robo) @# morpheus is FreeBSD with no node_exporter and no textfile directory, but diff --git a/README.md b/README.md index bcaa542..554476a 100644 --- a/README.md +++ b/README.md @@ -58,7 +58,7 @@ documents for different readers. can't run an agent (firewall, switch, UPS, iLO). One agent config, deployed identically everywhere. [How](docs/architecture.md#observability-data-flow) - **Dashboards and alerting as code.** 7 provisioned dashboards, 141 panels, and - 94 alert rules — 77 metric-based in Prometheus, 17 log-based in Loki — sharing + 95 alert rules — 78 metric-based in Prometheus, 17 log-based in Loki — sharing one Alertmanager routing tree. No dashboard exists only in a database. - **Secrets encrypted in-repo with SOPS + age.** Per-device credentials, decrypted at deploy time into gitignored paths, with `git log` showing which @@ -175,7 +175,7 @@ Full topology and data flow in [`docs/architecture.md`](docs/architecture.md). . ├── stacks/observability/ # the deployed stack — one compose file, eight services │ ├── compose.yaml -│ ├── prometheus/ # config, file_sd targets, 77 alert rules +│ ├── prometheus/ # config, file_sd targets, 78 alert rules │ ├── alertmanager/ # routing and inhibition │ ├── loki/ # single-binary config + 17 LogQL rules │ ├── alloy/ # the agent config directory, shipped to every host diff --git a/docs/observability.md b/docs/observability.md index c9507d5..0e01e6e 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -304,7 +304,7 @@ separates a quiet stream from a stopped one. ## Alerting -94 rules in total: 77 metric-based in `prometheus/rules/`, and 17 log-based in +95 rules in total: 78 metric-based in `prometheus/rules/`, and 17 log-based in `loki/rules/`. ### Log-based (Loki ruler) @@ -411,7 +411,7 @@ argument and for what to do when it exits 1. ### Metric-based (Prometheus) -77 rules across eleven files in `prometheus/rules/`: +78 rules across eleven files in `prometheus/rules/`: | File | Covers | | --- | --- | @@ -423,7 +423,7 @@ argument and for what to do when it exits 1. | `watchdog.rules.yaml` | One rule that always fires, so that its absence is detectable | | `blackbox.rules.yaml` | Whether an endpoint can actually be reached, from outside the service, and how many days its certificate has left — Grafana verified against the lab CA, the APC card's self-signed one read but not trusted, the wiki, Prometheus, Loki, Alertmanager and the switch UI over plain http. The iLO and pfSense UIs are written into `targets/blackbox.yaml` and left disabled: each needs a firewall pass from `10.0.99.20` that is a segmentation decision, not a monitoring one ([#91](https://github.com/Gerrrt/HomeLab/issues/91)) | | `dns.rules.yaml` | Whether the house is still filtering DNS, asked directly at AdGuard Home on port 53 rather than through pfSense — a probe sent down the normal resolver path always passes, because Unbound's fallback is doing its job. [ADR-0010](adr/0010-keep-the-resolver-on-the-gateway.md) made losing the filter silent on purpose, and these two rules are what distinguishes "this site was never on a list" from "AdGuard has been dead for three weeks". Warning, not critical: nothing is down and nobody is blocked. The targets are written into `targets/blackbox-dns.yaml` and left disabled until [#102](https://github.com/Gerrrt/HomeLab/issues/102) builds the mini PC ([#126](https://github.com/Gerrrt/HomeLab/issues/126)) | -| `backup.rules.yaml` | Whether the scheduled maintenance jobs are still being run at all — staleness, failure, and never-ran | +| `backup.rules.yaml` | Whether the scheduled maintenance jobs are still being run at all — staleness, failure, never-ran, and whether the age-key proof record exists to be held to its deadline | | `deploy.rules.yaml` | Whether this host is running what the repository says — an uncommitted edit made on the host, a revision that did not verify, and how far behind `main` the host is. Reads the record `scripts/converge.sh` writes hourly ([#99](https://github.com/Gerrrt/HomeLab/issues/99), [ADR-0021](adr/0021-converge-on-a-timer-instead-of-deploying-over-ssh.md)) | | `ids.rules.yaml` | Whether Suricata is running on each interface it is declared for, read from the firewall's process table over SNMP — the process metric `security.rules.yaml` says a log rule cannot be ([#90](https://github.com/Gerrrt/HomeLab/issues/90)) | @@ -434,11 +434,11 @@ as loaded and healthy and could not fire for any input ([#63](https://github.com `prometheus/tests/*.test.yaml` holds `promtool test rules` unit tests, which feed a rule synthetic series and assert it fires — paired with a case asserting it stays quiet, because a test that only ever expects silence would have passed -against the broken rule too. Coverage is fifty-six rules of 77 so far — the five +against the broken rule too. Coverage is fifty-seven rules of 78 so far — the five in `blackbox.rules.yaml`, both in `dns.rules.yaml`, `ContainerHighMemory`, `ContainerNearMemoryLimit`, `ContainerRestartLoop`, `ContainerCpuThrottled` and `PrometheusSizeRetentionActive`, `Watchdog`, the three iLO rules from -[#76](https://github.com/Gerrrt/HomeLab/issues/76), all five in +[#76](https://github.com/Gerrrt/HomeLab/issues/76), all six in `backup.test.yaml`, all five in `deploy.test.yaml`, `RemoteWriteJobStale`, `SuricataStopped`, the two gateway rules from [#353](https://github.com/Gerrrt/HomeLab/issues/353), and all eighteen in @@ -593,10 +593,11 @@ a textfile the node exporter already scrapes: The threshold each job is held to is a *fifth* series, `homelab_job_max_age_seconds`, written by `scripts/install-timers.sh` from the -same table that decides the cadence. That is what lets the five rules in -`prometheus/rules/backup.rules.yaml` cover every job without naming any of them, -and what makes `make check-timers` able to assert that a threshold is at least -twice its timer's real period. +same table that decides the cadence. That is what lets the rules in +`prometheus/rules/backup.rules.yaml` cover every job without naming any of them +— the two that do name one both concern `verify-key-backup`, the human proof, +below — and what makes `make check-timers` able to assert that a threshold is +at least twice its timer's real period. Two things are deliberate and easy to undo by accident: @@ -629,6 +630,17 @@ to `oracle` and fails if it cannot, so its failure alert doubles as "the config has stopped leaving this host". The volume sets do not leave; that is [#92](https://github.com/Gerrrt/HomeLab/issues/92). +That series has to exist for the nag to mean anything, and for four days it did +not ([#400](https://github.com/Gerrrt/HomeLab/issues/400)): it was written only +by a proof run or by adding a recipient, and this host had proved its key before +the series was invented, so the rule went quiet the day it was deployed while +the fallback it named — `ScheduledJobNeverRan` — was satisfied by the old proof. +The `recipient-state` timer now writes the recipient list daily, carrying proofs +forward and setting none, and `SecretsKeyRecipientsUnrecorded` fires when the +deadline row exists and the recipient series does not — an `unless` against the +declaration row, not an `absent()`, so it carries labels like every other rule +in the file. + Installing, tuning and troubleshooting all of it: [`runbooks/schedule-maintenance.md`](runbooks/schedule-maintenance.md). diff --git a/docs/runbooks/back-up-the-age-key.md b/docs/runbooks/back-up-the-age-key.md index 540b435..e8fa1c8 100644 --- a/docs/runbooks/back-up-the-age-key.md +++ b/docs/runbooks/back-up-the-age-key.md @@ -205,9 +205,23 @@ removable media, and no timer can mount that. So it is enforced from the other end. `make secrets-verify-backup` records the timestamp of a successful run **against the recipient it just proved**, and `SecretsKeyBackupUnproven` fires when any recipient's proof passes ninety days -old — routed to the normal alert channel like any other warning. Until the first -verification there is no timestamp at all and `ScheduledJobNeverRan` says so -instead, which is the honest reading of a key backup nobody has ever tested. +old — routed to the normal alert channel like any other warning. A recipient +that has never been proved is recorded as `0`, so it fires with an absurd age +rather than being invisible, which is the honest reading of a key backup nobody +has ever tested. + +That record has to exist for any of it to work, and until +[#400](https://github.com/Gerrrt/HomeLab/issues/400) nothing guaranteed it did: +the per-recipient series was written only by a proof run or by adding a +recipient, so a host that had proved its key *before* the series was invented +never wrote it, and the nag was silent from the day it was deployed. The +`recipient-state` timer now writes the recipient list every day — carrying every +existing proof forward and setting none, since it never touches a key — and +`SecretsKeyRecipientsUnrecorded` fires if the deadline is declared and the file +is missing anyway. On a host with exactly one recipient, the first write +inherits the pre-ADR-0024 proof from the old `verify-key-backup` series rather +than starting from never; with more than one, the old series cannot say which +key it was, and every recipient starts at `0`. The per-recipient part is [ADR-0024](../adr/0024-hold-a-second-age-recipient-and-prove-each-one-separately.md) and it only starts to matter once there is more than one. With one recipient diff --git a/docs/runbooks/deploy-stack.md b/docs/runbooks/deploy-stack.md index 5ed095d..4ec1eb8 100644 --- a/docs/runbooks/deploy-stack.md +++ b/docs/runbooks/deploy-stack.md @@ -118,7 +118,7 @@ Then in the UI: 1. **Prometheus → Status → Targets.** Every job `UP`. The four `snmp` targets take up to 45 seconds on their first scrape. -2. **Prometheus → Status → Rules.** 77 rules loaded, none in error. The +2. **Prometheus → Status → Rules.** 78 rules loaded, none in error. The page counts 57: the extra one is `homelab_suricata_expected_interface`, the stack's only recording rule. Everything counted in this repository is alert rules, so the two numbers differ by one and always have. diff --git a/docs/runbooks/schedule-maintenance.md b/docs/runbooks/schedule-maintenance.md index 8fc30ab..12e9082 100644 --- a/docs/runbooks/schedule-maintenance.md +++ b/docs/runbooks/schedule-maintenance.md @@ -23,8 +23,14 @@ The one job that genuinely proves off-host recoverability is a timer: [`verify-key-backup.sh`](../../scripts/verify-key-backup.sh) refuses the live key by device and inode, precisely so that what gets tested is a copy on removable media. No timer can mount that. So it is enforced from the other end — -a successful run records its timestamp, and `SecretsKeyBackupUnproven` fires when -that proof passes ninety days old. +a successful run records its timestamp against the recipient it proved, and +`SecretsKeyBackupUnproven` fires when that proof passes ninety days old. The +`recipient-state` timer is what makes that series exist at all: it writes one +row per recipient every day, carrying proofs forward and setting none, because +a host that had proved its key before the per-recipient series existed had +nothing else that would ever write it, and the nag was silent for as long as +that lasted ([#400](https://github.com/Gerrrt/HomeLab/issues/400)). +`SecretsKeyRecipientsUnrecorded` fires if the file is missing anyway. One job's output leaves this host: `backup-firewall` copies every export to `oracle` and **fails if it cannot**, so its `ScheduledJobFailed` also means "the @@ -65,6 +71,8 @@ the host. | `smart-state` | `make smart-state` | daily 08:30 | 2 days | | `smart-state-remote` | `make smart-state-remote` | daily 08:45 | 2 days | | `pkg-state` | `make pkg-state` | daily 09:00 | 2 days | +| `recipient-state` | `make recipient-state` | daily 09:15 | 2 days | +| `gateway-state` | `make gateway-state` | every 15 minutes | 90 minutes | | `verify-key-backup` | **you**, `make secrets-verify-backup KEY=…` | no timer | 90 days | Thresholds are roughly twice the period, never once: a threshold equal to the @@ -517,6 +525,7 @@ expected rather than a second fault. | Symptom | Cause | Fix | | --- | --- | --- | | `ScheduledJobNeverRan` right after install | The job has a threshold declared and has never reported a result | Expected for `verify-key-backup` until you first verify the key. For anything else, `systemctl start homelab-.service` and read the journal | +| `SecretsKeyRecipientsUnrecorded` | The ninety-day deadline is declared and no recipient has a proof series, so `SecretsKeyBackupUnproven` cannot fire however stale the proof is | `systemctl start homelab-recipient-state.service`. If that unit does not exist the timers predate [#400](https://github.com/Gerrrt/HomeLab/issues/400): `make install-timers` adds it and primes it. On a host with one recipient the first write inherits the old `verify-key-backup` proof rather than starting from never | | `ScheduledJobMetricsAbsent` | Nothing from the textfile directory has reached Prometheus in six hours | This is the whole directory, not one file — check Alloy is up and the directory still exists. A single malformed file shows as `node_textfile_scrape_error 1` and costs only that file | | One job's series missing, `node_textfile_scrape_error` is 1 | That job's `.prom` failed to parse — a truncated write, or something wrote it without the temp-then-rename | The other files are unaffected. Re-run the job; if it recurs, something is writing the file directly instead of through `run-scheduled.sh` | | Every `homelab_job_*` series missing, no scrape error | The `textfile` block in `config.alloy` is not reading the right path | It must carry the `ALLOY_ROOTFS` prefix (`/rootfs` in the container). `rootfs_path` does **not** apply to that argument, and a wrong path reports an empty directory rather than an error | @@ -532,7 +541,7 @@ expected rather than a second fault. | `check-versions` exits 1 | A document names an OS version the host is not running | Not an outage — nothing is broken. Read the FAIL lines: each names the document, the cell and what the host reports. Correct the document; the box is the source of truth. A `SKIP` for `morpheus` instead means `sysDescr` is not reaching Prometheus, which is a collection fault rather than a clean bill of health | | `docker info` fails only under systemd | The unit is missing `SupplementaryGroups=docker` | A login shell picks the group up from `/etc/group` and a unit does not, which is why this never reproduces by hand | | Timers exist but never fire | `WantedBy=timers.target` missing, or the timers were never enabled | `systemctl list-timers 'homelab-*'` shows nothing; re-run `make install-timers` | -| `ScheduledJobMetricsAbsent` fires and nothing else in `backup.rules.yaml` ever has | This step was never run at all | `systemctl list-unit-files 'homelab*'` reports *0 unit files* and `/var/lib/node_exporter/textfile_collector` does not exist. The four other rules here join against a series `--install` writes, so none of them can fire — that alert is the only one that can, and it is doing its job ([#215](https://github.com/Gerrrt/HomeLab/issues/215)). Run `make install-timers` | +| `ScheduledJobMetricsAbsent` fires and nothing else in `backup.rules.yaml` ever has | This step was never run at all | `systemctl list-unit-files 'homelab*'` reports *0 unit files* and `/var/lib/node_exporter/textfile_collector` does not exist. The other rules here join against a series `--install` writes, so none of them can fire — that alert is the only one that can, and it is doing its job ([#215](https://github.com/Gerrrt/HomeLab/issues/215)). Run `make install-timers` | | `converge` fails every hour with a signature error | GitHub's signing key was never imported into `robo`'s keyring, so nothing on this host can verify | The one-time import in [`converge-the-host.md`](converge-the-host.md) §Set it up. Every other job here is unaffected | | `refusing to install from …` | You are in a worktree or a second clone | The units hardcode the deployment path. Install from `/home/robo/code/Gerrrt/HomeLab` | | `make validate` fails on the schedule | A cadence and its threshold disagree | `make check-timers` names the job and both numbers. Fix the `JOBS` table or the `.timer`, not the alert | diff --git a/docs/runbooks/successor-handover.md b/docs/runbooks/successor-handover.md index b331721..d03ced1 100644 --- a/docs/runbooks/successor-handover.md +++ b/docs/runbooks/successor-handover.md @@ -193,7 +193,7 @@ inherits without knowing. | --- | --- | --- | | **Alert delivery to a destination you do not own** | Immediately, and silently | You do not. This is step 2 above, and it is the reason it is step 2 | | **The external heartbeat watcher** — a free-tier cron-monitor on somebody else's account | Whenever that account lapses | Nothing here can tell you. A watcher on this host would fail with the thing it watches, which is why it is off-host and therefore outside anything this repository can check | -| **The age key backup goes unproven** | 90 days after the last verification, *per recipient* | `SecretsKeyBackupUnproven`, routed to the normal alert channel, naming the recipient — proving one copy does not clear another ([ADR-0024](../adr/0024-hold-a-second-age-recipient-and-prove-each-one-separately.md)). Until the first verification there is no timestamp at all and `ScheduledJobNeverRan` says so instead | +| **The age key backup goes unproven** | 90 days after the last verification, *per recipient* | `SecretsKeyBackupUnproven`, routed to the normal alert channel, naming the recipient — proving one copy does not clear another ([ADR-0024](../adr/0024-hold-a-second-age-recipient-and-prove-each-one-separately.md)). A recipient never proved is recorded as never, and fires; `SecretsKeyRecipientsUnrecorded` fires instead if the record itself is missing ([#400](https://github.com/Gerrrt/HomeLab/issues/400)) | | **Grafana's leaf certificate** | 825 days from issue; the APC card's own certificate expires on its own clock | `TlsCertificateExpiringSoon` at 30 days, `TlsCertificateExpiryImminent` at 7 — read off the served handshake by `blackbox-exporter`, not off a file. Let it lapse and `up{job="grafana"}` goes to 0 as well | | **The UPS battery pack** | A pack was fitted 2026-08-28 and passed its self-test; packs are consumables and this one is on a biweekly test schedule | `UpsSelfTestFailed` and `UpsBatteryUnproven` key on the self-test result, which is the single honest signal this card emits — every charge, runtime and alarm value it reports was fabricated while the bay was empty. Two things remain open: the card's test *schedule* is unwatched ([#249](https://github.com/Gerrrt/HomeLab/issues/249)), and `upsBasicBatteryLastReplaceDate` still reads a pre-fit date, so it is not a usable record of the pack's age | | **Mains power to the monitoring path** | Any cut | The rack is on the UPS; the switch carrying `prometheus` and `oracle` is not, so both laptops keep running and go deaf. Stated in [`security.md`](../security.md#threat-model) | diff --git a/docs/security.md b/docs/security.md index f85a7f5..cf200ef 100644 --- a/docs/security.md +++ b/docs/security.md @@ -17,7 +17,7 @@ What this network is actually built to survive: | An attacker on the lab segment reaching the hypervisor's BMC | **Accepted.** `shiva` stays on VLAN 30 by decision ([ADR-0033](adr/0033-keep-the-ilo-on-the-lab-segment.md)); a BMC compromise in the lab costs the lab, and the tripwire watches what it initiates | | A range target with a path out | It has none — `ifrit`'s targets sit on a bridge with no physical port, on `172.30.30.0/24`, which the firewall does not route and on which nothing has a default route at all ([ADR-0014](adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md), [ADR-0017](adr/0017-buy-ifrit-for-iops-and-keep-the-range-disposable.md)) | | Someone with the trusted Wi-Fi key quietly joining | Kea's lease log reaches Loki; `UnknownDeviceOnTrustedSegment` fires the first time a MAC appears on VLAN 50 in seven days ([ADR-0019](adr/0019-read-device-joins-from-the-dhcp-server.md)) | -| Losing visibility of a failure | 94 alert rules, 30 days of metrics and logs | +| Losing visibility of a failure | 95 alert rules, 30 days of metrics and logs | | Someone on a reachable VLAN silencing an alert to hide a failure | Alertmanager binds to `127.0.0.1`; silences go through authenticated Grafana | | Mains power loss | **The rack, yes; the monitoring path, no.** A pack fitted to `mjolnir` on 2026-08-28 passed its self-test; the switch carrying `prometheus` and `oracle` still has no battery — see below | | The estate being down while the person who runs it is unavailable | **Documentation, yes; data, not yet.** ADR-0011 puts the emergency tier on paper; [ADR-0023](adr/0023-keep-the-household-recovery-path-outside-the-estate.md) extends the same reasoning to the sensitive tier's data before that tier exists — see below | diff --git a/scripts/install-timers.sh b/scripts/install-timers.sh index c3e1efc..0d74d5c 100755 --- a/scripts/install-timers.sh +++ b/scripts/install-timers.sh @@ -102,6 +102,15 @@ DEPLOY_ROOT="/home/robo/code/Gerrrt/HomeLab" # touching this table, which is the same property every other rule in # backup.rules.yaml has and the reason none of them name a job. # +# recipient-state is what makes that series exist (#400). key-recipients.sh +# used to be run only by a human proof and by add-recipient.sh, so a host that +# had proved its key before ADR-0024 landed never wrote the per-recipient file +# at all — and SecretsKeyBackupUnproven, which reads only that file, went quiet +# the day it was deployed. This row writes the file daily, carrying every +# existing proof forward and setting none; the priming loop below writes it at +# install time. SecretsKeyRecipientsUnrecorded is the rule that notices when +# the file is missing anyway. +# # converge is the only hourly row, and the only one whose threshold is three # times its period rather than two. It shares the `backups` lock with the two # backup jobs, so a run that collides with the weekly archive can legitimately @@ -123,6 +132,7 @@ JOBS=( "smart-state homelab-smart-state 172800 smart-state" "smart-state-remote homelab-smart-state-remote 172800 smart-state-remote" "pkg-state homelab-pkg-state 172800 pkg-state" + "recipient-state homelab-recipient-state 172800 recipient-state" "gateway-state homelab-gateway-state 5400 gateway-state" "verify-key-backup - 7776000 secrets-verify-backup" ) @@ -468,3 +478,4 @@ green "installed — systemctl list-timers 'homelab-*'" info "backup-volumes was NOT primed: it stops the stack. Run it when you can watch:" info " sudo systemctl start homelab-backup-volumes.service" info "verify-key-backup has no timer and never will — docs/runbooks/back-up-the-age-key.md" +info "recipient-state was primed, so SecretsKeyBackupUnproven has a series per recipient to read (#400)" diff --git a/scripts/key-recipients.sh b/scripts/key-recipients.sh index d511308..a365d92 100755 --- a/scripts/key-recipients.sh +++ b/scripts/key-recipients.sh @@ -30,6 +30,23 @@ # "never proved" and "not proved lately" are one alert — the same choice # run-scheduled.sh makes and for the same reason. # +# WHO RUNS --record. verify-key-backup.sh after a proof (with --proved), +# add-recipient.sh after a re-key, and — since #400 — the recipient-state timer +# every day with neither. That last caller is what guarantees the file exists: +# the first two only run when a human does something, and a host that proved +# its key before this script existed had nothing that would ever write it, so +# the alert that reads it was silent from the day it was deployed. +# +# THE FIRST WRITE CARRIES THE OLD PROOF OVER. Before ADR-0024 the only record of +# a proof was homelab_job_last_success_timestamp_seconds{homelab_job= +# "verify-key-backup"}, one series whichever key was mounted. With exactly one +# recipient that series IS that recipient's proof — the ADR says so — so when +# this file is written for the first time on a host with one recipient and an +# old proof, the recipient inherits that timestamp rather than starting at 0. +# Anything else would turn a proof that happened into an alert saying it never +# did. With two or more recipients the old series cannot say which key it was, +# and every recipient starts at 0, which is the ADR's own reading. +# # Usage: # scripts/key-recipients.sh --list [--stack ] # scripts/key-recipients.sh --record [--stack ] [--proved ] @@ -140,6 +157,20 @@ prior_for() { ' "${PROM}" } +# The pre-ADR-0024 proof, read from the job series run-scheduled.sh wrote for +# the human run. Only consulted on the first write of this file, and only when +# there is exactly one recipient; see the header. +legacy_proof() { + local f="${TEXTFILE_DIR}/verify-key-backup.prom" + [[ -r "${f}" ]] || { printf '0'; return; } + awk ' + $1 ~ /^homelab_job_last_success_timestamp_seconds\{/ { value = $NF } + END { print (value ~ /^[0-9]+$/) ? value : "0" } + ' "${f}" +} +first_write=1 +[[ -r "${PROM}" ]] && first_write=0 + # Every recipient of every stack shares one file, because node_exporter merges # the directory and a metric name may carry only one HELP string across it. A # second stack writing its own file would collide on that, not on the series. @@ -160,6 +191,10 @@ tmp="${PROM}.$$" ts="${NOW}" else ts="$(prior_for "${recipient}")" + if ((first_write)) && ((${#RECIPIENTS[@]} == 1)) && [[ "${ts}" == "0" ]]; then + ts="$(legacy_proof)" + [[ "${ts}" != "0" ]] && warn "first write: ${recipient} inherits the pre-ADR-0024 proof at ${ts} from verify-key-backup.prom" + fi fi printf 'homelab_key_recipient_last_proof_timestamp_seconds{stack="%s",recipient="%s"} %s\n' \ "${STACK}" "${recipient}" "${ts}" diff --git a/stacks/observability/README.md b/stacks/observability/README.md index 540343d..b20440c 100644 --- a/stacks/observability/README.md +++ b/stacks/observability/README.md @@ -32,7 +32,7 @@ prometheus/ prometheus.yaml scrape config; SNMP via file_sd targets/snmp.yaml SNMP targets — hot-reloaded, no restart needed targets/blackbox*.yaml probe targets, http and dns — hot-reloaded, no restart - rules/*.rules.yaml 77 alert rules across host/network/ups/containers/blackbox/dns/backup/ids/deploy + rules/*.rules.yaml 78 alert rules across host/network/ups/containers/blackbox/dns/backup/ids/deploy tests/*.test.yaml promtool unit tests — assert the rules can fire blackbox/blackbox.yaml probe modules — reachability, and what a resolver said alertmanager/ diff --git a/stacks/observability/prometheus/rules/backup.rules.yaml b/stacks/observability/prometheus/rules/backup.rules.yaml index 6e00c30..c7556a6 100644 --- a/stacks/observability/prometheus/rules/backup.rules.yaml +++ b/stacks/observability/prometheus/rules/backup.rules.yaml @@ -22,7 +22,9 @@ # JOBS table that also decides the cadence, so a threshold and a schedule cannot # drift apart. It also means NO RULE HERE NAMES A JOB: adding a timer is a row in # that table, not an edit to this file. That is deliberate — a rule file listing -# job names is a list that silently stops growing. +# job names is a list that silently stops growing. The two rules at the bottom +# are the one exception, and they name the one job that is not a timer: +# verify-key-backup, the human proof, whose deadline is applied per recipient. # # That claim has since been tested rather than merely asserted. #99 added # `converge`, an hourly job on a different cadence to everything above, and @@ -37,8 +39,10 @@ # "declared but never ran" would need one rule per job with the name hardcoded — # exactly the list this file is avoiding — and it reacts to a single missed # scrape. `unless on(homelab_job)` against the declaration series says the same -# thing generically. absent_over_time appears once, at the bottom, for the one -# question that genuinely has no series to join against. +# thing generically. absent_over_time appears once, in ScheduledJobMetricsAbsent, +# for the one question that genuinely has no series to join against. Even the +# per-recipient file being missing (#400) is asked as an `unless` against the +# verify-key-backup declaration row rather than as an absent(). groups: - name: backup interval: 60s @@ -195,9 +199,14 @@ groups: # # A recipient that has never been verified is recorded as 0, not omitted, # so it fires here with an absurd duration rather than being invisible. - # ScheduledJobNeverRan still speaks for the case before ANY verification - # has happened, because key-recipients.sh has not written its file yet - # and there are no per-recipient series to fire on. + # + # That only holds once key-recipients.sh has written its file. This rule + # used to claim ScheduledJobNeverRan covered the time before that, and it + # does not: NeverRan looks for the JOB series, and a host that proved its + # key before ADR-0024 has one — so the fallback was satisfied, this rule + # had no left side, and the deadline went unwatched from the day it was + # deployed (#400). SecretsKeyRecipientsUnrecorded, below, is the rule + # for that state, and the recipient-state timer is what ends it. expr: | (time() - homelab_key_recipient_last_proof_timestamp_seconds) > on(instance) group_left() homelab_job_max_age_seconds{homelab_job="verify-key-backup"} @@ -215,3 +224,43 @@ groups: DIFFERENT recipient does not clear this one. Get that copy out and run `make secrets-verify-backup KEY=`: docs/runbooks/back-up-the-age-key.md. + + - alert: SecretsKeyRecipientsUnrecorded + # The deadline is declared and nothing can be held to it. install-timers + # writes the verify-key-backup threshold row; key-recipients.sh writes + # the per-recipient series SecretsKeyBackupUnproven compares against it. + # If the first exists and the second does not, the ninety-day nag has + # no left side and is silent — which is exactly what #400 found: the + # host had proved its key under the old single series, ADR-0024 moved + # the rule onto a file nothing had yet written, and the fallback that + # rule named was satisfied by the old proof. + # + # An `unless` against the declaration row and not absent(), for the + # reason the header gives: absent() would have to invent the labels, + # and this way the alert carries homelab_job and instance like every + # other rule here. It is keyed on one job by name, which the header + # also explains — verify-key-backup is the one job that is not a timer. + # + # Since #400 the recipient-state timer writes the file daily and the + # installer primes it, so in steady state this cannot fire. It fires + # for a host installed before that timer existed, for a file removed + # from the textfile directory, and for a recipient-state job that has + # never run — the last of which ScheduledJobNeverRan also names. + expr: | + homelab_job_max_age_seconds{homelab_job="verify-key-backup"} + unless on(instance) homelab_key_recipient_last_proof_timestamp_seconds + for: 1h + labels: + component: backup + severity: warning + category: correctness + annotations: + summary: "The age-key deadline is declared but no recipient has a proof series to hold to it" + description: >- + scripts/install-timers.sh declared the ninety-day deadline for + verify-key-backup and nothing has written + homelab_key_recipient_last_proof_timestamp_seconds, so + SecretsKeyBackupUnproven has nothing to read and cannot fire however + stale the proof is. `systemctl start homelab-recipient-state.service` + writes it; if that unit does not exist, the timers predate #400 and + `make install-timers` adds it. docs/runbooks/back-up-the-age-key.md. diff --git a/stacks/observability/prometheus/tests/backup.test.yaml b/stacks/observability/prometheus/tests/backup.test.yaml index db55774..e753a5e 100644 --- a/stacks/observability/prometheus/tests/backup.test.yaml +++ b/stacks/observability/prometheus/tests/backup.test.yaml @@ -92,12 +92,44 @@ tests: # And SecretsKeyBackupUnproven does not fire off the JOB series either. # It reads the per-recipient series and nothing else, so a host where # verify-key-backup has run but key-recipients.sh has not written its file - # is covered by ScheduledJobNeverRan, not by a rule that would have to - # invent a recipient label it does not have. + # gets nothing from it. This case used to say ScheduledJobNeverRan covered + # that state; it cannot, because the job series exists — that is #400, and + # the next two assertions are the rule that does cover it. - eval_time: 2h30m alertname: SecretsKeyBackupUnproven exp_alerts: [] + # --- SecretsKeyRecipientsUnrecorded: silent inside its `for` ------------ + - eval_time: 30m + alertname: SecretsKeyRecipientsUnrecorded + exp_alerts: [] + + # --- SecretsKeyRecipientsUnrecorded: the #400 state ------------------- + # + # The deadline row is declared, the job series says a proof happened, and + # no per-recipient series exists. Before this rule, nothing fired here. + - eval_time: 2h30m + alertname: SecretsKeyRecipientsUnrecorded + exp_alerts: + - exp_labels: + alertname: SecretsKeyRecipientsUnrecorded + homelab_job: verify-key-backup + instance: prometheus + component: backup + severity: warning + category: correctness + exp_annotations: + summary: "The age-key deadline is declared but no recipient has a proof series to hold to it" + description: >- + scripts/install-timers.sh declared the ninety-day deadline for + verify-key-backup and nothing has written + homelab_key_recipient_last_proof_timestamp_seconds, so + SecretsKeyBackupUnproven has nothing to read and cannot fire + however stale the proof is. `systemctl start + homelab-recipient-state.service` writes it; if that unit does + not exist, the timers predate #400 and `make install-timers` + adds it. docs/runbooks/back-up-the-age-key.md. + # --- SecretsKeyBackupUnproven: one alert per recipient -------------------- # # The assertion ADR-0024 turns on. Two recipients, one of them verified @@ -139,6 +171,15 @@ tests: copy out and run `make secrets-verify-backup KEY=`: docs/runbooks/back-up-the-age-key.md. + # --- SecretsKeyRecipientsUnrecorded: silent once the file exists -------- + # + # The paired case for the #400 assertion above: the same declaration row, + # and this time the per-recipient series is present, so the "no series" + # rule has nothing to say even though one recipient is unproved. + - eval_time: 2h30m + alertname: SecretsKeyRecipientsUnrecorded + exp_alerts: [] + # --- ScheduledJobFailed: fires on non-zero, silent on zero --------------- - interval: 1m input_series: diff --git a/systemd/homelab-recipient-state.service b/systemd/homelab-recipient-state.service new file mode 100644 index 0000000..a80ff48 --- /dev/null +++ b/systemd/homelab-recipient-state.service @@ -0,0 +1,52 @@ +# Daily record of which age recipients can open the secrets, and when each was +# last proved to (#400). +# +# WHY A TIMER WRITES A FILE THAT A HUMAN RUN ALREADY WRITES. ADR-0024 moved +# SecretsKeyBackupUnproven from the verify-key-backup job series onto one series +# per recipient, and scripts/key-recipients.sh writes those — but only when +# called, and until this unit existed it was called by exactly two things: a +# human proving a key from removable media, and add-recipient.sh. On a host that +# had proved its key BEFORE the per-recipient series existed, neither ran again, +# the file was never written, and the ninety-day alert had nothing to read while +# the fallback it named (ScheduledJobNeverRan) was satisfied by the old proof. +# Silent, and wrong in the direction that looks clean. This unit is the +# unconditional statement: the recipient list exists on disk every day, whether +# or not anyone has mounted a key lately. +# +# Needs no root, no network and no private key. key-recipients.sh reads the +# recipient list out of the plaintext `sops:` metadata of the encrypted file and +# carries every existing proof timestamp forward; it never decrypts anything and +# never sets a timestamp of its own except through --proved, which this unit does +# not pass. So a run here cannot make a recipient look proved — it can only make +# an unproved one visible. +# +# The job is `recipient-state`, not `key-recipients`, on purpose: the wrapper +# writes .prom and the collector writes key-recipients.prom, and #360 is +# what happens when those are the same file. +# +# See homelab-backup-volumes.service for why each directive below is set. +[Unit] +Description=HomeLab: record which age recipients can open the secrets +Documentation=file:///home/robo/code/Gerrrt/HomeLab/docs/runbooks/back-up-the-age-key.md + +[Service] +Type=oneshot +User=robo +Group=robo +WorkingDirectory=/home/robo/code/Gerrrt/HomeLab +Environment=STACK=observability +EnvironmentFile=-/etc/default/homelab-timers +ExecStart=/home/robo/code/Gerrrt/HomeLab/scripts/run-scheduled.sh --job recipient-state -- /usr/bin/make -C /home/robo/code/Gerrrt/HomeLab recipient-state +TimeoutStartSec=120 +Restart=no +Nice=10 +ProtectHome=false +ProtectSystem=false +PrivateTmp=true +PrivateNetwork=true +NoNewPrivileges=true +RestrictSUIDSGID=true +LockPersonality=true +StandardOutput=journal +StandardError=journal +SyslogIdentifier=homelab-recipient-state diff --git a/systemd/homelab-recipient-state.timer b/systemd/homelab-recipient-state.timer new file mode 100644 index 0000000..b43ce2e --- /dev/null +++ b/systemd/homelab-recipient-state.timer @@ -0,0 +1,16 @@ +[Unit] +Description=HomeLab: daily age recipient record +Documentation=file:///home/robo/code/Gerrrt/HomeLab/docs/runbooks/back-up-the-age-key.md + +[Timer] +Unit=homelab-recipient-state.service +# 09:15 daily, last of the morning collectors, a quarter-hour after pkg-state. +# It reads a committed file and writes one small .prom; the slot only has to +# not coincide with anything that stops the stack. +OnCalendar=*-*-* 09:15:00 +RandomizedDelaySec=900 +Persistent=true +AccuracySec=1m + +[Install] +WantedBy=timers.target