packaging(ns8): back up the encryption key with the database

[shockalotti]

Follow-up to #141, found by actually running the restore leg instead of stopping at the backup.

state-include.conf is handed to restic as --files-from, so restic backs up only the paths listed in it. #141 added volumes/calnode-data, which fixed the database, but the module state directory was never captured at all. calnode.env, which holds CALNODE_ENCRYPTION_KEY, simply was not in the backup.

Measured on a live cluster, with the volume listed but not the key: backup → remove-module --no-preserve → restore returned the database and dropped the key, and the app then failed to start with:

keyvault: failed to unwrap DEK - is CALNODE_ENCRYPTION_KEY correct?

A restored database with no key is unusable, so this is the same class of silent data loss as #141, just one layer deeper.

Backup sizes for one instance, same data:

state-include.conf	files	size
absent	2	476 bytes (no database)
volume only	7	2.16 MB (database, no key)
volume + state/calnode.env	8	2.16 MB

Verified end to end after the fix: backup → destroy → restore → configure. The app served 200, logged keyvault: DEK unwrapped from keystore (primary), the key fingerprint was byte-identical to pre-backup, and an event type created before the backup was still there afterwards.

[pullfrog]

The configured model is not in OpenCode's catalog. Pick a different model for Calnode/calnode where it is set — the Pullfrog console, the workflow's model input, or the PULLFROG_MODEL variable — or contact support if this persists.

opencode prompt failed: {"name":"UnknownError","data":{"message":"Unexpected server error. Check server logs for details.","ref":"err_dea593b1"}} — likely cause: Model not found: opencode-go/space-bunny-free. Did you mean: space-bunny, cat-jam-free, longcat-2.5-preview-free?

  ^{｜ Rerun failed job ➔ ｜ View workflow run ｜ via Pullfrog ｜ Using space-bunny-free ｜ 𝕏}

[shockalotti]

@pullfrog review this PR

[pullfrog]

ℹ️ The fix itself checks out — one documentation nuance inline.

Reviewed changes

• Encryption key added to the cluster backup — packaging/ns8/imageroot/etc/state-include.conf gains state/calnode.env, so the restic --files-from list now covers CALNODE_ENCRYPTION_KEY next to the calnode-data volume. Cross-checked against ns8-core module-backup (--files-from is appended only when etc/state-include.conf exists in the install dir) and docs/modules/backup_restore.md, which documents state/ lines as AGENT_STATE_DIR contents; agent.get_state_volume_args() parses only volumes/ lines, so the new entry cannot perturb volume mounting on restore. Also confirmed a module backed up before its first configure-module still succeeds: restic expands --files-from entries with filepath.Glob and only warns-and-skips a non-matching pattern, so the absent calnode.env does not fail the run.
• Backup and restore docs rewritten — states that --files-from means "only the paths listed here", names both required entries with the failure each one prevents, and replaces the previous two-point measurement with a three-row table covering absent / volume-only / volume-plus-key.
• Live-test claim updated — the README now records a real backup → destroy → restore → configure cycle (200 through Traefik, keyvault: DEK unwrapped from keystore, byte-identical key fingerprint, pre-backup event type still present) instead of saying the restore leg is unverified. configure-module/20configure reading CALNODE_ENCRYPTION_KEY back out of the restored file before generating a new one (20configure:83) is what makes that round trip work.

  ^{｜ Fix all ➔ ｜ Fix 👍s ➔ ｜ View workflow run ｜ Using space-bunny-free (free) ｜ 𝕏}

[pullfrog]

"The SQLite database, the whole app state" is a stronger claim than a restic backup of a running instance delivers: internal/db/db.go:40 sets PRAGMA journal_mode=WAL, so the volume holds calnode.db plus calnode.db-wal and -shm, and restic reads those three files at three separate moments with no cross-file atomicity. The directory include is right (it does capture the -wal), so this is about the sentence rather than the line — either qualify the guarantee or make the pre-backup state deterministic.

Technical details

# Backup of a live WAL database is not a consistent snapshot

## Affected sites
- `packaging/ns8/README.md:85` — promises "the whole app state"; true for a checkpointed database, not for one mid-checkpoint at backup time.
- `internal/db/db.go:40` — `PRAGMA journal_mode=WAL`; the data file is only updated at checkpoint, so the `-wal` is where recent commits live.
- `packaging/ns8/imageroot/etc/state-include.conf:1` — the `volumes/calnode-data` line correctly covers the whole directory including `-wal`/`-shm`. No change needed here.

## Why it matters
SQLite docs: "The WAL file is part of the persistent state of the database and should be kept with the database if the database is copied or moved." Restic satisfies that, but reads each file once: `calnode.db` at t0, `calnode.db-wal` at t1. Auto-checkpoint only fires at 1000 pages (~4 MB) or when the last connection closes, so a low-traffic instance can sit with a large `-wal` and no fresh data in the `.db` for long stretches. Outcomes are "as of an earlier point" or, in the worst case, a restore whose newest frames are silently absent — the same class of quiet data loss this PR is fixing, in the one file no backup of this app can atomically capture.

## Suggested approach (optional, pick one)
- Qualify the README line: state that a backup reflects the last checkpoint, and that a restore may be slightly behind the moment the backup ran.
- Or make it deterministic: ns8-core `module-backup` runs `module-dump-state` (mode 755 in `imageroot/bin/`) *before* starting the restic container and `module-cleanup-state` after. A `module-dump-state` that forces `PRAGMA wal_checkpoint(TRUNCATE)` on `/data/calnode.db` before the backup would shrink the window to nothing. Note the module image has no `sqlite3` binary, so it needs to reach the running app container (`podman exec`) or an admin endpoint — worth scoping separately if taken.

## Open questions
- Is "as of the last checkpoint" acceptable for restore fidelity, or should the module guarantee a checkpoint at backup time?