Skip to content

packaging(ns8): back up the encryption key with the database #142

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
shockalotti merged 1 commit into from
Oct 15, 2026
Merged

packaging(ns8): back up the encryption key with the database #142

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 23 additions & 10 deletions packaging/ns8/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,12 +76,24 @@ Each backup of the instance contains:
- the `calnode-data` volume (the SQLite database file, the whole app state)
- the module state, including `calnode.env` with `CALNODE_ENCRYPTION_KEY`

This only works because the module ships `imageroot/etc/state-include.conf`
listing `volumes/calnode-data`. NS8 passes only the volumes a module declares
there to restic; a module with named volumes and no such file is backed up
successfully while silently omitting the data. Measured on a live cluster:
without it a backup of a populated instance contained 2 files and 476 bytes;
with it, 7 files and 2.1 MB.
This only works because the module ships `imageroot/etc/state-include.conf`.
NS8 hands that file to restic as `--files-from`, so it backs up **only** the
paths listed there: a module with named volumes and no such file is backed up
successfully while silently omitting the data, and a module that lists only its
volume silently omits the encryption key. Both lines are required:

- `volumes/calnode-data` - the SQLite database, the whole app state

@pullfrog pullfrog Bot Oct 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"The SQLite database, the whole app state" is a stronger claim than a restic backup of a running instance delivers: internal/db/db.go:40 sets PRAGMA journal_mode=WAL, so the volume holds calnode.db plus calnode.db-wal and -shm, and restic reads those three files at three separate moments with no cross-file atomicity. The directory include is right (it does capture the -wal), so this is about the sentence rather than the line — either qualify the guarantee or make the pre-backup state deterministic.

Technical details 
# Backup of a live WAL database is not a consistent snapshot

## Affected sites
- `packaging/ns8/README.md:85` — promises "the whole app state"; true for a checkpointed database, not for one mid-checkpoint at backup time.
- `internal/db/db.go:40` — `PRAGMA journal_mode=WAL`; the data file is only updated at checkpoint, so the `-wal` is where recent commits live.
- `packaging/ns8/imageroot/etc/state-include.conf:1` — the `volumes/calnode-data` line correctly covers the whole directory including `-wal`/`-shm`. No change needed here.

## Why it matters
SQLite docs: "The WAL file is part of the persistent state of the database and should be kept with the database if the database is copied or moved." Restic satisfies that, but reads each file once: `calnode.db` at t0, `calnode.db-wal` at t1. Auto-checkpoint only fires at 1000 pages (~4 MB) or when the last connection closes, so a low-traffic instance can sit with a large `-wal` and no fresh data in the `.db` for long stretches. Outcomes are "as of an earlier point" or, in the worst case, a restore whose newest frames are silently absent — the same class of quiet data loss this PR is fixing, in the one file no backup of this app can atomically capture.

## Suggested approach (optional, pick one)
- Qualify the README line: state that a backup reflects the last checkpoint, and that a restore may be slightly behind the moment the backup ran.
- Or make it deterministic: ns8-core `module-backup` runs `module-dump-state` (mode 755 in `imageroot/bin/`) *before* starting the restic container and `module-cleanup-state` after. A `module-dump-state` that forces `PRAGMA wal_checkpoint(TRUNCATE)` on `/data/calnode.db` before the backup would shrink the window to nothing. Note the module image has no `sqlite3` binary, so it needs to reach the running app container (`podman exec`) or an admin endpoint — worth scoping separately if taken.

## Open questions
- Is "as of the last checkpoint" acceptable for restore fidelity, or should the module guarantee a checkpoint at backup time?

- `state/calnode.env` - holds `CALNODE_ENCRYPTION_KEY`; without it a restored
database cannot be decrypted, and the app fails to start with
"keyvault: failed to unwrap DEK"

Measured on a live cluster, same instance, same data:

| state-include.conf | files | size |
|---|---|---|
| absent | 2 | 476 bytes (no database) |
| volume only | 7 | 2.16 MB (database, no key) |
| volume + `state/calnode.env` | 8 | 2.16 MB |

Restore to the same or a replacement node from the cluster admin backup
section, then re-run `configure-module` with the same hostname if the route
Expand All @@ -99,10 +111,11 @@ Live-tested 2026-10-13/14 on a single-node Rocky 9 cluster against the
`:edge` build: install, configure, host-based route (root 302, `/admin/` 200
through Traefik), SQLite on the `calnode-data` volume, encryption key
byte-identical across reconfigures, `remove-module --no-preserve` with
automatic route cleanup, and a cluster backup that captures the database
once `etc/state-include.conf` is present. Still untested live: a full
destroy-and-restore cycle (the backup capture is verified, the restore leg
is not), and the automated `tests/` robot suite, which needs a test cluster.
automatic route cleanup, and a full cluster backup → destroy → restore cycle:
after restoring, the app served 200, logged "keyvault: DEK unwrapped from
keystore", the encryption-key fingerprint was unchanged, and an event type
created before the backup was still there. Still untested live: the
automated `tests/` robot suite, which needs a test cluster.

## Uninstall

Expand Down
1 change: 1 addition & 0 deletions packaging/ns8/imageroot/etc/state-include.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
volumes/calnode-data
state/calnode.env
Loading