core: durable node_exporter textfile metrics are lost on reboot

[gsanchietti]

Steps to reproduce

1. Have an enabled backup with a recorded status, e.g. a known-broken one:

   cat /run/node_exporter/backup<id>.prom   # node_backup_status{...} 0

2. Reboot the machine — /run is memory-only, so it's lost upon a reboot:
3. After the boot:

   cat /run/node_exporter/backup<id>.prom   # No such file or directory

Expected behavior

The backup status shown to Prometheus should reflect the last known real verdict at all times, including right after a reboot. An alert for a genuinely broken backup should keep firing.

Actual behavior

The file — and the whole node_backup_status series — disappears until the next scheduled run completes, silently clearing any alert based on it (e.g. BackupFailed).

This is a specific instance of a general pattern: any durable status published only as a /run (tmpfs) textfile-collector .prom file is invisible after a reboot, silently clearing whatever alert depends on it. Worth keeping in mind for any future gauge written the same way — a certificate-expiry check was raised as a plausible example — not just backups.

Suggested fix or workaround

Stop keeping this state on a tmpfs. Point node_exporter's --collector.textfile.directory (currently /host/run/node_exporter, i.e. host /run/node_exporter) and the matching writers (run-backup, refresh-node-info) at a persistent, non-tmpfs path under /var/lib/nethserver/node/state/ instead. The whole host root is already bind-mounted read-only into the node_exporter container (-v /:/host:ro,rslave), so any path under /var/lib/nethserver is already visible to it — no new mount needed, just the directory + flag change, plus a one-time migration/cleanup of the old /run/node_exporter path on upgrade. This removes the reboot gap by construction, for this metric and for any future one, instead of reconstructing lost state after the fact.

References

See also #8125

[gsanchietti]

Testing versions: ns8-core:3.21.1-dev.2

Update an existing instance (run on the leader node):

update-core ghcr.io/nethserver/core:3.21.1-dev.2

Fresh install:

curl https://raw.githubusercontent.com/NethServer/ns8-core/main/core/install.sh > install.sh
bash install.sh ghcr.io/nethserver/core:latest ghcr.io/nethserver/core:3.21.1-dev.2

Test case 1: backup status survives a reboot

1. Set up a backup, make it fail once so there's a recorded "failed" status.
2. Check the file exists: cat /var/lib/nethserver/node/state/node_exporter/backup<id>.prom.
3. Reboot the node.
4. Right after it's back up (before the backup runs again), check the file again.

Expected: the file is still there with the same "failed" result — it does not disappear after reboot.

Test case 2: node info metrics survive a reboot too

1. Check cat /var/lib/nethserver/node/state/node_exporter/node-info.prom exists and has content.
2. Reboot the node.
3. Check the file again right after boot.

Expected: file is still present with data, not missing/empty.

Test case 3: update from a previous core version

1. On an older core version, confirm the old bug: after a reboot, /run/node_exporter/*.prom files are gone.
2. Update to 3.21.1-dev.2 with the command above.
3. Trigger a backup so a status file is written, then reboot.

Expected: after the update, status files live under /var/lib/nethserver/node/state/node_exporter/ and survive the reboot.

Note: the fix does not migrate old data from /run/node_exporter/, so right after the update the backup status metric is briefly missing (the BackupFailed alert may flap once) until the next backup run writes it to the new path. This one-time gap on update is expected; from then on the status survives reboots.