feat(canopy): N2: blob store backups and paired restore

.workhorse/specs/canopy/backup.md

@@ -50,7 +50,8 @@ There must be exactly one method table.
 
 ```toml
 [simple]                          # snapshot a path as-is
-path = "/var/lib/example"
+path = "/var/lib/example"         # exactly one of path / path_command
+# path_command = ["bestool", "tamanu", "blob-root"]   # resolve the path by command instead
 ```
 
 #### PostgreSQL
@@ -68,6 +69,9 @@ socket = "/var/run/postgresql"    # optional — override the unix socket direct
 ## Methods
 
 The `simple` method hands kopia a path verbatim; it contributes no extra tags and needs no preparation or cleanup.
+The path is either fixed (`path`) or resolved on every run by an argv-style command (`path_command`) whose output must be a single line naming an absolute path; a failed or malformed resolution fails the run.
+Run-time resolution serves a source whose location lives outside the definition and can move under it (the Tamanu blob store root is a database-backed setting an administrator can change, printed by `bestool tamanu blob-root`), so the capture follows the live location instead of a hardcoded path silently going stale.
+At restore, the same resolution names the destination, so a store restored after its database lands where the freshly restored database expects it.
 
 The `postgresql` method takes a crash-consistent physical copy of a postgres cluster, described under "The postgresql method" below.
 
@@ -117,7 +121,7 @@ A run that failed, was skipped because its type was already running, or exited d
 A follower is otherwise an ordinary definition: it registers as a capability, may be scheduled by Canopy or run manually on its own, and reports its runs like any other type.
 
 Following is for a capture that must be a superset of what another capture references.
-A definition that captures data the leader's rows point at follows the leader, so whatever the leader references is already stored when the follower's capture begins.
+The Tamanu blob store definition follows the database definition, so every blob the database capture references is already stored when the store capture begins; blobs are immutable and never removed while referenced, so the store capture can only hold more, never less, than the database capture needs.
 A follower run triggered on its own (by schedule or by hand) is still safe on these terms, being a superset for every earlier capture of the followed type; what only the chain provides is a store capture promptly after each database capture.
 
 ### Sharing a capture
@@ -250,10 +254,10 @@ A restore can equally take its source from a capture held on the device, describ
 Selection is by id across the whole repository — not scoped to the server issuing the restore — so a replacement host can restore a backup taken by the server it succeeds.
 It restores the snapshot into a staging area on the same filesystem as the target so the final move is atomic, then hands off to the method.
 
-Restoring a type also restores its followers, so a leader and the data it references come back as a consistent pair.
+Restoring a type also restores its followers, so a cycle like database-and-blob-store comes back as a consistent pair.
 A follower's snapshot is selected rather than named: the earliest snapshot of the follower's type, from the same source host as the chosen snapshot, taken at or after it.
 At-or-after is the safety rule: a later follower snapshot is a superset of what the restored data references, an earlier one may not be, and is never selected; when none exists at or after, the restore refuses.
-The whole cycle is planned up front, before any data is touched, and each follower restore is then a full restore of its own, with its own credentials, run id, and report, in chain order, so a follower lands against data its leader has just restored.
+The whole cycle is planned up front, before any data is touched, and each follower restore is then a full restore of its own, with its own credentials, run id, and report, in chain order, so a follower whose target is resolved by `path_command` resolves it against data its leader has just restored.
 `--no-followers` restores the named type alone; restoring a follower's type explicitly by snapshot id remains the operator's manual path around a refusal.
 `--target` redirects only the named type's destination while followers would still restore over their live paths, so combining it with planned followers is refused; pass `--no-followers` alongside it.
 Follower snapshots are recognised by the backup type they carry, as a tag or as their description.

crates/bestool/Cargo.toml

@@ -122,6 +122,7 @@ tamanu = [ # enable all tamanu subcommands
 	"tamanu-artifacts",
 	"tamanu-backup",
 	"tamanu-backup-configs",
+	"tamanu-blob-root",
 	"tamanu-config",
 	"tamanu-url",
 	"tamanu-doctor",
@@ -142,6 +143,9 @@ alertd-tamanu = ["alertd", "tamanu-config", "dep:bestool-postgres", "bestool-tam
 tamanu-artifacts = ["__tamanu", "dep:comfy-table", "dep:detect-targets", "dep:target-tuples"]
 tamanu-backup = ["__tamanu", "file", "tamanu-config", "dep:bestool-psql", "dep:algae-cli", "dep:duct"]
 tamanu-backup-configs = ["__tamanu", "tamanu-backup", "dep:walkdir", "dep:zip"]
+# Print the blob store root, resolved from Tamanu's `blobStorage.root` setting
+# in the database, for blob store backup defs to use as their `path_command`.
+tamanu-blob-root = ["__tamanu", "tamanu-config", "dep:bestool-postgres"]
 tamanu-config = ["__tamanu"]
 tamanu-doctor = [
 	"__tamanu",

crates/bestool/USAGE.md

@@ -72,6 +72,7 @@ This document contains the help content for the `bestool` command-line program.
 * [`bestool tamanu artifacts`↴](#bestool-tamanu-artifacts)
 * [`bestool tamanu backup`↴](#bestool-tamanu-backup)
 * [`bestool tamanu backup-configs`↴](#bestool-tamanu-backup-configs)
+* [`bestool tamanu blob-root`↴](#bestool-tamanu-blob-root)
 * [`bestool tamanu config`↴](#bestool-tamanu-config)
 * [`bestool tamanu db-url`↴](#bestool-tamanu-db-url)
 * [`bestool tamanu doctor`↴](#bestool-tamanu-doctor)
@@ -1531,6 +1532,7 @@ Alias: t
 * `artifacts` — List available artifacts for a Tamanu version
 * `backup` — Backup a local Tamanu database to a single file
 * `backup-configs` — Backup local Tamanu-related config files to a zip archive
+* `blob-root` — Print the Tamanu blob store root
 * `config` — Find and print the current Tamanu config
 * `db-url` — Generate a DATABASE_URL connection string
 * `doctor` — Gather server info + healthchecks for a Tamanu install
@@ -1907,6 +1909,22 @@ If `--key` or `--key-file` is provided, the backup file will be encrypted. Note
 
 
 
+## `bestool tamanu blob-root`
+
+Print the Tamanu blob store root.
+
+The root is Tamanu's `blobStorage.root` setting (database-backed and editable in the admin panel, so no config file carries it), resolved against the server package directory when not absolute. A blob store backup def names this command as its `path_command`, so every capture and restore follows the live setting instead of a hardcoded path going stale.
+
+**Usage:** `bestool tamanu blob-root [OPTIONS]`
+
+###### **Options:**
+
+* `-p`, `--package <PACKAGE>` — Package to read the setting for (central-server or facility-server).
+
+   Detected from the config and database when not given.
+
+
+
 ## `bestool tamanu config`
 
 Find and print the current Tamanu config.

crates/bestool/src/actions/canopy/backup/config.rs

@@ -57,7 +57,7 @@ pub struct BackupDef {
 	/// The Canopy backup-type name (label only).
 	pub r#type: String,
 	/// A type this def follows: after a successful run of that type, this def
-	/// runs too, so a pair like database-then-secrets stays ordered.
+	/// runs too, so a pair like database-then-blob-store stays ordered.
 	pub after: Option<String>,
 	/// Extra kopia tags merged with the canopy-* tags.
 	pub tags: BTreeMap<String, String>,
@@ -101,7 +101,10 @@ struct RawDef {
 impl RawDef {
 	fn into_def(self) -> Result<BackupDef> {
 		let method = match (self.simple, self.postgresql) {
-			(Some(simple), None) => Method::Simple(simple),
+			(Some(simple), None) => {
+				simple.validate(&self.r#type)?;
+				Method::Simple(simple)
+			}
 			(None, Some(postgresql)) => Method::Postgresql(postgresql),
 			(None, None) => bail!(
 				"backup def '{}' has no method table; add exactly one of [simple] or [postgresql]",
@@ -269,17 +272,17 @@ mod tests {
 	}
 
 	#[test]
-	fn parses_after() {
+	fn parses_after_and_path_command() {
 		let def = parse_def(
 			r#"
-			type = "tamanu-secrets"
+			type = "tamanu-blobs"
 			after = "tamanu-postgres"
 			[simple]
-			path = "/var/lib/containers/storage/secrets"
+			path_command = ["bestool", "tamanu", "blob-root"]
 			"#,
 		)
 		.unwrap();
-		assert_eq!(def.r#type, "tamanu-secrets");
+		assert_eq!(def.r#type, "tamanu-blobs");
 		assert_eq!(def.after.as_deref(), Some("tamanu-postgres"));
 		assert_eq!(def.method.name(), "simple");
 	}
@@ -355,6 +358,32 @@ mod tests {
 		assert!(format!("{err}").contains("its own"));
 	}
 
+	#[test]
+	fn rejects_simple_with_both_path_forms() {
+		let err = parse_def(
+			r#"
+			type = "bad"
+			[simple]
+			path = "/a"
+			path_command = ["resolve-path"]
+			"#,
+		)
+		.unwrap_err();
+		assert!(format!("{err}").contains("exactly one"));
+	}
+
+	#[test]
+	fn rejects_simple_with_no_path_form() {
+		let err = parse_def(
+			r#"
+			type = "bad"
+			[simple]
+			"#,
+		)
+		.unwrap_err();
+		assert!(format!("{err}").contains("neither"));
+	}
+
 	#[tokio::test]
 	async fn followers_of_selects_by_after_in_type_order() {
 		let dir = std::env::temp_dir().join(format!("bestool-followers-{}", std::process::id()));
@@ -366,8 +395,8 @@ mod tests {
 		.await
 		.unwrap();
 		tokio::fs::write(
-			dir.join("secrets.toml"),
-			"type = \"tamanu-secrets\"\nafter = \"tamanu-postgres\"\n[simple]\npath = \"/srv/secrets\"\n",
+			dir.join("blobs.toml"),
+			"type = \"tamanu-blobs\"\nafter = \"tamanu-postgres\"\n[simple]\npath = \"/srv/blobs\"\n",
 		)
 		.await
 		.unwrap();
@@ -380,8 +409,8 @@ mod tests {
 
 		let followers = followers_of(&dir, "tamanu-postgres").await.unwrap();
 		let types: Vec<&str> = followers.iter().map(|d| d.r#type.as_str()).collect();
-		assert_eq!(types, vec!["assets", "tamanu-secrets"]);
-		assert!(followers_of(&dir, "tamanu-secrets").await.unwrap().is_empty());
+		assert_eq!(types, vec!["assets", "tamanu-blobs"]);
+		assert!(followers_of(&dir, "tamanu-blobs").await.unwrap().is_empty());
 
 		tokio::fs::remove_dir_all(&dir).await.ok();
 	}

crates/bestool/src/actions/canopy/backup/method.rs

@@ -103,10 +103,78 @@ pub(super) enum Teardown {
 }
 
 /// `[simple]` method: snapshot a path verbatim.
+///
+/// The path is either fixed (`path`) or resolved on every run by a command
+/// (`path_command`), for sources whose location lives outside the def, e.g.
+/// the Tamanu blob store root, a database-backed setting an administrator can
+/// move (`bestool tamanu blob-root` prints it). Exactly one of the two.
 #[derive(Debug, Clone, Deserialize)]
 pub struct SimpleConfig {
 	/// The path kopia snapshots.
-	pub path: PathBuf,
+	#[serde(default)]
+	pub path: Option<PathBuf>,
+	/// A command (argv-style, no shell) whose output is the absolute path to
+	/// snapshot and restore.
+	#[serde(default)]
+	pub path_command: Option<Vec<String>>,
+}
+
+impl SimpleConfig {
+	/// Enforce exactly one of `path` / `path_command` at def load.
+	pub fn validate(&self, backup_type: &str) -> Result<()> {
+		match (&self.path, &self.path_command) {
+			(Some(_), None) => Ok(()),
+			(None, Some(command)) if !command.is_empty() => Ok(()),
+			(None, Some(_)) => bail!("backup def '{backup_type}' has an empty [simple] path_command"),
+			(None, None) => bail!(
+				"backup def '{backup_type}' has a [simple] table with neither path nor path_command"
+			),
+			(Some(_), Some(_)) => bail!(
+				"backup def '{backup_type}' has both [simple] path and path_command; exactly one is allowed"
+			),
+		}
+	}
+
+	/// The path to snapshot or restore: the fixed one, or the command's output.
+	///
+	/// spec: BAK#methods
+	pub async fn resolve_path(&self) -> Result<PathBuf> {
+		use miette::{Context as _, IntoDiagnostic as _};
+
+		if let Some(path) = &self.path {
+			return Ok(path.clone());
+		}
+		let command = self
+			.path_command
+			.as_ref()
+			.expect("validated: path or path_command is set");
+		let (program, args) = command
+			.split_first()
+			.expect("validated: path_command is not empty");
+		let output = tokio::process::Command::new(program)
+			.args(args)
+			.output()
+			.await
+			.into_diagnostic()
+			.wrap_err_with(|| format!("running path_command {program}"))?;
+		if !output.status.success() {
+			bail!(
+				"path_command {program} exited with {}: {}",
+				output.status,
+				String::from_utf8_lossy(&output.stderr).trim()
+			);
+		}
+		let stdout = String::from_utf8_lossy(&output.stdout);
+		let path = stdout.trim();
+		if path.is_empty() || path.lines().count() != 1 {
+			bail!("path_command {program} must output exactly one line, got: {path:?}");
+		}
+		let path = PathBuf::from(path);
+		if !path.is_absolute() {
+			bail!("path_command {program} must output an absolute path, got {}", path.display());
+		}
+		Ok(path)
+	}
 }
 
 /// `[postgresql]` method: physical, crash-consistent cluster snapshot.
@@ -183,7 +251,7 @@ impl Method {
 	pub async fn prepare(&self, backup_type: &str, within: Option<&VolumeCapture>) -> Result<Prepared> {
 		match self {
 			Method::Simple(config) => {
-				let live = config.path.clone();
+				let live = config.resolve_path().await?;
 				let frozen = within.and_then(|capture| capture.contains(&live));
 				if let (Some(source), Some(capture)) = (&frozen, within) {
 					info!(
@@ -289,7 +357,7 @@ impl Method {
 			Method::Simple(config) => {
 				let target = match target_override {
 					Some(target) => target.to_path_buf(),
-					None => config.path.clone(),
+					None => config.resolve_path().await?,
 				};
 				target
 					.parent()
@@ -309,7 +377,7 @@ impl Method {
 			Method::Simple(config) => {
 				let target = match &opts.target {
 					Some(target) => target.clone(),
-					None => config.path.clone(),
+					None => config.resolve_path().await?,
 				};
 				ensure_not_clobbering(&target, opts.clobber)?;
 				replace_dir(staging, &target).await
@@ -465,7 +533,8 @@ mod tests {
 	#[tokio::test]
 	async fn simple_prepare_returns_its_path_and_no_tags() {
 		let method = Method::Simple(SimpleConfig {
-			path: PathBuf::from("/data/custom"),
+			path: Some(PathBuf::from("/data/custom")),
+			path_command: None,
 		});
 		let prepared = method.prepare("custom", None).await.unwrap();
 		assert_eq!(prepared.path, PathBuf::from("/data/custom"));
@@ -516,6 +585,83 @@ mod tests {
 		assert_eq!(c_volume().contains(Path::new(r"C:\")), Some(c_volume().root));
 	}
 
+	#[test]
+	fn simple_config_requires_exactly_one_path_form() {
+		let fixed = SimpleConfig {
+			path: Some(PathBuf::from("/data")),
+			path_command: None,
+		};
+		assert!(fixed.validate("t").is_ok());
+
+		let resolved = SimpleConfig {
+			path: None,
+			path_command: Some(vec!["/bin/echo".into(), "/data".into()]),
+		};
+		assert!(resolved.validate("t").is_ok());
+
+		let neither = SimpleConfig {
+			path: None,
+			path_command: None,
+		};
+		assert!(format!("{}", neither.validate("t").unwrap_err()).contains("neither"));
+
+		let both = SimpleConfig {
+			path: Some(PathBuf::from("/data")),
+			path_command: Some(vec!["/bin/echo".into()]),
+		};
+		assert!(format!("{}", both.validate("t").unwrap_err()).contains("exactly one"));
+
+		let empty = SimpleConfig {
+			path: None,
+			path_command: Some(vec![]),
+		};
+		assert!(format!("{}", empty.validate("t").unwrap_err()).contains("empty"));
+	}
+
+	#[cfg(unix)]
+	#[tokio::test]
+	async fn path_command_resolves_trimmed_absolute_output() {
+		let config = SimpleConfig {
+			path: None,
+			path_command: Some(vec!["/bin/echo".into(), "/var/lib/tamanu/blobs".into()]),
+		};
+		assert_eq!(
+			config.resolve_path().await.unwrap(),
+			PathBuf::from("/var/lib/tamanu/blobs")
+		);
+	}
+
+	#[cfg(unix)]
+	#[tokio::test]
+	async fn path_command_rejects_relative_and_multiline_output() {
+		let relative = SimpleConfig {
+			path: None,
+			path_command: Some(vec!["/bin/echo".into(), "data/blobs".into()]),
+		};
+		assert!(
+			format!("{}", relative.resolve_path().await.unwrap_err()).contains("absolute")
+		);
+
+		let multiline = SimpleConfig {
+			path: None,
+			path_command: Some(vec!["/bin/echo".into(), "/a\n/b".into()]),
+		};
+		assert!(
+			format!("{}", multiline.resolve_path().await.unwrap_err())
+				.contains("exactly one line")
+		);
+	}
+
+	#[cfg(unix)]
+	#[tokio::test]
+	async fn path_command_failure_is_an_error() {
+		let failing = SimpleConfig {
+			path: None,
+			path_command: Some(vec!["/bin/sh".into(), "-c".into(), "exit 3".into()]),
+		};
+		assert!(format!("{}", failing.resolve_path().await.unwrap_err()).contains("exited"));
+	}
+
 	#[test]
 	fn clobber_guard_blocks_occupied_dir_unless_forced() {
 		let tmp = tempfile::tempdir().unwrap();