Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
387eee1
record reporting schema builds
dannash100 Sep 6, 2026
29ad8a9
dispatch and report schema builds
dannash100 Sep 6, 2026
e448ee6
builders register their group's artifacts
dannash100 Sep 6, 2026
8f9084b
grade unbuilt reporting schemas
dannash100 Sep 6, 2026
a14a5e0
test reporting schema pairs
dannash100 Sep 6, 2026
a2a9f6b
present pairs to operators
dannash100 Sep 6, 2026
30dce51
test schema dispatch and publishing
dannash100 Sep 6, 2026
40d2739
describe the schema api
dannash100 Sep 6, 2026
3b78fb7
rebuild when a version's artifacts move
dannash100 Sep 6, 2026
a5c780f
cover schema alerting
dannash100 Sep 7, 2026
4d77c40
test the pairs section
dannash100 Sep 7, 2026
4b5ec0b
cover builder authorisation
dannash100 Sep 7, 2026
980f6ad
fix pair derivation
dannash100 Sep 7, 2026
e286119
cover the build report
dannash100 Sep 7, 2026
d12c030
cover the schema round trip
dannash100 Sep 7, 2026
7971f28
seed reporting schema builds
dannash100 Sep 7, 2026
51184fe
Merge remote-tracking branch 'origin/feat/group-scoped-artifacts' int…
dannash100 Sep 7, 2026
6afbb5d
refuse a schema range
dannash100 Sep 7, 2026
6811260
cover the planned pair
dannash100 Sep 7, 2026
821c8b8
heading level and doc placement
dannash100 Sep 7, 2026
790c580
never redact a schema build
dannash100 Sep 7, 2026
0c60b4c
Merge remote-tracking branch 'origin/feat/group-scoped-artifacts' int…
dannash100 Sep 7, 2026
25c4f5b
pin what is not a pair
dannash100 Sep 7, 2026
7add5f3
settling ignores the snapshot
dannash100 Sep 7, 2026
c08eea6
Merge remote-tracking branch 'origin/feat/group-scoped-artifacts' int…
dannash100 Sep 7, 2026
f74ae93
settle what clears the check
dannash100 Sep 7, 2026
dad51b0
name a pair's servers
dannash100 Sep 7, 2026
51c2e99
cover the servers column
dannash100 Sep 8, 2026
18846bd
Merge branch 'feat/group-scoped-artifacts' into feat/reporting-schemas
dannash100 Sep 8, 2026
1c64e62
keep a build's output out of its own trigger
dannash100 Sep 8, 2026
77f0b36
Merge remote-tracking branch 'origin/feat/group-scoped-artifacts' int…
dannash100 Sep 8, 2026
b8de277
tighten artifact registration
dannash100 Sep 8, 2026
98e949a
count range artifacts
dannash100 Sep 8, 2026
9aa0f52
index artifact lookups
dannash100 Sep 8, 2026
29de010
settle pairs per group
dannash100 Sep 8, 2026
5cb2ae0
unstick a doc comment
dannash100 Sep 8, 2026
f257a06
count every pair
dannash100 Sep 8, 2026
f3b4c68
gate schema publishing
dannash100 Sep 9, 2026
d9c2ce5
dispatch marked builders
dannash100 Sep 9, 2026
054b733
expose publish flag
dannash100 Sep 9, 2026
218c238
add publish switch
dannash100 Sep 9, 2026
a543e55
respec schema publishing
dannash100 Sep 9, 2026
a347b52
spell out intent semantics
dannash100 Sep 9, 2026
83615a3
Merge branch 'feat/group-scoped-artifacts' into feat/reporting-schemas
dannash100 Sep 9, 2026
33b7d95
split group registration out
dannash100 Sep 9, 2026
2f1024c
Merge branch 'feat/group-scoped-artifacts' into feat/reporting-schemas
dannash100 Sep 9, 2026
ac4ae31
one publisher and one sweep query
dannash100 Sep 9, 2026
4e98362
narrow the schema sweep
dannash100 Sep 10, 2026
e298275
keep a later ask
dannash100 Sep 10, 2026
1c0d2f7
guard the publisher mark
dannash100 Sep 10, 2026
cc74c5d
load a group's members once
dannash100 Sep 10, 2026
71f8a29
narrow the version lookup
dannash100 Sep 10, 2026
9a6693d
index restore checks by run
dannash100 Sep 10, 2026
af1b740
refuse an ask with no pair
dannash100 Sep 10, 2026
cd3a717
read range artifacts once
dannash100 Sep 10, 2026
0e75194
hide the build button from viewers
dannash100 Sep 10, 2026
f709f79
regenerate the private api
dannash100 Sep 10, 2026
867c581
render the section without a provider
dannash100 Sep 10, 2026
da2f502
Merge remote-tracking branch 'origin/feat/group-scoped-artifacts' int…
dannash100 Sep 10, 2026
ec86842
Merge remote-tracking branch 'origin/feat/group-scoped-artifacts' int…
dannash100 Sep 14, 2026
860a5ec
Merge remote-tracking branch 'origin/feat/group-scoped-artifacts' int…
dannash100 Sep 14, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .workhorse/specs/platform/artifacts.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,9 @@ Canopy passes a group-scoped artifact's bytes only to a caller it is offered to.

## Registration

A registration names the version or range, the type, the platform, and the group where the artifact has one, and carries either the location of an unscoped artifact or the bytes of a group-scoped one.
Registering an unscoped artifact and registering a group's are separate: they name different things, carry different bodies, and are authorised differently, so each is its own path rather than one path that changes shape on a parameter.
An unscoped registration names the version or range, the type and the platform, and carries the artifact's location.
A group-scoped one names the group as well, carries the artifact's bytes, and names an exact version Canopy already holds rather than drafting one.
The group is named on the registration rather than inferred from the caller.

A releaser device registers unscoped artifacts, and carries no authorisation for any group.
Expand Down
14 changes: 8 additions & 6 deletions .workhorse/specs/public-server/reporting-schemas.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ A **schema builder** produces a reporting schema from a database Canopy has rest
It is a restore consumer (see [RST](restore-replicas.md)): a build operates on a replica, so the builder is dispatched, credentialled, and reports over the replica pathways and authorisations, and it advertises an intent carrying `reporting-schema`.
How the builder produces a schema is the builder's own.

An **operator** declares which groups have a builder, reads which schema each application runs, and asks for the builds the derivation does not produce.
An **operator** declares which groups have a builder, marks the declaration that publishes a group's schema, reads which schema each application runs, and asks for the builds the derivation does not produce.

The **device of a machine a Tamanu application runs on** fetches the schema Canopy offers that application and applies it (see [DID](machine-identity.md)).

Expand All @@ -30,8 +30,8 @@ Canopy owns which pairs exist, the replica a build is given, the artifact that r
## Pairs

A reporting schema is unique per pair of group and Tamanu version, and Canopy holds zero or one per pair.
The pairs are, for each group covered by an enabled declaration of a `reporting-schema` intent, each version a Tamanu application of the group reports running and the version its open plan moves it to (see [UPG](../private-server/upgrade-plans.md)).
That declaration is what covers a group: it names the group, is enabled or disabled, and is audited (see [RST](restore-replicas.md)).
The pairs are, for each group covered by an enabled declaration marked as publishing its schema, each version a Tamanu application of the group reports running and the version its open plan moves it to (see [UPG](../private-server/upgrade-plans.md)).
That declaration is what covers a group: it names the group, is enabled or disabled, carries the operator's mark, and is audited (see [RST](restore-replicas.md)).
Only a published version is in a pair, since a version's migrations reach a builder as its published artifacts (see [ART](../platform/artifacts.md)) and an unpublished one has none.

A pair with no schema is built, and a pair with one is settled.
Expand All @@ -50,8 +50,10 @@ The replica is migrated to the named version before the build reads it, and is n
The builder obtains read credentials for the restore per run as any consumer does, and no storage credential of any kind for what it publishes (see [RST](restore-replicas.md)).

In the run it reports, the builder registers the **reporting schema** as an artifact of the exact version being built for, scoped to the group, of type `reporting-schema` on platform `any`, carrying a digest and the bytes themselves, which Canopy holds and serves (see [ART](../platform/artifacts.md)).
It may register further artifacts beside the schema for the same version and group, under types of its choosing, which Canopy offers as it offers any artifact.
The builder is authorised to register artifacts for a group its enabled `reporting-schema` declaration covers and for no other, and is the one device other than a releaser that registers artifacts (see [ART](../platform/artifacts.md)).
The schema is the only type it may register: what is published for a group is offered to every machine in it and fetched, so the authorisation stays defined with the artifact it was written for.
The builder is authorised to register artifacts for a group whose enabled declaration an operator has marked as publishing its reporting schema, and for no other, and is the one device other than a releaser that registers artifacts (see [ART](../platform/artifacts.md)).
The mark is the operator's alone, and is the whole of the authorisation: a consumer registers the set of semantics it advertises itself, so they shape what Canopy dispatches to it and grant it nothing, and what is published for a group is offered to every machine in it and applied.
Only a group-wide, non-redacting declaration of an intent carrying `reporting-schema` can carry the mark, which is the same declaration a build is dispatched for, so Canopy asks for no build it would refuse the result of.

A schema is published for the exact version and never for a range, since it follows from the migrations that version applies, and one built against a patch is not the schema another patch of the same minor describes.

Expand Down Expand Up @@ -80,6 +82,6 @@ The device compares what its application runs with what it is offered, applies t
A failed build raises a reporting-schema check on the group's central Tamanu application, carrying the failure description (see [CHK](../monitoring/checks.md)).
The check is a warning rather than a failure, and does not escalate: the application is up and its reports return rows, and a schema that cannot be built for the version its group is moving to is for whoever maintains the reports rather than whoever is on call.
A replica that failed to restore or come up is the restore's own health rather than a build failure, and is dispatched again as any unhealthy restore is.
The check recovers when the pair is built, and an operator asking for the build is what clears it.
The check recovers when the pair is built, and nothing else clears it: an operator asking for the build puts the pair back on the worklist, and the warning stands until a build lands, since asking changes nothing about whether the group's applications can be offered a schema.

Pairs are presented per group, showing which have a schema, which are being built, and which failed, so whether a group's applications can be offered the schema for the version they run or are moving to is answered in one place.
2 changes: 2 additions & 0 deletions .workhorse/specs/public-server/restore-replicas.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,7 @@ The recognised semantics are:
`once` for such an intent is keyed to the snapshot and the target version together (see [Pre-upgrade migration testing](#pre-upgrade-migration-testing)).
- **reporting-schema** — the intent builds a Tamanu reporting schema from the replica it restores and registers it as an artifact (see [RPT](reporting-schemas.md)).
It carries `migrate` alongside, and its entries name the version of the pair of group and Tamanu version being built for, on a central server of the group, rather than the server's candidate.
Canopy dispatches a build only for a declaration an operator has marked as publishing its group's schema, and accepts a published schema only from one (see [RPT](reporting-schemas.md)).
`once` for such an intent is keyed to the group and the version rather than the snapshot, so a newer snapshot does not rebuild a schema the pair already has, and a failed build settles the pair.
A settled pair is reinstated when the version's artifacts change or an operator asks for the build (see [RPT](reporting-schemas.md)).
- **redact** — the intent can de-identify the restored data before serving it.
Expand Down Expand Up @@ -129,6 +130,7 @@ Each declaration carries:
- a human-readable **name**, distinct from every other declaration assigned to the same consumer;
- **parameter values** for the intent's schema, defaulted where the schema provides one;
- whether the replica **redacts**, offered only for an intent carrying `redact` (see [Redaction](#redaction));
- whether the declaration **publishes its group's reporting schema**, offered only for a group-wide, non-redacting declaration of an intent carrying `reporting-schema` (see [RPT](reporting-schemas.md));
- an **overdue bound**: the maximum time a replica may go without meeting its intent's health expectation before Canopy considers it overdue, interpreted per the intent's semantics (see [Alerting](#alerting));
- whether the declaration is **enabled**.

Expand Down
119 changes: 114 additions & 5 deletions crates/canopy-api/src/generated.rs
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ pub const OPENAPI_VERSION: &str = "1.0.0";

/// BLAKE3 digest of that document, so a document that changed without the
/// version moving with it can be told from one that did not.
pub const OPENAPI_BLAKE3: &str = "626c2115b5bedb1e43372e734c98bcbf018e1c8f5ccb81683f2d1b13c4ddcee1";
pub const OPENAPI_BLAKE3: &str = "3cfc9a09c64793448410328da3f666a868f4ba77d17b0f6b9031e7c9b669be3b";

/// Error types.
pub mod error {
Expand Down Expand Up @@ -1440,7 +1440,7 @@ opts into and the settings it accepts per replica.*/
/// "$ref": "#/components/schemas/BTreeMap"
/// },
/// "semantics": {
/// "description": "Behaviours this intent opts into. Recognised values are `check` (a\nhealth report is expected for each replica), `once` (a given snapshot\nis only ever dispatched to a replica once, rather than repeatedly\nuntil overdue), and `url` (a replica's health report includes a link\nto it). Unrecognised values are stored but have no effect.",
/// "description": "Behaviours this intent opts into. Recognised values are `check` (a\nhealth report is expected for each replica), `once` (a given snapshot\nis only ever dispatched to a replica once, rather than repeatedly until\noverdue), `url` (a replica's health report includes a link to it),\n`migrate` (Canopy names a target version and the replica applies that\nversion's migrations), `redact` (the replica de-identifies the restored\ndata before serving it), and `reporting-schema` (the replica builds a\nTamanu reporting schema and registers it for the group). Unrecognised\nvalues are stored but have no effect, so a consumer may advertise ahead\nof Canopy support.",
/// "type": "array",
/// "items": {
/// "type": "string"
Expand All @@ -1466,9 +1466,14 @@ parameter name.*/
pub params: ::std::option::Option<BTreeMap>,
/**Behaviours this intent opts into. Recognised values are `check` (a
health report is expected for each replica), `once` (a given snapshot
is only ever dispatched to a replica once, rather than repeatedly
until overdue), and `url` (a replica's health report includes a link
to it). Unrecognised values are stored but have no effect.*/
is only ever dispatched to a replica once, rather than repeatedly until
overdue), `url` (a replica's health report includes a link to it),
`migrate` (Canopy names a target version and the replica applies that
version's migrations), `redact` (the replica de-identifies the restored
data before serving it), and `reporting-schema` (the replica builds a
Tamanu reporting schema and registers it for the group). Unrecognised
values are stored but have no effect, so a consumer may advertise ahead
of Canopy support.*/
#[serde(default, skip_serializing_if = "::std::vec::Vec::is_empty")]
pub semantics: ::std::vec::Vec<::std::string::String>,
}
Expand Down Expand Up @@ -2423,6 +2428,78 @@ run: if progress reports already carried it, that value stands.*/
#[serde(rename = "type")]
pub type_: ::std::string::String,
}
///What a reporting-schema build reports beyond its replica's restore health.
///
/// <details><summary>JSON schema</summary>
///
/// ```json
///{
/// "description": "What a reporting-schema build reports beyond its replica's restore health.",
/// "type": "object",
/// "required": [
/// "built"
/// ],
/// "properties": {
/// "artifacts": {
/// "description": "The artifacts the build registered, of which the schema is one.",
/// "type": "array",
/// "items": {
/// "type": "string",
/// "format": "uuid"
/// }
/// },
/// "built": {
/// "description": "Whether a schema came out of the build.",
/// "type": "boolean"
/// },
/// "error": {
/// "description": "What went wrong, where the build failed.",
/// "type": [
/// "string",
/// "null"
/// ]
/// },
/// "target_version": {
/// "description": "The version the schema was built for, as semver, echoed from the\nworklist entry's `target_version`.",
/// "type": [
/// "string",
/// "null"
/// ]
/// },
/// "target_version_id": {
/// "description": "The same version as the identifier, echoed from `target_version_id`.\nAccepted for a consumer that reports the identifier; omit it when\n`target_version` is sent.",
/// "type": [
/// "string",
/// "null"
/// ],
/// "format": "uuid"
/// }
/// }
///}
/// ```
/// </details>
#[derive(::serde::Deserialize, ::serde::Serialize, Clone, Debug)]
#[derive(::bon::Builder)]
#[non_exhaustive]
pub struct ReportingSchemaArgs {
///The artifacts the build registered, of which the schema is one.
#[serde(default, skip_serializing_if = "::std::vec::Vec::is_empty")]
pub artifacts: ::std::vec::Vec<::uuid::Uuid>,
///Whether a schema came out of the build.
pub built: bool,
///What went wrong, where the build failed.
#[serde(default, skip_serializing_if = "::std::option::Option::is_none")]
pub error: ::std::option::Option<::std::string::String>,
/**The version the schema was built for, as semver, echoed from the
worklist entry's `target_version`.*/
#[serde(default, skip_serializing_if = "::std::option::Option::is_none")]
pub target_version: ::std::option::Option<::std::string::String>,
/**The same version as the identifier, echoed from `target_version_id`.
Accepted for a consumer that reports the identifier; omit it when
`target_version` is sent.*/
#[serde(default, skip_serializing_if = "::std::option::Option::is_none")]
pub target_version_id: ::std::option::Option<::uuid::Uuid>,
}
///A request to certify a key for a name.
///
/// <details><summary>JSON schema</summary>
Expand Down Expand Up @@ -3361,6 +3438,17 @@ impl ::std::fmt::Display for UrlField {
/// "type": "string",
/// "format": "uuid"
/// },
/// "reporting_schema": {
/// "oneOf": [
/// {
/// "type": "null"
/// },
/// {
/// "description": "What a reporting-schema build produced, where the replica was restored\nfor one. Absent on any other report.",
/// "$ref": "#/components/schemas/ReportingSchemaArgs"
/// }
/// ]
/// },
/// "run_id": {
/// "description": "This must be the run-uuid the client minted for this run.\nThe field is optional only so older clients don't break; it WILL be made\nmandatory in future.",
/// "type": [
Expand Down Expand Up @@ -3469,6 +3557,8 @@ checks. A replica only counts as verified when the outcome is
type, and intent, so a report that named no declaration could not be
attributed to one of them.*/
pub replica_id: ::uuid::Uuid,
#[serde(default, skip_serializing_if = "::std::option::Option::is_none")]
pub reporting_schema: ::std::option::Option<ReportingSchemaArgs>,
/**This must be the run-uuid the client minted for this run.
The field is optional only so older clients don't break; it WILL be made
mandatory in future.*/
Expand Down Expand Up @@ -3970,6 +4060,25 @@ impl<T: crate::CanopyTransport> crate::CanopyClient<T> {
pub async fn applications_self(&self) -> crate::Result<SelfResponse> {
self.call_json(::http::Method::GET, "/applications/self", None::<&()>).await
}
/// Register a reporting schema for one group, carrying its bytes.
///
/// Requires a device certificate whose restore declaration for the named group
/// advertises that it builds reporting schemas. The bytes travel on this
/// connection and Canopy holds them, so the builder is issued no credential to
/// any store. The path names the group the artifact is for, the exact version
/// it was built against, and the artifact's type and target platform.
///
/// The version must be one Canopy already holds: a build is dispatched for a
/// group and version Canopy knows about, so a version that does not exist is
/// refused rather than drafted. A range pattern is refused for the same reason:
/// a schema follows the migrations one exact version applies.
///
/// Returns the created artifact record.
///
/// `POST /artifacts/groups/{group}/{version}/{artifact_type}/{platform}`
pub async fn artifacts_groups(&self, group: &str, version: &str, artifact_type: &str, platform: &str) -> crate::Result<Artifact> {
self.call_json(::http::Method::POST, &format!("/artifacts/groups/{}/{}/{}/{}", group, version, artifact_type, platform), None::<&()>).await
}
/// Register a downloadable artifact for a version or version range.
///
/// Requires a device certificate with the releaser role (or admin). The
Expand Down
18 changes: 15 additions & 3 deletions crates/commons-types/src/backup.rs
Original file line number Diff line number Diff line change
Expand Up @@ -418,6 +418,13 @@ pub mod semantics {
/// worklist entry, withholds an entry from a server whose product has no
/// manifest, and holds the replica to the redaction outcome reported back.
pub const REDACT: &str = "redact";
/// The intent builds a Tamanu reporting schema from the replica it restores
/// and registers it as a group-scoped artifact: Canopy names the pair's
/// version on the worklist entry, restores a machine of the group running a
/// central Tamanu application, and keys `once` to the group and the version
/// rather than the snapshot.
// spec: RPT
pub const REPORTING_SCHEMA: &str = "reporting-schema";
}

/// The parameters Canopy owns on behalf of the `redact` semantic.
Expand Down Expand Up @@ -498,9 +505,14 @@ pub struct IntentDescriptor {
pub description: Option<String>,
/// Behaviours this intent opts into. Recognised values are `check` (a
/// health report is expected for each replica), `once` (a given snapshot
/// is only ever dispatched to a replica once, rather than repeatedly
/// until overdue), and `url` (a replica's health report includes a link
/// to it). Unrecognised values are stored but have no effect.
/// is only ever dispatched to a replica once, rather than repeatedly until
/// overdue), `url` (a replica's health report includes a link to it),
/// `migrate` (Canopy names a target version and the replica applies that
/// version's migrations), `redact` (the replica de-identifies the restored
/// data before serving it), and `reporting-schema` (the replica builds a
/// Tamanu reporting schema and registers it for the group). Unrecognised
/// values are stored but have no effect, so a consumer may advertise ahead
/// of Canopy support.
#[serde(default)]
pub semantics: Vec<String>,
/// Configurable parameters this intent accepts per replica, keyed by
Expand Down
9 changes: 9 additions & 0 deletions crates/database/src/applications.rs
Original file line number Diff line number Diff line change
Expand Up @@ -800,6 +800,15 @@ impl Application {
.map_err(AppError::from)
}

/// What to call this application to an operator: the name it was given,
/// else the host it answers on, else its id.
pub fn label(&self) -> String {
self.name
.clone()
.or_else(|| self.host.as_ref().map(|h| h.0.to_string()))
.unwrap_or_else(|| self.id.to_string())
}

/// All live (non-archived) applications in a group, ordered by name. Used to
/// expand a group-wide restore-replica declaration into per-server entries.
pub async fn list_live_in_group(
Expand Down
Loading