Skip to content

Sign release images and verify signatures in the updater; ship schema migrations only in new images #188

Open

Description

@jatmn

Problem

The in-app updater (crates/updater/src/oci.rs) checks every manifest, config and layer download against its sha256 digest, checks the image labels, and checks the binary against the files.sha256 list inside the image. It does not verify any signature. Because that checksum list ships inside the same image, it catches corruption but not a bad publish. If a malicious or broken image were pushed to the registry, the updater would install it and it would run as the bot or the API. This applies to both the current two-app layout and a combined single-app layout.

There is also an availability risk. When a pod is replaced (a crash, a host move, or any rollout), the launcher downloads the recorded release from the registry again and refuses to start a binary whose schema range doesn't match the database. If the registry is unreachable, or a self-update ran a schema migration that the image's built-in binary can't open, the bot stays stopped until an owner steps in. That conflicts with the rule that the bot must always be able to moderate.

Proposed

  1. Sign each release image in CI (for example with cosign keyless signing or a pinned key).
  2. Have the updater verify the signature against a pinned key or identity before staging anything. Refuse unsigned or mismatched images, and report a clear status.
  3. Never let the in-app updater run a schema migration. Migrations ship only in new container images. App-only updates through the updater must stay within the current schema range.
  4. After an in-app update, cut a matching container image soon after, so a pod replacement doesn't depend on a fresh registry download.
  5. Keep JMODGUARD_SELF_UPDATE, JMODGUARD_UPDATE_APPLY_ENABLED and JMODGUARD_UNATTENDED_UPDATES off by default until 1-3 are done.

Acceptance

  • Tests show that an unsigned or wrongly signed image is refused.
  • Tests show that an app-only update that requires a schema change is refused by the updater.
  • The docs (docs/BUNNY-UPDATES.md) describe the signing setup and the images-only migration rule.

Activity

  1. jatmn commented on Oct 9, 2026

    @jatmn
    OwnerAuthor

    Owner decision: the update loop stays off by default and only runs on a manual trigger, both before and after signing lands. Unattended updates are not part of the plan. Signing is an extra layer on top of manual confirmation, not a replacement for it.

  2. jatmn commented on Oct 9, 2026

    @jatmn
    OwnerAuthor

    Owner direction on the boot repair path: the launcher's startup step that re-downloads the recorded release (so a pod replacement doesn't roll back an in-app update) should stay automatic and should not depend on the unattended-update setting. It already behaves this way today. Two additions belong in this issue:

    1. The boot repair download must pass the same signature check as a manual install once signing lands.
    2. If the registry is unreachable at boot, fall back to the image's built-in binary when its schema range fits the database, and log loudly. Stop only when no available binary can open the database, so moderation isn't blocked by a registry outage.
  3. jatmn commented on Oct 9, 2026

    @jatmn
    OwnerAuthor

    Adding one more requirement for the boot repair fallback:

    Version-mismatch alert channel

    • If boot repair can't reach the registry (or signature verification fails) and the bot or API starts on a version older than the last recorded release, the bot posts an alert to an owner-chosen alert channel.
    • The alert says which component is affected, the version it's running, the version it expected, and why repair failed (registry unreachable, bad signature, schema mismatch), so the owner/admin can act by hand.
    • New web UI setting to pick that alert channel, saved through the bot over the control link like other channel selections (the API stays read-only).
    • Post once per boot or mismatch state, not on every retry, and clear once versions match again.
    • If no alert channel is set, log a warning and show the mismatch in the web UI instead.

    — jatmn

  4. jatmn commented on Oct 9, 2026

    @jatmn
    OwnerAuthor

    Two known limits from #185 to fix here, before self-update is ever turned on (neither can happen while it stays off, which is the default):

    1. Paired bot+API upgrades stall at the API's "prepared" step. After feat(bot): keep database writes on the bot process #185 the API launcher writes nothing, so a paired upgrade can't advance past the API side.
    2. Owner recovery after a failed paired upgrade can't reach a stopped bot. If a failed paired upgrade stops both apps, the owner's recovery action goes over the control link to the bot and returns "control link unavailable."

    Likely direction: route upgrade-state writes and owner recovery through something that keeps running while the bot app is down (e.g. the bot launcher), still as the only writer and still lease-fenced.

    — jatmn

  5. jatmn commented on Oct 9, 2026

    @jatmn
    OwnerAuthor

    Correction to my last comment: in production only unattended updates are off. Self-update and update-apply stay enabled, so the owner's manual Update button can reach both gaps today. Until this issue lands, #185 will make the manual install refuse paired bot+API upgrades with a clear message (single-app updates keep working). Removing that guard is part of finishing this issue.

    — jatmn

  6. jatmn commented on Oct 28, 2026

    @jatmn
    OwnerAuthor

    Work guidelines for this issue

    This issue holds several separate deliverables. Each one is its own PR, done in this order, with only one open at a time.

    1. Sign release images in CI. Done in Sign release images #379.
    2. Refuse in-app schema migrations. Done in Refuse in-app database schema migrations #380.
    3. The updater verifies the image signature against the pinned key before staging, and refuses unsigned or mismatched images with a clear status. Tests cover refusal. The OCI layer handling stays as it is unless a test proves the signature can't be checked without changing it. Publish and CI pipeline changes are out of scope, because they belong to Sign release images #379 and Publish: decide all-or-nothing promotion for signed releases #387.
    4. Boot repair version-mismatch alert to the owner's chosen channel (from the comment above).
    5. The paired bot and API upgrade gaps from feat(bot): keep database writes on the bot process #185 (from the comment above).
    6. Cut a matching container image after an in-app update.

    Rules for every PR above:

    • Stay in scope: don't merge in other branches or stack PRs. Each PR targets main and covers one numbered item.
    • Target size: about 800 changed lines or less. If a PR grows past that, stop and ask the owner.
    • Review rounds: after 3 rounds, stop fixing and audit for drift. A finding counts only if it breaks something this issue, the docs or an existing test states.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions