Skip to content

ci(mobile): roll back or republish OTA updates from the release workflow, and gate publishes on a boot test - #1663

Merged
janicduplessis merged 2 commits into
fix/1661-notify-state-upgradefrom
ci/1657-mobile-rollback
Sep 27, 2026
Merged

janicduplessis merged 2 commits into
fix/1661-notify-state-upgradefrom
ci/1657-mobile-rollback

Conversation

@janicduplessis

@janicduplessis janicduplessis commented Sep 27, 2026 •

Copy link
Copy Markdown
Collaborator

Description

The OTA updates for #1650 and #1655 crashed the TestFlight app at launch. mode=auto had published them to channel production with no check that the JS boots, and the rollback had to be run by hand outside the workflow. This PR adds rollback and republish modes to mobile-release.yml, and a Jest boot test that runs before anything is published.

Stacked on #1662, which fixes the crash itself. Without it, the boot test fails on main.

Solution

Rollback and republish modes. Both run under the same release environment approval and on-main check as the other modes.

  • mode=rollback runs eas update:roll-back-to-embedded --channel production --runtime-version <rt> --platform ios. The runtime is the runtime input, or the latest finished production build's (the same build:list query auto uses).
  • mode=republish -f group=<id> validates the id, logs the group's branch, runtime and message from eas update:view, and runs eas update:republish --group <id> --platform ios. It warns when the group's runtime is not the latest build's. A group from another branch goes to production with --destination-channel production.

Neither mode runs the tests, since a rollback has to work while main is broken. They use their own concurrency group, so they don't queue behind a 2-hour build. Each writes a ::notice:: naming the runtime and the group it acted on.

Boot gate. src/app-boot.test.tsx renders the whole Expo Router tree, drawer menu included, in Jest, starting from the state the previous release stored on the phone. It fails on anything thrown or any console.error from React, and asserts that the notifier processed the live status. mobile.yml already runs pnpm test. mobile-release.yml now runs it too, for every publishing mode, before EXPO_TOKEN is loaded.

  • Setup: a paired machine, an older stim's cached status, and notification prefs and state in their old shapes. A fake socket then serves mock-server/fixtures/status.json, and the app goes inactive -> active as it does on iOS.
  • New dev deps: @testing-library/react-native@14.0.1 and its peer test-renderer. There is also a Jest moduleNameMapper for @/assets/*.
  • The test stubs Reanimated, the drawer and Lottie. The Reanimated mock that expo-router/testing-library installs fails to load on 4.7.

On main (without #1662) it reproduces the production crash:

TypeError: Cannot read properties of undefined (reading '/Users/dev/Developer/stim')
  at oversee (src/lib/oversight.ts:627)

Trade-offs (reasoning is in the README runbook):

  • No staging channel: promoting from staging only verifies something if a build reads that channel and a person checks it.
  • No macOS simulator smoke run: it needs a simulator build per runtime and a macOS runner per update. The boot test covers JS that throws at launch, but not native crashes or screens other than home.

Runbook. A "Roll back a bad update" section in apps/mobile/README.md covers detection, the dispatch commands, approval, and when to cut a TestFlight build. RELEASE.md points to it.

Test plan

  • Boot test: fails on main with the error above, and passes on fix(mobile): start over on notification state the previous release saved #1662. It also fails on an injected render-time TypeError in devicesOf.
  • eas-cli 24.8.0 flags checked with --help, then run for real against throwaway channels rollback-smoke and rollback-smoke-dest, never production:
    • eas update --channel rollback-smoke ... published group 7283934e-...;
    • update:republish --group on the same branch, and with --destination-channel;
    • update:roll-back-to-embedded --channel rollback-smoke --runtime-version 87cad17f... --platform ios --non-interactive --json.
  • Ran the two new step scripts with bash, pointed at the smoke channels. Rollback resolved the runtime from the latest production build (build 13, 76f2bd5) and logged the new group. Republish logged the group's details, promoted it to the other channel, and refused x; echo pwned as a group id. eas update:list --branch production still shows the manual rollback group 1ea65e13-... on top.
  • actionlint passes on mobile-release.yml.

Fixes #1657

@janicduplessis janicduplessis left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fresh review of #1663 (issue #1657). Findings, most severe first.

1. High: the boot test times out on the CI runner, so it would block every auto/update/build release.
apps/mobile/src/app-boot.test.tsx:211. The check job of this PR fails: thrown: "Exceeded timeout of 5000 ms for a test." (suite took 15.5 s; run 36312631813). Locally the test takes about 0.7 s, so the likely cause is the first render transforming the whole Expo Router tree on a cold runner. Because mobile-release.yml now runs pnpm test --ci before publishing, a release would fail the same way.
Fix: give the test an explicit timeout, e.g. it('boots ...', async () => { ... }, 60_000), and confirm the check job goes green. If it still times out with 60 s, something stalls under CI, such as timers or the fake socket, rather than transform time.

2. Medium: the drawer stub drops the menu, which production renders at launch, so a crash there passes the gate.
apps/mobile/src/app-boot.test.tsx:49-55. The mock renders only children and drops renderDrawerContent. react-native-drawer-layout 4.2.10 calls renderDrawerContent() on every render, outside any open branch (src/views/Drawer.native.tsx:529), so Menu (src/screens/menu.tsx, which reads useMacs().connections, recents, filters and app update) mounts on every launch. I added throw new TypeError('injected') as the first line of Menu and the boot test still passed. The README ("renders the whole app, from src/app/_layout.tsx down to home") and the PR description overstate the coverage.
Fix: render it in the stub, e.g. Drawer: ({ children, renderDrawerContent }) => <GestureHandlerRootView>{renderDrawerContent()}{children}</GestureHandlerRootView>, then check the injected throw fails the test.

3. Medium: nothing asserts that the live status arrived, and the crash path depends on it.
apps/mobile/src/app-boot.test.tsx:220,228. 'Mock Mac' is the stored pairing name, so it renders with or without a live status, and sent containing status.subscribe only proves the request went out. I removed the status event reply from FakeSocket and the test still passed. Any later change to the handshake or subscription that stops the status from being delivered makes the test silently stop covering the notifier-on-live-status path that crashed in production.
Fix: assert on something only the live run produces. The most direct check is that the notifier rewrote its state: expect(JSON.parse(mockStore('stim.notifications').get('state')!)['status:mac1'].workspaces).toBeDefined(). I checked it: it passes as the test stands, and fails when the status event reply is removed. Alternatively, assert on text that only the new payload renders, such as something tied to a field olderStatus() deletes.

4. Low: a rollback can wait behind a running build or an unapproved run, and the runbook does not say so.
.github/workflows/mobile-release.yml:26-28 (concurrency: group: mobile-release, cancel-in-progress: false). A mode=rollback dispatch queues behind an in-progress run, and a build run can take up to 120 min. A run still waiting for release approval also holds the group until someone approves or rejects it. GitHub keeps only one pending run per group, so a second dispatch while the rollback is pending cancels the pending rollback.
Fix: document it in "Roll back a bad update" (check gh run list --workflow mobile-release.yml and cancel or reject a queued or in-progress run first). Or give rollback and republish their own concurrency group, since they do not race a build.

Checked and fine:

  • Mode selection: on tag push inputs.mode is empty, so the test step's if: is true and the mode step forces build. Rollback and republish still run the on-main check, and the job-level environment: release gates every mode.
  • Inputs reach shell only through env: and are quoted. The group id is validated by regex before use, and runtime is passed as a quoted argument.
  • The test step runs before expo-github-action loads EXPO_TOKEN.
  • eas-cli 24.8.0 flags match --help: update:roll-back-to-embedded --channel --runtime-version --platform --message --json --non-interactive, and update:republish --group --destination-channel --platform --message --json.
  • JSON shapes match the eas-cli 24.8.0 source. update:view --json returns an array with string branch, runtimeVersion, platform and isRollBackToEmbedded. roll-back-to-embedded prints getUpdateJsonInfosForUpdates, so .[0].group is correct.
  • An empty destination array expands safely under Actions' bash -eo pipefail (no -u).
  • README anchors (#each-release, #roll-back-a-bad-update, RELEASE.md step 7 under #4-cut-the-release) resolve. The claim that an update applies on the next launch matches app.config.ts, which leaves fallbackToCacheTimeout at its default of 0.
  • The boot test fails on main without #1662 with the production TypeError, as claimed; I reproduced it.

@janicduplessis janicduplessis left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up review of 60dfd93. All four findings are resolved, and I have nothing further that needs a change.

  1. Timeout: the test now has a 60 s timeout, and the check job passes (run 36313208705).
  2. Drawer stub: it now renders renderDrawerContent() next to children, so Menu mounts the way it does in production.
  3. Live status: the new assertion that status:mac1 in stim.notifications has workspaces is the one I verified earlier. It fails when the status never arrives.
  4. Concurrency: the workflow-level group now reads inputs.mode, which that context allows. On a tag push inputs.mode is empty, so tag builds stay in mobile-release. Rollback and republish no longer wait behind a build. The runbook's note to cancel a publishing run that is in progress or waiting for approval covers the one new risk: a publish finishing after the rollback and undoing it.

All required checks on the new head are green.

@janicduplessis
janicduplessis marked this pull request as ready for review September 27, 2026 10:45
@janicduplessis
janicduplessis merged commit 329b574 into main Sep 27, 2026
9 checks passed
@janicduplessis
janicduplessis deleted the ci/1657-mobile-rollback branch September 27, 2026 10:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

mobile: roll back or republish OTA updates from the release workflow, and gate publishes on a boot test

1 participant