ci(mobile): roll back or republish OTA updates from the release workflow, and gate publishes on a boot test - #1663
Conversation
…low, and gate publishes on a boot test Fixes #1657
janicduplessis
left a comment
There was a problem hiding this comment.
Fresh review of #1663 (issue #1657). Findings, most severe first.
1. High: the boot test times out on the CI runner, so it would block every auto/update/build release.
apps/mobile/src/app-boot.test.tsx:211. The check job of this PR fails: thrown: "Exceeded timeout of 5000 ms for a test." (suite took 15.5 s; run 36312631813). Locally the test takes about 0.7 s, so the likely cause is the first render transforming the whole Expo Router tree on a cold runner. Because mobile-release.yml now runs pnpm test --ci before publishing, a release would fail the same way.
Fix: give the test an explicit timeout, e.g. it('boots ...', async () => { ... }, 60_000), and confirm the check job goes green. If it still times out with 60 s, something stalls under CI, such as timers or the fake socket, rather than transform time.
2. Medium: the drawer stub drops the menu, which production renders at launch, so a crash there passes the gate.
apps/mobile/src/app-boot.test.tsx:49-55. The mock renders only children and drops renderDrawerContent. react-native-drawer-layout 4.2.10 calls renderDrawerContent() on every render, outside any open branch (src/views/Drawer.native.tsx:529), so Menu (src/screens/menu.tsx, which reads useMacs().connections, recents, filters and app update) mounts on every launch. I added throw new TypeError('injected') as the first line of Menu and the boot test still passed. The README ("renders the whole app, from src/app/_layout.tsx down to home") and the PR description overstate the coverage.
Fix: render it in the stub, e.g. Drawer: ({ children, renderDrawerContent }) => <GestureHandlerRootView>{renderDrawerContent()}{children}</GestureHandlerRootView>, then check the injected throw fails the test.
3. Medium: nothing asserts that the live status arrived, and the crash path depends on it.
apps/mobile/src/app-boot.test.tsx:220,228. 'Mock Mac' is the stored pairing name, so it renders with or without a live status, and sent containing status.subscribe only proves the request went out. I removed the status event reply from FakeSocket and the test still passed. Any later change to the handshake or subscription that stops the status from being delivered makes the test silently stop covering the notifier-on-live-status path that crashed in production.
Fix: assert on something only the live run produces. The most direct check is that the notifier rewrote its state: expect(JSON.parse(mockStore('stim.notifications').get('state')!)['status:mac1'].workspaces).toBeDefined(). I checked it: it passes as the test stands, and fails when the status event reply is removed. Alternatively, assert on text that only the new payload renders, such as something tied to a field olderStatus() deletes.
4. Low: a rollback can wait behind a running build or an unapproved run, and the runbook does not say so.
.github/workflows/mobile-release.yml:26-28 (concurrency: group: mobile-release, cancel-in-progress: false). A mode=rollback dispatch queues behind an in-progress run, and a build run can take up to 120 min. A run still waiting for release approval also holds the group until someone approves or rejects it. GitHub keeps only one pending run per group, so a second dispatch while the rollback is pending cancels the pending rollback.
Fix: document it in "Roll back a bad update" (check gh run list --workflow mobile-release.yml and cancel or reject a queued or in-progress run first). Or give rollback and republish their own concurrency group, since they do not race a build.
Checked and fine:
- Mode selection: on tag push
inputs.modeis empty, so the test step'sif:is true and the mode step forcesbuild. Rollback and republish still run the on-main check, and the job-levelenvironment: releasegates every mode. - Inputs reach shell only through
env:and are quoted. The group id is validated by regex before use, andruntimeis passed as a quoted argument. - The test step runs before
expo-github-actionloadsEXPO_TOKEN. - eas-cli 24.8.0 flags match
--help:update:roll-back-to-embedded --channel --runtime-version --platform --message --json --non-interactive, andupdate:republish --group --destination-channel --platform --message --json. - JSON shapes match the eas-cli 24.8.0 source.
update:view --jsonreturns an array with stringbranch,runtimeVersion,platformandisRollBackToEmbedded. roll-back-to-embedded printsgetUpdateJsonInfosForUpdates, so.[0].groupis correct. - An empty
destinationarray expands safely under Actions'bash -eo pipefail(no-u). - README anchors (
#each-release,#roll-back-a-bad-update, RELEASE.md step 7 under#4-cut-the-release) resolve. The claim that an update applies on the next launch matchesapp.config.ts, which leavesfallbackToCacheTimeoutat its default of 0. - The boot test fails on
mainwithout #1662 with the productionTypeError, as claimed; I reproduced it.
…est; queue rollbacks apart
janicduplessis
left a comment
There was a problem hiding this comment.
Follow-up review of 60dfd93. All four findings are resolved, and I have nothing further that needs a change.
- Timeout: the test now has a 60 s timeout, and the
checkjob passes (run 36313208705). - Drawer stub: it now renders
renderDrawerContent()next tochildren, soMenumounts the way it does in production. - Live status: the new assertion that
status:mac1instim.notificationshasworkspacesis the one I verified earlier. It fails when the status never arrives. - Concurrency: the workflow-level group now reads
inputs.mode, which that context allows. On a tag pushinputs.modeis empty, so tag builds stay inmobile-release. Rollback and republish no longer wait behind a build. The runbook's note to cancel a publishing run that is in progress or waiting for approval covers the one new risk: a publish finishing after the rollback and undoing it.
All required checks on the new head are green.
Description
The OTA updates for #1650 and #1655 crashed the TestFlight app at launch.
mode=autohad published them to channelproductionwith no check that the JS boots, and the rollback had to be run by hand outside the workflow. This PR addsrollbackandrepublishmodes tomobile-release.yml, and a Jest boot test that runs before anything is published.Stacked on #1662, which fixes the crash itself. Without it, the boot test fails on
main.Solution
Rollback and republish modes. Both run under the same
releaseenvironment approval and on-main check as the other modes.mode=rollbackrunseas update:roll-back-to-embedded --channel production --runtime-version <rt> --platform ios. The runtime is theruntimeinput, or the latest finished production build's (the samebuild:listqueryautouses).mode=republish -f group=<id>validates the id, logs the group's branch, runtime and message fromeas update:view, and runseas update:republish --group <id> --platform ios. It warns when the group's runtime is not the latest build's. A group from another branch goes toproductionwith--destination-channel production.Neither mode runs the tests, since a rollback has to work while
mainis broken. They use their own concurrency group, so they don't queue behind a 2-hour build. Each writes a::notice::naming the runtime and the group it acted on.Boot gate.
src/app-boot.test.tsxrenders the whole Expo Router tree, drawer menu included, in Jest, starting from the state the previous release stored on the phone. It fails on anything thrown or anyconsole.errorfrom React, and asserts that the notifier processed the live status.mobile.ymlalready runspnpm test.mobile-release.ymlnow runs it too, for every publishing mode, beforeEXPO_TOKENis loaded.stim's cached status, and notification prefs and state in their old shapes. A fake socket then servesmock-server/fixtures/status.json, and the app goesinactive->activeas it does on iOS.@testing-library/react-native@14.0.1and its peertest-renderer. There is also a JestmoduleNameMapperfor@/assets/*.expo-router/testing-libraryinstalls fails to load on 4.7.On
main(without #1662) it reproduces the production crash:Trade-offs (reasoning is in the README runbook):
Runbook. A "Roll back a bad update" section in
apps/mobile/README.mdcovers detection, the dispatch commands, approval, and when to cut a TestFlight build. RELEASE.md points to it.Test plan
mainwith the error above, and passes on fix(mobile): start over on notification state the previous release saved #1662. It also fails on an injected render-timeTypeErrorindevicesOf.--help, then run for real against throwaway channelsrollback-smokeandrollback-smoke-dest, neverproduction:eas update --channel rollback-smoke ...published group7283934e-...;update:republish --groupon the same branch, and with--destination-channel;update:roll-back-to-embedded --channel rollback-smoke --runtime-version 87cad17f... --platform ios --non-interactive --json.76f2bd5) and logged the new group. Republish logged the group's details, promoted it to the other channel, and refusedx; echo pwnedas a group id.eas update:list --branch productionstill shows the manual rollback group1ea65e13-...on top.actionlintpasses onmobile-release.yml.Fixes #1657