Skip to content
63 changes: 63 additions & 0 deletions docs/BUILD_INCIDENT_2026-10.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,3 +62,66 @@ reported error. Build failures, nonzero driver exits, test failures and timeouts
after the driver begins remain failures. Recovery does not rebuild the APP or
retry failed assertions. The exact underlying failure in the original silent
launch remains unknown; new verbose attempt evidence makes recurrence diagnosable.

## 11 October: retained evidence identifies missed VM-service discovery

The complete PR verification passed, but the subsequent [main run
38142017784](https://github.com/getbible/app/actions/runs/38142017784) failed both
iOS runtime jobs. All seven package builds and the other platform checks passed.
The new watchdog retained evidence and stopped each unsuccessful launch attempt
after three minutes; it did not resolve the underlying connection problem.

On the first attempt, both apps actually started. The iPhone unified log records
the Dart VM service at 13:19:29 UTC; the iPad records it at 13:18:46 UTC. Flutter
continued waiting for that service in both cases and never invoked the host test
driver. The second attempts stalled inside `simctl launch`. No application test
assertion ran or failed in these jobs. Increasing the discovery timeout alone
does not address a service announcement missed by the live log listener.

Flutter 3.44.6 provides `drive --use-existing-app=<URL>` to attach directly to an
already-running application. The launcher now installs and launches the built
APP paused, obtains its process ID, and discovers the authenticated service URL
from retained unified logs scoped to that process, executable and launch time.
It uses the supported explicit connection path to run the same six acceptance
tests. Install, launch and discovery share one three-minute startup budget;
driver attachment and test failures never enter the reboot path. The published
alpha 5 assets remain unchanged.

Native verification of this connection path succeeded on iPhone on its first
attempt: retained-log discovery took 9.29 seconds and all six tests passed. It
also exposed a separate post-exit cleanup error: macOS rejected a process-group
signal after the driver had exited successfully. The existing-app command now
explicitly requests Flutter's application/device cleanup. A denied group signal
is recorded as a warning only when the command has already exited; it cannot
replace a completed command's result. Denied signals for running commands remain
errors, and the final simulator shutdown is still attempted.

The same [verification run
38153311245](https://github.com/getbible/app/actions/runs/38153311245) retained a
different iPad failure: installation and boot completed, but both `simctl launch`
attempts timed out without a process ID, Runner log or VM-service announcement.
The simulator screenshots show an unlocked home screen and the installed app.
This establishes a failure before application execution; it does not identify
the underlying CoreSimulator cause.

Review also found that the runner sent TERM/KILL to every command's process
group after normal exit, including simulator discovery, boot and installation.
Normal command completion does not authorize terminating persistent platform
services initialized by that command. The runner now reaps naturally completed
commands without signalling their groups, regardless of exit code. Bounded
group termination still runs after timeouts or interruptions. Regression tests
cover both natural completion and aborted-command cleanup. Native CI is needed
to determine whether this lifecycle correction also resolves the iPad launch
stall; the recorded iPad evidence alone cannot prove that causal connection.

The next [verification run
38154773547](https://github.com/getbible/app/actions/runs/38154773547) passed iPad
acceptance on its first attempt, including all six tests and shutdown. The iPhone
launched PID 9094 and announced its VM service at 16:25:36.991 UTC. Its first
retained-log query was then aborted by an independent 30-second query limit,
while about 90 seconds remained in the shared startup budget. That premature
timeout triggered a reboot; the second launch stalled before application startup.
Log discovery now uses the remaining shared budget, just like install and
launch. The overall three-minute startup bound is unchanged. Regressions cover
a log query taking more than 30 seconds, shared-deadline exhaustion and the
shorter overall session deadline.
38 changes: 30 additions & 8 deletions docs/TESTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -440,20 +440,42 @@ window/activity/power state, all-buffer logcat and a native screenshot. Syntheti
widget focus alone does not establish Android clipboard permission.

The iOS acceptance launcher builds the integration APP once and reuses it for
bounded launch attempts. It permits one simulator reboot only for an observed
install/launch/VM-service stall before the host test driver starts; reported
errors, failed tests and post-start timeouts are never retried. Build, boot,
attachment and test phases have separate limits inside a 38-minute session,
followed by a bounded cleanup reserve. Attempt logs go directly to disk, with
flushed stage events and simulator diagnostics. This prevents a silent launch
from consuming the whole 50-minute job and losing its evidence.
bounded launch attempts. It installs and launches the APP paused through
`simctl`, obtains the launched process ID, and discovers its authenticated Dart
VM-service URL from retained unified logs. Discovery checks the current process
and launch time; an old application instance cannot satisfy it. The URL must be
local to the simulator host. `flutter drive --use-existing-app` then connects
the same host driver and resumes the paused integration journey. VM-service
authentication remains enabled. This avoids depending on a live log listener
being ready when the app announces its service.
Log reads share the remaining install/launch/discovery budget; a separate short
query timeout must not reboot a running app while that startup budget remains.

The existing-app driver explicitly uses `--no-keep-app-running`, so Flutter
stops the application and disposes its device log readers after the journey.
Naturally completed commands are reaped without signalling their process
groups: a completed `simctl` command can leave platform services running that
later stages need. Group termination is reserved for interrupted or timed-out
commands. If a command exits during that termination and macOS refuses a group
signal, the runner records the warning without replacing the original failure.
A denied signal while the command is still running remains fatal. Simulator
shutdown still runs within the separate cleanup reserve.

One simulator reboot is permitted only for a bounded startup timeout before
driver attachment. Reported command errors, driver connection failures, failed
tests and post-start timeouts are never retried. Build, boot, attachment and test
phases have separate limits inside a 38-minute session, followed by a bounded
cleanup reserve. Attempt logs go directly to disk, with flushed stage events
and simulator diagnostics. This prevents a silent launch from consuming the
whole 50-minute job and losing its evidence.
Initial device discovery receives the same three-minute startup allowance as
boot: its first `simctl` invocation also initializes CoreSimulator on a cold host.
`scripts/tests/test_ios_simulator.py` covers timeout boundaries, child-process
termination, partial output, split/large log markers, one-time recovery and
failure preservation. These Python checks complement actual iPhone/iPad CI
execution; they do not establish native runtime acceptance by themselves. See
the [alpha 5 incident](BUILD_INCIDENT_2026-10.md#10-october-ipad-runtime-timeout-blocked-alpha-5-publication).
the [alpha 5 incident](BUILD_INCIDENT_2026-10.md#10-october-ipad-runtime-timeout-blocked-alpha-5-publication)
and the [post-merge discovery evidence](BUILD_INCIDENT_2026-10.md#11-october-retained-evidence-identifies-missed-vm-service-discovery).

The production Web build uses bundled rendering resources, generated static
shell inventory and Chromium, Firefox and WebKit runtime journeys under plain
Expand Down
Loading
Loading