Skip to content

Prevent run_cvd abort and keep monitor socket alive on graceful VM sh… - #3154

Open
SuperStrongDinosaur wants to merge 1 commit into
google:mainfrom
SuperStrongDinosaur:FixCuttlefishLauncherCrashOnGracefulGuestVMShutdown2
Open

Prevent run_cvd abort and keep monitor socket alive on graceful VM sh…#3154
SuperStrongDinosaur wants to merge 1 commit into
google:mainfrom
SuperStrongDinosaur:FixCuttlefishLauncherCrashOnGracefulGuestVMShutdown2

Conversation

@SuperStrongDinosaur

@SuperStrongDinosaur SuperStrongDinosaur commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator

Problem

When a Cuttlefish guest VM gracefully powers off, crosvm terminates cleanly, which in turn leads the ProcessMonitor subprocess to stop all monitored processes and exit with code 0.

Previously, run_cvd treated any readability on the monitor status pipe as an not normal failure. This caused run_cvd to immediately abort and abandon launcher_monitor.sock. As a result:
1. The VM entered an Unreachable state .
2. Subsequent cvd restart, cvd stop, or cvd powerwash commands failed with Connection refused.

Solution

  1. Keep run_cvd Server Loop Alive on Exit:
    - Track process_monitor_active. When the monitor exit pipe is triggered in Select(), mark process_monitor_active = false so subsequent iterations only poll server_ .
    - Immediately invoke process_monitor.StopMonitoredProcesses() to wait for and clean up the terminated monitor child process, preventing lingering <defunct> zombie processes.
    - Keep the server loop running to service subsequent launcher actions (cvd restart, cvd stop, cvd powerwash, and cvd status).

  2. Harden StopMonitoredProcesses():
    - Make StopMonitoredProcesses() safe to call multiple times and handle cases where the monitor has already exited or finished.
    - Eliminate mid-shutdown race conditions: if SendEmptyRequest encounters a closed/shut down socket because the monitor child is already in its exit path, do not abort early. Instead, proceed to wait for the process to exit using waitpid(last_monitor, &wstatus, 0) and log abnormal exits as warnings rather than failing the stop/restart request.

• Verified lifecycle workflows:
• Graceful guest VM shutdown via VePSM / adb reboot -p leaves launcher_monitor.sock active and eliminates zombie processes.
• cvd restart successfully connects, reaps state, and re-executes run_cvd.
• cvd stop cleanly terminates run_cvd with exit code 0.
• cvd status and cvd fleet successfully respond without connection errors.

Bug: b/534717429

@SuperStrongDinosaur
SuperStrongDinosaur force-pushed the FixCuttlefishLauncherCrashOnGracefulGuestVMShutdown2 branch 2 times, most recently from 2fb0464 to e6e2de6 Compare September 9, 2026 10:28
…utdown

When a guest VM shuts down gracefully (e.g., via VePSM or adb reboot -p),
the guest VMM and process monitor exit cleanly. Previously, run_cvd's
server loop treated the readable monitor status pipe as an abnormal exit
and aborted via CF_ERR("process monitor has died"), deleting or abandoning
launcher_monitor.sock and rendering the device unreachable.

This change:
1. Keeps run_cvd's server loop alive when the process monitor exits,
   clearing process_monitor_active so subsequent select() calls only
   listen on the server monitor socket for commands (e.g., restart/stop/status).
2. Immediately reaps the monitor process upon receiving the exit signal to
   avoid leaving zombie (<defunct>) processes in the process table.
3. Updates ProcessMonitor::StopMonitoredProcesses() to safely handle already-
   exited or mid-shutdown monitors, reaping the child process even if the
   control socket was closed during exit and avoiding spurious errors.

Bug: b/534717429
Test: bazel test //cuttlefish/host/commands/run_cvd:server_loop_format_test_C++_with_clang-format //cuttlefish/host/libs/process_monitor:process_monitor_format_test_C++_with_clang-format
Test: bazel build //cuttlefish/host/commands/run_cvd:run_cvd //cuttlefish/host/libs/process_monitor:process_monitor
TAG=agy
CONV=43370940-200d-4deb-b38e-a3f572a2e586
@SuperStrongDinosaur
SuperStrongDinosaur force-pushed the FixCuttlefishLauncherCrashOnGracefulGuestVMShutdown2 branch from e6e2de6 to 5a46719 Compare September 10, 2026 09:21
@SuperStrongDinosaur SuperStrongDinosaur added the kokoro:run Run e2e tests. label Sep 10, 2026
@GoogleCuttlefishTesterBot GoogleCuttlefishTesterBot removed the kokoro:run Run e2e tests. label Sep 10, 2026
@SuperStrongDinosaur
SuperStrongDinosaur marked this pull request as ready for review September 10, 2026 15:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants