Skip to content

fleet: GPU wall wake latency, a measured breakdown per phase and the cuts #1257

Open
Open
fleet: GPU wall wake latency, a measured breakdown per phase and the cuts#1257

Description

@ejc3
ejc3
opened on Oct 25, 2026

The staging wall's wake takes about 2 min 30 s from the coordinator's power change to the GPU half's last address, and 43 to 52 s for each EC2 tile. This issue tracks measuring each phase and cutting the ones that are ours.

The wake of 2026-10-25 11:11:33 UTC (coordinator log, ASG activity, instance consoles)

GPU host (g4dn.metal, spot group fcvm-fleet-hosts):

phase from to took
Auto Scaling reacts to SetDesiredCapacity 0 to 1 11:11:33 desired set 11:11:44 "instance was started in response to a difference" 11 s
launch request to EC2 LaunchTime / activity 11:11:44 11:11:46 LaunchTime, activity 11:11:47 to 11:11:52 2 to 8 s
metal firmware to kernel start 11:11:46 about 11:12:25 (console: cloud-init init-local at 11:12:50, uptime 25.6 s) about 39 s
kernel to fcvm-fleet-store.service start 11:12:25 11:12:57 (uptime 32.5 s) 32 s
store restore (store.sh, 3.2 GB zstd tar from S3 in us-east-2 onto NVMe btrfs) 11:12:57 11:13:50 (cloud-final starts the agent at uptime 86 s) 53 s
agent start to every slot's address in DNS 11:13:50 11:13:54 to 11:13:56 4 to 6 s
DNS confirmation 11:13:56 11:14:02 to 11:14:03 names published 7 s (being removed separately; src/fleet/coord/dns.rs is out of scope here)

EC2 tile (g4dn.xlarge, one-time spot, template fcvm-fleet-tile-x86_64): launched 11:11:34 to 11:11:35, addresses reported 11:12:18 to 11:12:27. One tile's console: kernel start about 8 s after launch, the nvidia module loads at uptime 17.9 s (a 9.7 s gap after udev starts), cloud-init init-local at 25.6 s, the renderer and TLS units start at 33 s. A single relaunch at 11:14:40 reported at 11:15:19 (39 s).

Spot fallback: the coordinator moves a short spot group's capacity to the on-demand group only after --spot-wait 120 seconds, although Auto Scaling records the failed spot launch (UnfulfillableCapacity, MaxSpotInstanceCountExceeded) within about 10 s of the desired-capacity change.

Two gaps in what the coordinator records: the wake record's host phases stop at booted because the power state turns awake once the EC2 half's 8 GPUs are up (about 53 s), before the host's agent says hello, so store_ready_at, agent_at and rendering_at stay empty; and none of the phases reach the coordinator's log.

Plan (each a separate PR)

  1. Record each wake's breakdown in the coordinator log: host phases (instance pending, in service, store ready with the store's own fetch and extract times, agent hello, each slot running with its timings) and each tile's (launch accepted, instance running seen, /health first answer, ready), and keep recording host phases after the power state turns awake.
  2. The 14 s before Auto Scaling launches: it is Auto Scaling's own reconciliation after SetDesiredCapacity; decide whether to keep it (see 3).
  3. Warm pools: Auto Scaling warm pools do not support spot instances, and the host's store is on instance-store NVMe, which a stop wipes. Options and their cost, with one measured trial on staging.
  4. Fall back to the on-demand group as soon as Auto Scaling reports a failed spot launch for the wake, and log which path each wake took.
  5. Other waits in the wake path: tile readiness polls every 3 s; page reconnect backoff in fleet/web; agent dial backoff.

Activity

  1. ejc3
    added 2 commits that reference this issue on Oct 25, 2026

    fleet coord: cancel only persistent spot requests; age pooled tiles o…

    c4e6083

    fleet coord: a resume starts a pooled tile only if it is stopped and …

    368c93f

  2. ejc3 commented on Oct 25, 2026

    @ejc3
    ejc3
    on Oct 25, 2026
    OwnerAuthor

    Measurements and PRs so far. Instance types and timings only; every number is from the staging wake of 2026-10-25 11:11:33 UTC (coordinator log, ASG activities, consoles) or from instances launched for this (a g4dn.metal gpubox and g4dn.xlarge tiles from the current AMIs, us-east-1).

    GPU host, before and after

    phase 2026-10-25 wake with these PRs how
    SetDesiredCapacity to Auto Scaling starting a launch 11 s (also 6 s and 10 s on earlier wakes) same Auto Scaling's own reconciliation; the coordinator calls SetDesiredCapacity the moment the power state changes
    launch request to EC2 LaunchTime 2 to 3 s same
    metal firmware, LaunchTime to kernel start 39 s (gpubox: 39 s cold, 39 s from stopped) same
    kernel to fcvm-fleet-store start 32 s (gpubox 31 s) same cold; 11 s from a stopped instance first-boot cloud-init and lazily read EBS
    store restore 53 s (gpubox 51.8 s) 29.2 s #1270, measured end to end on the gpubox
    agent start to addresses 4 to 6 s same
    DNS confirmation 7 s being removed separately
    spot shortage never fell back (2026-10-24 21:44: 12 failed spot launches over 13 min, no on-demand activity) on-demand raised at the first poll after the failed activity (about 12 s into the wake) #1263, ejc3/aws#670

    EC2 tile, before and after

    2026-10-25 wake with these PRs
    launch to ready 43 to 52 s started from the pool: 15.5 s (persistent spot) or 17.2 to 17.5 s (on demand) to a first frame, against 81 to 84 s for a launch by the same measure (#1273, ejc3/aws#674)
    ready seen by the coordinator up to 3 s after the renderer is ready within 250 ms (#1269)

    PRs

    Not done, and why

    • Auto Scaling warm pool for the GPU hosts: warm pools take no spot instances, so it could only sit on the on-demand group, which inverts spot first. A stopped g4dn.metal also keeps the 39 s firmware and the store restore (the NVMe is wiped by a stop), so it saved 20 s (kernel to store 31 s to 11 s). Its stop took 20.5 minutes and its first StartInstances failed with InsufficientInstanceCapacity (the second, 30 s later, worked). Left for an owner decision.
    • The 11 s of Auto Scaling reconciliation: removing it means the coordinator launching hosts itself (like the tiles) instead of through the group. Left for an owner decision.
    • The staging trial of the tile pool waits for fleet coord: a pool of stopped EC2 tiles that the next wake starts instead of launching #1273 to merge and fcvm-fleet: a pool of stopped EC2 tiles the coordinator starts at a wake aws#674 to be applied.
  3. ejc3
    added 3 commits that reference this issue on Oct 25, 2026

    fcvm-fleet: fill the GPU host's root volume at 300 MiB/s at launch

    8e38a6e

    fcvm-fleet: fill the GPU host's root volume at 300 MiB/s at launch (#679

    cabe4d3

    fcvm-fleet: the coordinator launches the GPU hosts; remove the host a…

    102d17a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions