You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fleet coord hosts: follow-ups to the coordinator launching the GPU hosts #1289
Follow-ups from #1282 (the coordinator launches the GPU hosts) and ejc3/aws#676, none of which #1282 relies on.
A lost launch's token is dropped when the fleet sleeps (CodeRabbit on fleet coord: launch the GPU hosts with an instant EC2 Fleet, spot then on demand in one pass #1282, hosts.rs around the power-state change). If a CreateFleet reply is lost and the fleet goes to sleep before a list names the instance, the asleep pass cannot terminate a host it does not see, and the next asleep check comes 300 s later (ASLEEP_CHECK). Bounded at about 5 minutes of one host, and only after a lost reply followed by a sleep in the same few seconds. Fix: keep tokens while asleep, reconcile them to learn the ids, and poll at poll while any token or recent launch remains.
No alert for an on-demand host.fcvm-fleet: the coordinator launches the GPU hosts; remove the host auto scaling groups aws#676 removes the fcvm-fleet-on-demand-hosts alarm with the on-demand group whose metric it read. The coordinator caps on-demand hosts at 8 and logs spot launch refused; launching on demand now; an alert on that log line, or a CloudWatch metric the coordinator emits, would replace it.
From #1282's last Codex round (P2): publish the final running observation before ending the host watch, so a host that finishes after the EC2 half made the fleet awake still gets its last phase in the wake breakdown.
Activity
ejc3 commentedon Oct 26, 2026
From #1282's last Codex round (P2): publish the final running observation before ending the host watch, so a host that finishes after the EC2 half made the fleet awake still gets its last phase in the wake breakdown.