fcvm-fleet: the coordinator launches the GPU hosts; remove the host auto scaling groups by ejc3 · Pull Request #676 · ejc3/aws

ejc3 · 2026-10-25T15:45:59Z

The fcvm fleet coordinator launches the GPU hosts itself (ejc3/fcvm#1282, part of ejc3/fcvm#1257); this removes the host auto scaling groups and gives the coordinator what it needs.

Draft until ejc3/fcvm#1282 is ready to merge. Order: apply this, then merge the fcvm PR. Until the fcvm image is released, the running coordinator sleeps and wakes the fleet (--sleep-when-idle) but launches no GPU host, and the groups are gone. The fcvm image needs no further apply: it reads the hosts from the task environment this sets.

Why

CloudTrail for fcvm-fleet-hosts (us-east-1), the coordinator's SetDesiredCapacity 1 to Auto Scaling's first CreateFleet: 11 s, 7 s, 13 s, 10 s, 11 s and 11 s on the six wakes of 2026-10-24 and 25. Auto Scaling asks one AZ per call and spaces its retries 10 s, 60 s, then 2 and 4 minutes (10-24 21:44: 12 refused calls over 13 minutes, no host; 10-25 13:18: a host after 3 min 10 s). It never moves capacity to the on-demand group itself.

What changed

modules/fcvm-fleet-region: the spot group, the on-demand group, the SNS topic, its email subscription and the fcvm-fleet-on-demand-hosts alarm are removed (the alarm read the on-demand group's metric, which no longer exists), with the host_on_demand_max and alert_email variables. New outputs: host_template_names (empty unless hosts are enabled and an AMI exists, which is the gate the groups' max_size was), host_subnet_ids (the AZs that offer a host type), host_template_arns, host_types. The host launch templates are unchanged.
fcvm-fleet.tf: the task environment carries FLEET_HOST_TEMPLATES, FLEET_HOST_SUBNETS and FLEET_HOST_ON_DEMAND_MAX (8); the command has --sleep-when-idle in place of --scale-group, --on-demand-group and --spot-wait. The current image refuses an unknown argument and the new one has no --scale-group, so the environment is what lets either image start.
Coordinator IAM: ec2:CreateFleet in the fleet regions; RunInstances from the host templates with the host AMIs and the fleet's subnets and group, as the tile statements do; a deny on any instance type outside the four fleet types, and on a tile template launching a host type or the reverse; Purpose must be fcvm-fleet-tile or fcvm-fleet-host at launch and for termination; tags only at RunInstances or CreateFleet; PassRole for the host role as well. The autoscaling statements are gone.
aws_iam_service_linked_role for ec2fleet.amazonaws.com: the fleet account has no EC2 Fleet role (checked 2026-10-25), and the coordinator may not create one.
scripts/test-fcvm-fleet.py and AGENTS.md follow.

Not replaced: the email on an on-demand host in service. The coordinator caps on-demand hosts at 8 and logs each on-demand launch (spot launch refused; launching on demand now); an alert on that log is a follow-up.

Tests

$ python3 -S -B scripts/test-fcvm-fleet.py
Ran 33 tests ... OK

Against main's files the new and changed tests fail (5 failures, 3 errors), e.g. test_the_coordinator_launches_the_hosts_and_no_group_does and test_the_host_templates_reach_the_coordinator_in_its_environment. terraform fmt -check -recursive is clean and terraform validate passes for modules/fcvm-fleet-region; the root is validated by the drift workflow.

Not verified offline: that EC2 evaluates the ec2:LaunchTemplate condition and aws:RequestTag/Purpose on the RunInstances an instant fleet makes for its caller, as it does for a direct RunInstances. If it does not, the coordinator reports scale degraded with EC2's refusal at the first wake and the statements need adjusting.

The coordinator in ejc3/fcvm#1273 keeps its EC2 tiles stopped between wakes (--ec2-pool) and starts them at the next one instead of launching: on g4dn.xlarge from the tile AMI a stopped tile reached its first frame in 15.5 s (persistent spot) against 81 to 84 s for a launch (2026-10-25). g4dn has no hibernation and Auto Scaling warm pools take no spot. - modules/fcvm-fleet-region: launch template fcvm-fleet-tile-pool-<arch>, the spot tile with a persistent request that stops on interruption, its spot request tagged Purpose=fcvm-fleet-tile; in tile_template_arns, so RunInstances may use it. - The coordinator's role: StartInstances/StopInstances and CreateTags/DeleteTags (keys FleetPool and FleetTile only) on instances tagged Purpose=fcvm-fleet-tile, and CancelSpotInstanceRequests on requests so tagged (a persistent request launches a replacement when its instance is terminated, so the coordinator cancels it first). - Coordinator arguments --ec2-pool 8 --ec2-pool-template fcvm-fleet-tile-pool-x86_64 behind local.fcvm_fleet_tile_pool. Standing cost: 8 stopped 32 GB gp3 root volumes, $20.48 a month; no Elastic IPs. Tested: python3 scripts/test-fcvm-fleet.py (39 tests OK, 3 new), terraform fmt. terraform validate was not run here: this box has 1.16.6 and main.tf pins 1.10.3.

…uto scaling groups ejc3/fcvm#1257: Auto Scaling took 7 to 13 s after each SetDesiredCapacity to make its first CreateFleet call (CloudTrail, six staging wakes on 2026-10-24 and 25), asked one AZ per call with retries 10 s to 4 minutes apart, and never moved capacity to the on-demand group itself. The coordinator now launches the hosts from the host templates (instant EC2 Fleet over every host subnet, spot first, on demand in the same pass when spot is refused, at most 8 on demand across the fleet) and terminates them while the fleet sleeps. - modules/fcvm-fleet-region: drop aws_autoscaling_group.hosts and hosts_on_demand, the SNS topic, its subscription and the on-demand alarm (it read the on-demand group's metric), and the host_on_demand_max and alert_email variables. New outputs host_template_names (only while hosts are enabled and an AMI exists), host_subnet_ids, host_template_arns, host_types. - fcvm-fleet.tf: the coordinator gets FLEET_HOST_TEMPLATES, FLEET_HOST_SUBNETS and FLEET_HOST_ON_DEMAND_MAX in its environment and --sleep-when-idle in place of --scale-group, --on-demand-group and --spot-wait. Both the running image and the image with the launcher start with these: the old one ignores the environment, the new one has no --scale-group. - IAM: ec2:CreateFleet in the fleet regions; RunInstances from the host templates with the host AMIs, the fleet's subnets and group; a deny on any type outside the four fleet types and on a template launching another kind's types; Purpose fcvm-fleet-tile or fcvm-fleet-host required at launch and for termination; tags at RunInstances or CreateFleet; PassRole for the host role too. The autoscaling statements are gone. - EC2 Fleet's service-linked role (ec2fleet.amazonaws.com), which the account lacks. - scripts/test-fcvm-fleet.py and AGENTS.md follow. Tested: python3 -S -B scripts/test-fcvm-fleet.py (33 tests OK; the 8 new or changed ones fail against main's files); terraform fmt -check; terraform validate of modules/fcvm-fleet-region.

coderabbitai · 2026-10-25T15:46:07Z

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 39 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: defaults
Review profile: CHILL
Plan: Advanced
Run ID: 9eb5ac3b-de23-45b2-afe9-e987ccc8b08c

Commits

Reviewing files that changed from the base of the PR and between 7124f6e and dd44218.

Files selected for processing: 5

AGENTS.md
fcvm-fleet.tf
modules/fcvm-fleet-region/main.tf
scripts/fcvm-fleet-on-demand-notify.py
scripts/test-fcvm-fleet.py

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

…r-launches-hosts # Conflicts: # fcvm-fleet.tf # modules/fcvm-fleet-region/main.tf # scripts/test-fcvm-fleet.py

chatgpt-codex-connector · 2026-10-25T22:44:31Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-26T03:00:59.448049Z	`dd44218`	Manual request
🔒 Security Review	✅ Completed 2026-10-26T02:59:19.413954Z	`dd44218`	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

…authorization

…lerts # Conflicts: # fcvm-fleet.tf

…w/fcvm-676-iam-alerts # Conflicts: # fcvm-fleet.tf

ejc3 · 2026-10-26T02:48:39Z

Scoped CreateFleet and RunInstances to the fleet resources/types/tags, removed the unnecessary EC2 Fleet service-linked role, and retained the existing confirmed SNS subscriptions with an accepted-launch alarm and narrowly scoped forwarding Lambda. Includes the reviewed pool permission fix and current DNS changes. 29 read-only authorization simulations passed; final combined ARM tests and independent review are running.

ejc3 · 2026-10-26T02:48:41Z

@codex review

ejc3 · 2026-10-26T02:48:42Z

@codex security review

ejc3 · 2026-10-26T02:48:43Z

@coderabbitai review

ejc3 · 2026-10-26T02:48:45Z

@greptileai review

coderabbitai · 2026-10-26T02:49:09Z

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

chatgpt-codex-connector · 2026-10-26T02:51:34Z

Codex Review: Didn't find any major issues. Nice work!

Reviewed commit: d48d018071

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-10-26T02:52:11Z

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: d48d018071

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

You comment "@codex security review"
A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

…lerts # Conflicts: # fcvm-fleet.tf

ejc3 · 2026-10-26T02:55:29Z

Updated against merged #674 and #693. Corrected duplicate pool grant blocks introduced during conflict resolution. At the corrected head, coordinator/fleet/alert source and tests are byte-for-byte identical to the reviewed d48d018 tree; newly incorporated behavior outside those files is the already-reviewed Oregon AMI pin.

ejc3 · 2026-10-26T02:55:30Z

@codex review

ejc3 · 2026-10-26T02:55:32Z

@codex security review

ejc3 · 2026-10-26T02:55:33Z

@coderabbitai review

ejc3 · 2026-10-26T02:55:35Z

@greptileai review

ejc3 · 2026-10-26T02:56:05Z

@codex review

ejc3 · 2026-10-26T02:56:06Z

@codex security review

ejc3 · 2026-10-26T02:56:08Z

@coderabbitai review

ejc3 · 2026-10-26T02:56:09Z

@greptileai review

coderabbitai · 2026-10-26T02:56:14Z

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

coderabbitai · 2026-10-26T02:56:28Z

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

chatgpt-codex-connector · 2026-10-26T02:59:18Z

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: dd44218d1e

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

You comment "@codex security review"
A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

chatgpt-codex-connector · 2026-10-26T03:00:58Z

Codex Review: Didn't find any major issues. Keep them coming!

Reviewed commit: dd44218d1e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

ejc3 · 2026-10-26T04:16:19Z

Live verification found that CreateFleet authorizes the created instance without ec2:InstanceType or aws:RequestTag/Purpose context. FleetTypesOnly therefore explicitly denies the intended host launches; the tag deny and instance Allow have the same unsupported assumptions. The earlier simulations supplied those fields. A targeted correction is being reviewed using the real context, while preserving the strict RunInstances template/type/Purpose and PassRole boundaries.

ejc3 added 2 commits October 25, 2026 14:01

ejc3 mentioned this pull request Oct 25, 2026

fleet coord: launch the GPU hosts with an instant EC2 Fleet, spot then on demand in one pass ejc3/fcvm#1282

Merged

Merge remote-tracking branch 'origin/main' into fcvm-fleet-coordinato…

8dabd3b

…r-launches-hosts # Conflicts: # fcvm-fleet.tf # modules/fcvm-fleet-region/main.tf # scripts/test-fcvm-fleet.py

ejc3 mentioned this pull request Oct 25, 2026

fleet coord hosts: follow-ups to the coordinator launching the GPU hosts ejc3/fcvm#1289

Open

ejc3 marked this pull request as ready for review October 25, 2026 22:44

ejc3 added 5 commits October 26, 2026 02:30

fix: authorize tagged tile spot requests with supported context

721185a

fix: scope fleet launches and preserve on-demand cost alerts

36328ec

Merge remote-tracking branch 'origin/main' into review/fcvm-674-spot-…

10c9c89

…authorization

Merge remote-tracking branch 'origin/main' into review/fcvm-676-iam-a…

454c3fa

…lerts # Conflicts: # fcvm-fleet.tf

Merge remote-tracking branch 'origin/fcvm-fleet-tile-pool' into revie…

d48d018

…w/fcvm-676-iam-alerts # Conflicts: # fcvm-fleet.tf

Merge remote-tracking branch 'origin/main' into review/fcvm-676-iam-a…

81e27b4

…lerts # Conflicts: # fcvm-fleet.tf

Remove duplicate pool grants introduced by merge resolution

dd44218

ejc3 merged commit 102d17a into main Oct 26, 2026
3 checks passed

ejc3 mentioned this pull request Oct 26, 2026

Fix CreateFleet authorization for GPU host launches #694

Merged

Conversation

ejc3 commented Oct 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What changed

Tests

Uh oh!

coderabbitai Bot commented Oct 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review limit reached

Uh oh!

chatgpt-codex-connector Bot commented Oct 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

ejc3 commented Oct 26, 2026

Uh oh!

ejc3 commented Oct 26, 2026

Uh oh!

ejc3 commented Oct 26, 2026

Uh oh!

ejc3 commented Oct 26, 2026

Uh oh!

ejc3 commented Oct 26, 2026

Uh oh!

coderabbitai Bot commented Oct 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot commented Oct 26, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 26, 2026

🛡️ Codex Security Review

Uh oh!

ejc3 commented Oct 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ejc3 commented Oct 26, 2026

Uh oh!

ejc3 commented Oct 26, 2026

Uh oh!

ejc3 commented Oct 26, 2026

Uh oh!

ejc3 commented Oct 26, 2026

Uh oh!

ejc3 commented Oct 26, 2026

Uh oh!

ejc3 commented Oct 26, 2026

Uh oh!

ejc3 commented Oct 26, 2026

Uh oh!

ejc3 commented Oct 26, 2026

Uh oh!

coderabbitai Bot commented Oct 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Oct 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot commented Oct 26, 2026

🛡️ Codex Security Review

Uh oh!

chatgpt-codex-connector Bot commented Oct 26, 2026

Uh oh!

Uh oh!

ejc3 commented Oct 26, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

ejc3 commented Oct 25, 2026 •

edited

Loading

coderabbitai Bot commented Oct 25, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Oct 25, 2026 •

edited

Loading

coderabbitai Bot commented Oct 26, 2026 •

edited

Loading

ejc3 commented Oct 26, 2026 •

edited

Loading

coderabbitai Bot commented Oct 26, 2026 •

edited

Loading

coderabbitai Bot commented Oct 26, 2026 •

edited

Loading