AWS development infrastructure

This is the live Terraform control plane for the ejc3 development fleet. It is not a generic module or a tutorial environment: the defaults describe the deployed system.

It manages fleet infrastructure and account-wide controls across the main AWS account, the isolated staging/recovery account, Cloudflare, and several regions. The platform provides:

persistent ARM64 and x86 bare-metal Firecracker development servers;
two independent Terraform administration jumpboxes;
authenticated, permanent URLs for the kids' Next.js projects;
ephemeral high-speed shared storage and on-demand 96/192-core compute;
autoscaled ARM64 and x86 GitHub Actions runners;
backups, cross-region/cross-account recovery, and cost alerts;
credential-free Terraform validation in GitHub, with live plans confined to admin hosts;
scoped IAM capabilities for agents, including temporary EBS volumes and Bedrock access (Claude models, plus two named DeepSeek models for opencode on the metal boxes, which the setup installs and points at the instance role: opencode there needs no key).

Start here

The supported control plane is the original jumpbox, with jumpbox-2 as the independent fallback. Terraform runs directly there. There is no container wrapper and no Makefile.

The operating rules are:

Make permanent AWS and Cloudflare changes in Terraform. Do not mutate managed infrastructure with the AWS CLI.
Run terraform plan immediately before terraform apply and read the plan. A plan has previously caught a change that would have exposed every kids' URL publicly.
Commit and push every applied change. Git, Terraform state, and live infrastructure should describe the same system.
Do not run concurrent plans or applies. State uses an S3 backend with DynamoDB locking, but a fresh plan is still required after another operator changes anything.
Treat every instance-store NVMe disk as disposable. Persistent data belongs on EBS or in a backed-up repository.

Normal operation from an admin box:

cd ~/aws
git pull --ff-only
terraform init
terraform fmt -recursive
terraform validate
terraform providers
terraform plan
terraform apply
git add <changed-files>
git commit -m "<what changed>"
git push origin main

The jumpboxes already have Terraform, Git, the AWS CLI, this repository, and an EC2 administrator instance role. There is no AWS login or credential-refresh step. The committed .terraform.lock.hcl keeps provider resolution identical across admin hosts; review and commit any intentional provider upgrade.

main.tf enforces Terraform 1.10.3, which is the version on both jumpboxes and supports the ephemeral Workers Builds control-token read. Any Terraform upgrade must update the constraint, both jumpboxes, and the Terraform validation workflow together.

Cloudflare is intentionally pinned to the signed ejc3/cloudflare 5.24.0 provider release. That fork adds typed Worker-native Access destinations and the Workers Builds APIs used below while preserving ordinary terraform init on both ARM64 jumpboxes and amd64 CI; there is no local provider override or machine-specific installation step. Return to cloudflare/cloudflare only after an upstream release includes the same fields and regression coverage.

The fork has a different provider source address. On the first live-jumpbox update to the fork, stop after terraform providers if state still lists cloudflare/cloudflare. Under the exclusive state lock, migrate all existing Cloudflare resources before planning:

terraform state replace-provider \
  registry.terraform.io/cloudflare/cloudflare \
  registry.terraform.io/ejc3/cloudflare

Terraform writes a mandatory state backup. Do not repeat the command once state lists ejc3/cloudflare, and never apply a plan made before the migration. The reverse command is required when the stack eventually returns to the upstream provider namespace.

Prerequisites

Existing deployment

To operate the existing system, a new person or agent needs:

SSH access to either jumpbox as ubuntu;
the private fcvm-ec2 SSH key;
access to the ejc3/aws GitHub repository;
a clean, current main checkout at ~/aws;
exclusive use of the Terraform state lock for the duration of plan/apply.

Run terraform output for the current addresses and ready-to-use SSH commands. The fcvm-ec2 private key is backed up in Secrets Manager for jumpbox disaster recovery; AWS itself stores only the public half of an EC2 key pair.

Replacement admin host and empty-state recovery

This repository is the control plane for one existing account, not a portable new-account installer. A replacement admin host should use the existing S3 state. It additionally needs:

Terraform 1.10.3 to match both jumpboxes, plus Git and AWS CLI v2;
AWS administrator credentials in the Organizations management account;
access to the existing S3 backend bucket ejc3-terraform-state and DynamoDB lock table ejc3-terraform-locks;
the existing fcvm-ec2 EC2 key pair and its private key;
the existing AWSBackupDefaultServiceRole;
the Cloudflare cc-games.dev zone and Secrets Manager values cloudflare-tunnel-token (including Workers Scripts Read/Write), cloudflare-tunnel-credentials, and cloudflare-google-idp;
the raw user-owned Cloudflare token in cloudflare-workers-builds-control-token when the checked-in Workers Builds rollout gate is enabled. From Create Additional Tokens, choose Use template, retain User → API Tokens → Edit, and add Workers Builds Configuration Edit plus Workers Scripts Read for account 12ea67fb7ced068de03f35c22688e436; an account-owned token is rejected;
the Cloudflare Workers and Pages GitHub App authorized for only CoderColton/colton-games by that repository's owner, starting from the target Worker's Settings → Builds → Connect → GitHub flow. ejc3 has write, not admin, and cannot grant the App;
the github-pat-ejc3 Secrets Manager value, the runner-only GitHub PAT, and github-webhook-admin-pat in Secrets Manager -- a fine-grained PAT whose only repository permission is Webhooks: Read and write on ejc3/fcvm, used by the integrations/github provider to own the runner webhook;
for games multiplayer, the games/colton-games-read Secrets Manager value: a fine-grained token that Colton creates as CoderColton, limited to colton-games with Contents read-only. The plan-time preflight reads it, so create the empty secret with a targeted apply first; the exact steps are under First time only in docs/games-multiplayer.md;
current ARM64 and x86 runner AMIs tagged Purpose=github-runner;
the Google OAuth callback https://ejc3.cloudflareaccess.com/cdn-cgi/access/callback when Google login is enabled.

Clone and initialize the configuration:

git clone https://github.com/ejc3/aws.git ~/aws
cd ~/aws
terraform init

Follow the normal provider-migration gate above before planning from a replacement host.

terraform plan should then refresh the recovered remote state and propose no unexplained re-creation. Stop if the backend appears empty. The configuration contains imported resources, fixed account identities, and protected volumes; applying it against an empty state or a different account is not a supported bootstrap and can collide with or replace live infrastructure. Recover the backend state first, or inventory and import every pre-existing resource before any full apply.

The alert address, runner PAT, SSH-key backup and the app-runner controller tokens use Terraform-managed containers whose payloads are intentionally kept out of Terraform state. For a true cold start, create those containers only after state/import reconciliation is complete. enable_runner_app_webhooks=false keeps this apply from reading the controller tokens, which do not exist yet:

terraform apply -var enable_runner_app_webhooks=false \
  -target=aws_ssm_parameter.alert_email \
  -target='aws_ssm_parameter.github_runner_pat[0]' \
  -target=aws_secretsmanager_secret.fcvm_ec2_ssh_key \
  -target='aws_secretsmanager_secret.github_runner_repo_pat["CoderColton/colton-games"]' \
  -target='aws_secretsmanager_secret.github_runner_repo_pat["dolphin-labs-hq/dolphin-labs"]' \
  -target='aws_secretsmanager_secret.github_runner_repo_pat["dolphin-labs-hq/dolphin-films"]' \
  -target='aws_secretsmanager_secret.github_runner_repo_pat["dolphin-labs-hq/dolphin-maps"]' \
  -target='aws_secretsmanager_secret.github_runner_repo_pat["dolphin-labs-hq/dolphin-family"]' \
  -target='aws_secretsmanager_secret.github_runner_repo_pat["ejc3/imagine"]' \
  -target=aws_secretsmanager_secret.elevenlabs_api_key

Then populate /alerts/email, /github-runner/pat, and fcvm-ec2-ssh-key through the AWS console or AWS CLI without printing their values.

The runners for CoderColton/colton-games, dolphin-labs-hq/dolphin-labs, dolphin-labs-hq/dolphin-films, dolphin-labs-hq/dolphin-maps and ejc3/imagine also need one controller token each, in Secrets Manager github-runner/repo-pat/<owner>/<repo> (the containers created above, from runner-repos.tf; Terraform never holds the value). The full apply reads them all to create the repos' webhooks, so populate them before it. That holds for a repo added later too: apply only its container (the same command with that one -target; nothing untargeted changes, so the existing hooks stay), put its token, then run the full apply. Each is a fine-grained token limited to that one repo with Administration and Webhooks read-write and Actions read-only, minted by the repo's owner: CoderColton for colton-games (ejc3 has write, not admin, there), ejc3 as org admin for dolphin-labs, dolphin-films, dolphin-maps and dolphin-family, ejc3 for imagine (Administration read-write and Actions read-only only: its hook is made by the administration GitHub App). dolphin-films, dolphin-maps, dolphin-family and imagine are each behind their own gate in runner-app.tf, dolphin_films_token_ready, dolphin_maps_token_ready, dolphin_family_token_ready and imagine_token_ready (each starts false): until the token has a value, that repo's hook, token read and controller entry are not planned, so every plan passes and the controller does not report it as missing a token. Put the token, then set that default to true in a commit and apply. Put each value without echoing it:

read -rs T; printf %s "$T" | aws secretsmanager put-secret-value --region us-west-1 \
  --secret-id github-runner/repo-pat/<owner>/<repo> --secret-string file:///dev/stdin; unset T

People's email addresses are not in this repository. people/addresses (us-west-1, from people.tf) holds them as JSON: owner (the browser manager's owner), family (the Cloudflare Access allowlist for every *.cc-games.dev host) and staging_account (the dev-staging account's root email), dolphin_account (the Dolphin account's root email), astryx_account (the astryx account's root email) and, optionally, dolphin_steve (Steve's Identity Center sign-in to the Dolphin account). To add someone to the allowlist, put the whole JSON again with the new address in family (a plan on a jumpbox then shows the Access policies changing; apply it). Terraform fails a plan, naming the key, if the secret is missing a key or family is empty. The optional claude_master_dashboard list names who, besides the owner, may open the claw-master dashboard: the same apply changes its Access policy and the allowlist the Worker itself checks (a Workers KV value Terraform writes), so nothing is set on the Worker by hand. Put it without echoing:

read -rs T; printf %s "$T" | aws secretsmanager put-secret-value --region us-west-1 \
  --secret-id people/addresses --secret-string file:///dev/stdin; unset T

The ElevenLabs key for the games, games/elevenlabs-api-key (us-west-1, from dev-ai-services.tf), is populated the same way. Until it has a value the dev boxes get ResourceNotFoundException for it, and the colton-games Vercel env ELEVENLABS_API_KEY, which Terraform reads from it, cannot be planned, so populate it before the full apply:

read -rs T; printf %s "$T" | aws secretsmanager put-secret-value --region us-west-1 \
  --secret-id games/elevenlabs-api-key --secret-string file:///dev/stdin; unset T

The claw-master backup API key, claude-master/backup-api-key (us-west-1, from dev-ai-services.tf), is the paid last resort after every subscription profile is out of quota. Terraform creates only the container, so until it has a value get-secret-value returns ResourceNotFoundException and claw-master has no fallback. The server checks every 5 minutes that the key can pay and uses it only while it can (the server log says API-key backup unavailable when its organization is out of credit; adding credit brings it back at the next check, no restart). Populate it from the key file (read from the file, never on the command line), then remove the file:

aws secretsmanager put-secret-value --region us-west-1 --secret-id claude-master/backup-api-key \
  --secret-string file://$HOME/claude_api.txt && shred -u ~/claude_api.txt

The claw-master account names, claude-master/account-labels (us-west-1, from claw-master-server.tf), turn the acct-<8 hex> keys in the metrics into names: one ACCOUNT_UUID=NAME line per account (UUID = oauthAccount.accountUuid in that user's ~/.claude.json; claw-master account-key UUID shows which key it replaces). Terraform creates only the container. Set it from a file and nothing else: a timer on the server (claude-master-account-labels.timer) copies a changed secret into the server's labels file within 10 minutes, and claw-master reloads that file within a minute, with no restart:

aws secretsmanager put-secret-value --region us-west-1 --secret-id claude-master/account-labels \
  --secret-string file://labels.txt && shred -u labels.txt

The server log says account labels reloaded with the number of names. A value with a line that is not ACCOUNT_UUID=NAME is not copied, and the names already in use stay. The next Terraform apply also derives the claw-master dashboard's account aliases from it (acct-<8 hex>=NAME, the Workers KV key account-aliases, claw-master-dashboard-site.tf), so the dashboard folds an account's older, unnamed points into the same name.

The dolphin-films credentials (dolphin-films.tf) are four JSON secrets in us-west-1, one per environment and kind. Terraform never reads them: an administrator loads production's into the Cloudflare Worker dolphin-films-prod with wrangler secret bulk. Production and non-production get different values throughout: a Google OAuth client each, an AUTH_SECRET each, a Turso database each with its own token.

Secret	JSON keys	Readers besides administration
`dolphin-films/prod/auth`	`AUTH_SECRET`, `AUTH_GOOGLE_ID`, `AUTH_GOOGLE_SECRET`	none
`dolphin-films/prod/turso`	`TURSO_DATABASE_URL`, `TURSO_AUTH_TOKEN`	none
`dolphin-films/nonprod/auth`	as `prod/auth`	dev-server-role (the metal boxes)
`dolphin-films/nonprod/turso`	as `prod/turso`	dev-server-role

The site's data is in Turso (libSQL), one database per environment. Two kinds of credential are involved, and the site only ever gets the second:

The platform token (turso/api-token, readable by administration and the metal dev boxes) belongs to the account. It creates and deletes databases and mints their tokens. It is never one of the site's variables.
The per-database values, TURSO_DATABASE_URL and TURSO_AUTH_TOKEN: one database's address and a token for that database alone. These are what the two turso secrets hold.

The databases are made once, with the platform token, from a metal dev box, which writes each pair to a private file. An administrator puts them: a dev box cannot write a container, and cannot read production's. The site applies its own migrations when it deploys, so nothing here runs one.

Write each as a JSON file readable only by you, put it, and remove the file:

aws secretsmanager put-secret-value --region us-west-1 --secret-id dolphin-films/<env>/<kind> \
  --secret-string file://<a 0600 JSON file> && shred -u <that file>

The site is served by dolphin-films-prod on Cloudflare (yourfantasymovie.tf). Its Vercel project was in the dolphin-labs team, which is retired (Steve ends the team: dolphin-labs-hq/dolphin-labs#689), and Terraform no longer writes to it (ejc3/aws #503 forgot what it wrote and destroyed the vercel-api-token-dolphin-labs secret). FILMS_SEED_EMAILS and FILMS_ADMIN_EMAILS, the lists of addresses the site reads, come from the dolphin_films key of the people/addresses secret ({"prod": {"seeds": [...], "admins": [...]}, "nonprod": {...}}) and are loaded into the Worker with its other secrets. No address belongs in this repository.

This is a one-time secret-payload bootstrap, not a parallel way to manage infrastructure. The alert sender address must also be verified in SES in us-west-1.

Do not create the Workers Builds control-token container in this generic bootstrap. Its first creation is deliberately deferred until after the 15-minute S3 versioning wait in the staged rollout below; that Terraform write is the backend VersionId probe.

Terraform owns the ejc3/fcvm workflow_job webhook and generates its HMAC secret, so there is nothing to mirror by hand. Before the first full apply, if the recovered state does not already contain the hook, adopt it -- otherwise the apply adds a second hook delivering to the same URL and every event arrives twice:

terraform import 'github_repository_webhook.runner[0]' fcvm/589197362

Only then, with state, prerequisites, secret payloads, and the hook import reconciled, run and review the full plan:

terraform plan
terraform apply

Then confirm the SNS email subscription. Do not leave placeholder secret/parameter values in a live account. Optional Mac development needs available EC2 Dedicated Host quota and an explicit new mac_teardown_at at least 24 hours after allocation; its checked-in date is historical and must not be reused. Terraform creates its VNC password.

Fleet

Resource	Region / purchase	Persistent state	Purpose and lifecycle
`jumpbox`	`us-west-1`, on-demand `t4g.large`	40 GB protected `/home/ubuntu` EBS; separate 40 GB root	Primary admin host. Runs Terraform with an administrator instance role and stays up.
`jumpbox-2`	`us-west-1`, on-demand `t4g.micro`	Protected 20 GB encrypted root	Fully Terraform-bootstrapable backup admin host with the same IAM reach.
`fcvm-metal-arm`	`us-west-1`, persistent Spot `r8gd.metal-24xl`	800 GB backed-up EBS root, including `/home/ubuntu`; local NVMe is ephemeral	64-vCPU ARM64 Firecracker/KVM and nested-virtualization work. Uses the 12-hour idle-stop policy.
`fcvm-metal-x86`	`us-west-1`, persistent Spot `c5d.metal`	300 GB EBS root; 3.6 TB local NVMe is ephemeral	x86 Firecracker/KVM work. Uses the 12-hour idle-stop policy.
`nextjs-dev`	`us-west-1`, on-demand `t4g.xlarge`	400 GB encrypted EBS root (including Dolphin's `/home/ejc3`), protected by AWS Backup	Always-on shared development box. Deliberately not Spot and not idle-stopped.
`io-box`	`us-west-2d`, persistent Spot `i8ge.large`	20 GB EBS root; 1.25 TB shared NVMe is ephemeral	Private NFS bulk scratch at `/mnt/io`. Uses a 12-hour multi-metric idle policy and returns with an empty scratch disk after every stop.
`parallel-box`, `parallel-box-2` .. `parallel-box-8`	`us-east-2c` (`/infra/parallel-box`), one-time Spot, 32 to 192 vCPU by the flavor `pbox up` must name (`pbox-flavors.tf`)	Protected EBS each (300 GB for `parallel-box`, 100 GB for `parallel-box-2` .. `parallel-box-8`) at `/mnt/work`; roots are disposable	Temporary fan-out compute, eight independent boxes so eight jobs can run at once; the us-east-2 spot quota (3,072 vCPU, `ohio-pbox.tf`) fits all eight at 192 vCPU beside the runners and members' boxes. Each terminates after 30 idle minutes; `pbox` recreates them. When the home AZ has no capacity, `pbox up N FLAVOR --fallback` (or `--regions r1,r2`) launches in another US AZ (`pbox-fallback.tf`: us-east-1, us-east-2, us-west-1, us-west-2) with a FRESH, empty `/mnt/work` deleted with the box and no `/srv/io`.
`gpu-box`, `gpu-box-2` .. `gpu-box-4`	`us-west-2` (any AZ), on-demand `g4dn.xlarge` or the next small NVIDIA size	None; disposable 100 GB root	Browser-game performance tests on a real GPU (dev boxes render WebGL in software). Four independent slots, launched from `nextjs-dev` with `gbox up [1-4]`; each terminates after 30 idle minutes or 4 hours, whichever comes first.
`wbox-<member>`, `pbox-<member>`, `gbox-<member>`	per member (`member-boxes.tf`: colton, connor, colin, skevh): one-time Spot `g4dn.xlarge` Windows in `us-west-1a`; one-time Spot Graviton in `us-east-2c`; on-demand small NVIDIA in `us-west-2a`	Protected 50 GB `D:` per wbox (Roblox and its caches) and 100 GB `/mnt/work` per pbox; the rest is disposable	Each member's own three boxes, each with a fixed private IP. The member runs the same `wbox` / `pbox` / `gbox` commands from their own account, with no slot number (`pbox up FLAVOR`: any flavor but kvm). Reaped when idle (wbox 1h below 15%, the others 30 minutes) and at 6h (wbox, pbox) or 4h (gbox) from launch.
GitHub runners	`us-west-1`, one-time Spot metal	Disposable	Webhook-launched ARM64/x86 runners. At most eight healthy ARM64 and four x86 runners; idle, expired, and wedged runners terminate. Maximum instance lifetime 13h30m (drains from 12h).
Mac dev	`us-west-2`, optional Dedicated Host	Disposable 200 GB gp3 root	Temporary macOS build host. Disabled by default; teardown terminates the instance and releases the host after its 24-hour minimum.

The primary VPC is 10.0.0.0/16 in us-west-1. A private inter-region VPC peer reaches the 172.31.0.0/16 default VPC in us-west-2; NFS is never exposed publicly. Only the dev fleet's subnets (10.0.1.0/24, 10.0.2.0/24) route to the peer. The games router and match engines have their own subnets in the same VPC (10.0.64.0/22) with no peer route (docs/games-multiplayer.md). The Imagine backend's subnets (10.0.70.0/24, 10.0.71.0/24) use the same route table. The Windows playtest boxes' subnets (10.0.68.0/24 for wbox, 10.0.69.0/24 for the members' wboxes) route to the internet only.

GitHub workflow jobs -> API Gateway -> front Lambda -> queued webhook Lambda -> disposable Spot runners

Cloudflare Access -> outbound tunnel -> nextjs-dev -> 127.0.0.1 project ports

us-west-1 dev VPC <-> private VPC peer <-> us-west-2 io-box + parallel-box
        |
        |   fallback parallel boxes: their own isolated VPCs in four US regions, no peer, SSH only
        |
        +-> AWS Backup -> us-east-1 DR vault + dev-staging account vault

Access and trust boundaries

The Next.js containers share one memory pool in machine.slice: reclaim starts at 80% of host RAM, with bursts allowed up to an 85% hard limit and 1 GiB of combined swap. Any one person can still use their existing 75% allowance when the others are quiet. If the pool cannot reclaim enough memory, the kernel kills a process inside the containers; host SSH, networking and Cloudflare tunnels are outside that pool. Setup applies and persists these limits without restarting containers or the host. The remaining 15% is headroom for host services, not an allocation guarantee against memory consumed by other host processes.

The fcvm staging coordinator has a main-only CI release path (fcvm-fleet-ci.tf). CI builds and tests its ARM64 image, publishes an immutable commit tag, and invokes an image-only release function. That function takes Terraform's backend lock, rolls out the new image and records its digest in /fcvm-fleet/stage/coordinator-image; Terraform still checks the task's roles, commands, networking and other infrastructure. The CI job then checks Access, relaunches one tile in each half, and receives renderer frames over IPv4 under fresh DNS names. This can wake the GPU fleet; a capacity shortage fails the live check. CI has no Terraform-state access, administrator role or direct ECS permissions. Its deployed code has the coordinator's existing fleet-account runtime permissions. Environment variables and the staging-only Access token are provisioned by Terraform, with no manual CI setup. If CI releases between a saved plan and its apply, an apply-time parameter-version check rejects the stale plan; run a fresh plan. This check also detects a release attempt that rolled back to the same image.

List current connection commands:

cd ~/aws
terraform output

The ubuntu account on ARM, x86, and nextjs-dev has stable aliases:

ssh fcvm-arm
ssh fcvm-x86
ssh nextjs
ssh io

The aliases use Elastic IPs for the three long-lived us-west-1 boxes and fixed private address 172.31.48.10 for io-box. Dev servers carry a dedicated dev-hop key, not the fcvm-ec2 admin key, so ordinary dev-to-dev access does not grant an admin shell on a jumpbox. Dev hosts have no authorized dev-to-jumpbox login or privileged delegation path; the public SSH endpoint is still network-reachable. pbox launches the parallel boxes itself with a tag-scoped IAM grant (parallel-box-launch.tf); the forced-command key it used to carry is gone. gbox launches the GPU test box the same way (gpu-box.tf); both grants require their own launch template, so the two cannot be combined into a launch neither allows.

The metal, Next.js, and temporary-compute roles use a shared SSM connectivity policy without account-wide Parameter Store reads. Application parameter access is explicit; the metal role retains its intended runner SSH key access, but cannot send commands to AMI builders. Runner debug output uses that SSH path: the metal role explicitly denies account-wide Run Command input/output listing and reads, which AWS cannot scope to a runner. Its IPv6 assignment permissions cover only the two Terraform-managed metal ENIs. The unused predecessor aws-infrastructure-dev-instance-role retains its role/profile for history but has a protected Terraform-managed denial of all AWS actions. Next.js can read only its two Cloudflare connector credentials and the dev-only hop key through its setup policy, not the Cloudflare control-plane API token. The separate Cloudflare Access service-token workflow is unchanged. Runner credential retirement followed its own canary gates; the completed September 12 deployment and real-job acceptance are recorded under GitHub Actions below.

security-iam-passwords.tf sets both accounts' IAM-user console-password minimum to 14 characters, requires uppercase/lowercase/number/symbol, and prevents reuse of the last 24 passwords. It adds no periodic expiry or account-wide password-change permission. The checked-in import adopts main's existing singleton policy; recovery receives its first custom policy. At the September 12 inventory, neither account had an enabled IAM user console password, so this is a future-login guardrail. It does not create passwords, change root/SSO credentials or MFA, or enable centralized root management. Those controls are independent; IAM password policies do not govern root or Identity Center authentication. Both policies were applied and independently read back on September 12 at 19:07 UTC: all strength settings matched, with ExpirePasswords=false. The main root had MFA and no access keys; recovery root had no password or keys. Centralized root management is not enabled, and this audit did not verify the owner's Identity Center MFA settings.

security-account-contacts.tf manages only the SECURITY alternate contact in each account, with title Owner, the existing /alerts/email address, and that account's primary name/phone read directly from AWS. The owner approved these details on September 12, 2026. Primary, billing and operations contacts remain unchanged. Names and phone numbers are not committed to Git; target contact fields are marked sensitive. The native primary-contact data source still stores all returned contact/address fields in the existing encrypted, access-restricted Terraform state, so raw plans/state remain private. Before first creation, check get-alternate-contact --alternate-contact-type SECURITY under each account's own credentials; an existing contact must be reviewed and imported instead of silently overwritten. With this own-account provider routing, the import ID is SECURITY for each resource. After apply, compare each contact to its own primary name/phone and the approved alert address without printing personal values.

SSH and Eternal Terminal provide interactive access. t-claude supplies Claude's phone-oriented remote-control workflow; Codex uses its own app-server remote-control daemon. Conversation history is synchronized through claude-code-sync. On each metal box, fcvm-claude-rc.service starts remote-control sessions at boot for the user's active repositories under /home/ubuntu. “Active” means t-claude was used for that repository on this host during the preceding 30 days, its local Git checkout has moved during that period, or its worktree has uncommitted changes. Only github.com/ejc3/* checkouts qualify. The launcher uses the normal path-derived t-claude session so a later interactive launch attaches to the same work when run from that repository root.

OpenCode from a phone or tablet

Open opencode.cc-games.dev in your browser and sign in through Cloudflare Access. The connection is automatic: no server address, pairing link or OpenCode password. Add the site to your home screen if desired. Only the owner is admitted; the tunnel also validates the Access token before reaching the container.

opencode-web@ starts the existing OpenCode 2 shared server at boot inside the owner's container. It keeps the personal model configuration and sessions; the inference key is read from abliteration/api-key into the process environment at startup. opencode-web-proxy@ supplies the private upstream credential after Cloudflare login, so it never reaches the browser; the native server stays on loopback. The tunnel routes to the container only; there is no public web port or OpenTunnel endpoint. Restart opencode-web@<account>.service inside that container after rotating the inference key.

From the MacBook to the Next.js server

On the owner's Mac, ssh nextjs opens the owner's ejc3 container on the Next.js server. It uses the Mac's existing Touch ID SSH key; approve its authentication prompt when connecting. The public key is declared in local.nextjs_user_keys.ejc3 so access survives a rebuild. The Mac's SSH entry uses the server's stable Elastic IP.

SSH to the MacBook: `ssh mac`

There is one way into EJ's MacBook Pro, the same from every box, and one alias for it, mac: Cloudflare Access in front of macOS Remote Login (127.0.0.1:22 on the Mac). No reverse tunnel, and no inbound port on the Mac or in AWS. A plain ssh mac builds on the external drive, scripts System Events and Chrome, and drives the screen with Peekaboo; mac-gui <cmd> runs a command in the logged-in GUI session for what needs the login keychain (signing).

From	Service token (Secrets Manager)	Configured by
fcvm-arm, `ubuntu`	`macbook-ssh-access` (fcvm-arm's instance only)	`dev-user-data.tf`
nextjs-dev, `ejc3` only	`macbook-ssh-access-nextjs-ejc3` (`nextjs-dev-role`)	`nextjs-user-data.tf`

fcvm-arm also takes it, without a reboot, from the daily fcvm-mac-ssh-clients association. Both run the one installer in mac-ssh-client.tf (entry mac of local.mac_ssh_macs; Colton's Mac is the other entry): ~/.ssh/config.d-mac (Host mac: macbook-ssh.cc-games.dev through ProxyCommand ~/.ssh/mac-access-proxy %h, the account's own ~/.ssh/to-mac key, HostKeyAlias ejs-macbook-pro-via-fcvm checked against ~/.ssh/known_hosts.mac), the proxy command, and the pinned cloudflared in ~/.local/bin. Terraform never installs or copies the private key; restoring it is a prerequisite on a fresh root. The installer retires the old names (macbook, macbook-desktop, the reverse-tunnel mac on 127.0.0.1:2222) and their files, backs ~/.ssh/config up once per content (config.before-mac.<hash>), and is safe to rerun. Keep the Mac logged in, awake and online: its connector is a LaunchAgent and runs only in the logged-in session. These are local Mac services, not the optional EC2 Mac.

The MacBook: remote access and its scratch drive

How agents reach EJ's MacBook Pro over SSH, build on it, sign on it, drive its screen, and use its external scratch drive. The Cloudflare side is macbook-ssh.tf, the clients' side mac-ssh-client.tf, and everything installed on the Mac for agents comes from scripts/mac-setup.sh (run it from fcvm-arm or nextjs-dev; it is idempotent).

MacBook: The route

fcvm-arm (ubuntu) / nextjs-dev (ejc3)                                          MacBook
ssh mac ─> cloudflared access ssh ─> Cloudflare Access ═══ tunnel macbook-ssh ═══> connector ─> 127.0.0.1:22
           (the account's own        (service tokens     macbook-ssh.cc-games.dev  (the Mac      Remote Login
            service token)            only, no person)                              dials out)   (keys only)
                                                                                                    │ mac-gui
                                                                                                    ▼
                                                                                      the GUI session (login keychain)

Access (macbook-ssh.tf): the application on macbook-ssh.cc-games.dev admits the browser-manager owner policy (EJ, Google or one-time PIN) and one non_identity policy listing exactly the clients' service tokens. The connector also refuses a request without a valid Access token for the application.
The proxy command reads the account's token from Secrets Manager for each connection and hands it to cloudflared through TUNNEL_SERVICE_TOKEN_ID and TUNNEL_SERVICE_TOKEN_SECRET, never its arguments (ps shows every argument to every user of a shared box). A rotation needs nothing on the boxes.
The connector (com.ejcampbell.macbook-ssh.cloudflared on the Mac) is remotely managed: its ingress, ssh://127.0.0.1:22, comes from macbook-ssh.tf on apply.
nextjs-dev's token is box-wide in practice: every account there has sudo and can read what nextjs-dev-role reads. It opens only the Access gate; the Mac still wants ejc3's key. The containerized nextjs box hands it to ejc3's container only.

MacBook: What works where

macOS decides what a program may do (TCC) by its responsible process. Every Remote Login session, and everything it starts, is attributed to /usr/libexec/sshd-keygen-wrapper, so one set of grants covers every ssh mac:

Need	How	Why it works
Builds on `/Volumes/Scratch` (a freshly built binary included)	plain `ssh mac`	Full Disk Access on `sshd-keygen-wrapper` ("Allow full disk access for remote users")
`osascript` to System Events and Google Chrome	plain `ssh mac`	Automation grants for `sshd-keygen-wrapper` (System Events, Chrome)
Screenshots, clicks, typing	`peekaboo`	the Peekaboo app holds Screen Recording and Accessibility; the CLI talks to it over its Bridge socket
The login keychain: `codesign`, `xcodebuild archive`/`-exportArchive` with signing, `security` on it	`mac-gui <cmd>`	runs `<cmd>` in the GUI session's launchd namespace (`sudo launchctl asuser`), where securityd serves the unlocked login keychain

A plain Remote Login session cannot reach the login keychain (errSecInternalComponent from codesign, "User interaction is not allowed" from security), whatever its grants; mac-gui is the one way to it. Until 2026-10-18 the route ended at a private desktop-session sshd on 127.0.0.1:22222 instead, which had the keychain but made every new binary its own responsible process: each one stopped at a "would like to access files on a removable volume" prompt on the screen and hung there. It was retired.

MacBook: The agent tools (`scripts/mac-setup.sh`)

All in ~/.local/bin on the Mac, first on PATH in every SSH session (a managed block, # >>> mac-setup >>>, kept last in ~/.zshenv, so nothing earlier in the file can put /usr/bin ahead of the shims):

mac-gui [--timeout S] [--] CMD ...: runs CMD in the GUI session, keeping the working directory and environment (passed in a 0600 file, never on a command line); stdin, stdout, stderr and the exit status are CMD's, 124 on timeout.
mac-lease: the desktop lease, so two sessions' clicks, typing and app relaunches do not interleave. Advisory and lightweight: mac-lease run NAME [--ttl S] [--wait S] -- CMD holds it for one command; take NAME / release NAME hold it across several ssh mac calls (default ttl 600 s, renewable by taking it again); status shows the holder. The peekaboo wrapper waits (up to MAC_LEASE_WAIT, default 120 s) before any click, typing, app, window, menu or dialog command while another session holds the lease; pass yours as MAC_LEASE=NAME. Looking (see, image, capture, permissions) never waits. Take the lease before anything that relaunches or signs out of an app (Chrome above all: a relaunch can sign App Store Connect out).
mac-build: at most two heavy builds at once (MAC_BUILD_SLOTS). Shims named swift, xcodebuild and xcrun send swift build|test|run, xcodebuild actions and their xcrun forms through it, so every agent's builds take a slot without asking; a third waits and says so once. mac-build --status shows who holds the slots. Four parallel Swift builds once pushed the load average to 62 and timed out even ssh mac ps.
peekaboo: the Peekaboo CLI through the app's Bridge, with the lease check above. It runs ~/Library/Application Support/Peekaboo/cli-current/peekaboo, the CLI matching the installed app.
Peekaboo itself, pinned by scripts/mac/peekaboo-install.sh (openclaw/Peekaboo 4.9.5): the app in ~/Applications/Peekaboo.app, the CLI release in .../Peekaboo/cli-<version>, cli-current pointing at the one for the installed app, and the LaunchAgent com.ccgames.peekaboo (open -g -a ~/Applications/Peekaboo.app at login; it replaces the hand-made com.ejcampbell.peekaboo). Both downloads are checked against their pinned sha256 and Team ID (FWJYW4S8P8, OpenClaw Foundation) on the Mac before anything is replaced. A running Peekaboo of another version is left alone, and cli-current keeps pointing at its own CLI. To update it, quit Peekaboo at the Mac and run the script again, or run scripts/mac-setup.sh ALIAS --update-peekaboo, which quits it, swaps it (the old app goes to ~/.local/state/mac-setup/backups) and reopens it. An update from one release to the next keeps its privacy grants, because it has the same identifier and Team ID.
Peekaboo's permissions are a one-time step at the Mac, never scripted: System Settings > Privacy & Security > Screen Recording and Accessibility, Peekaboo on. Then from a client, ssh ALIAS 'peekaboo permissions status'.
xcode-select points at /Applications/Xcode.app (the Command Line Tools have no platform SDKs and no xcodebuild).

The same script sets up Colton's Mac: scripts/mac-setup.sh colton-mac from fcvm-arm.

MacBook: Permission prompts and other dialogs (policy)

Agents may approve permission prompts their own work triggers (owner, 2026-10-18: "Click allow"): Automation, Accessibility, Screen Recording, Full Disk Access, keychain access for our signing. Prefer granting them to sshd-keygen-wrapper (Remote Login), the one identity every session shares. A toggle in System Settings asks for Touch ID or the password; that one goes to the owner, as does anything else asking for a password.
Never answer another app's own dialog: a keychain password prompt from Roblox Studio, a mail account password, Time Machine offering to use Scratch, an app's sign-in. Report it (a screenshot and what it asks) and leave it.
Take the lease before driving the screen, and release it when done.

MacBook: Signing (archive, export, upload)

From a plain ssh mac, the build runs as usual and only the signing steps go through mac-gui. A minimal check, verified 2026-10-18 on a generated macOS app:

ssh mac 'cd /Volumes/Scratch/<project> &&
  mac-gui --timeout 1800 xcodebuild -scheme App -configuration Release \
    -archivePath build/App.xcarchive archive &&
  codesign -dv --verbose=2 build/App.xcarchive/Products/Applications/App.app 2>&1 | grep Authority'
#   Authority=Apple Development: ... (signed with the login keychain's identity)

xcodebuild -exportArchive and xcrun altool/notarytool go through mac-gui the same way. An upload that should survive the SSH connection dropping runs as a one-off LaunchAgent in the GUI domain instead (launchctl bootstrap gui/$(id -u) job.plist with RunAtLoad and no KeepAlive, logging to a file; launchctl bootout gui/$(id -u)/<label> afterwards). TestFlight groups, testers and build assignment go through the App Store Connect API with the games-worth-watching/app-store-connect key, not through Chrome.

MacBook: On the Mac

Rows marked connector setup come from scripts/mac-ssh-connector-setup.sh mac (run on a jumpbox; the same script as Colton's Mac), rows marked mac-setup from scripts/mac-setup.sh. Run against the MacBook as it is, the connector setup reloads nothing and changes nobody's access: its names are the ones the connector was set up with by hand, and the managed block holds the same two keys.

What	Where	State
Remote Login	System Settings > General > Sharing > Remote Login	on, Administrators only, "Allow full disk access for remote users" on
Keys only	`/etc/ssh/sshd_config.d/050-keys-only.conf`	`PasswordAuthentication no`, `KbdInteractiveAuthentication no` (connector setup checks it, and installs it when sudo allows)
Clients' keys	`~/.ssh/authorized_keys`, block `# >>> mac clients ...`	fcvm-arm `ubuntu`'s and nextjs-dev `ejc3`'s `to-mac` public keys (connector setup)
Host key	`/etc/ssh/ssh_host_ed25519_key.pub`	`host_key` of `mac` in `local.mac_ssh_macs`
Automation (`sshd-keygen-wrapper`)	System Settings > Privacy & Security > Automation	System Events and Google Chrome on
Agent tools	`~/.local/bin/{mac-gui,mac-lease,mac-build,mac-build-shim,peekaboo}`, shims `swift`, `xcodebuild`, `xcrun`	`scripts/mac-setup.sh`
Peekaboo app and CLI	`~/Applications/Peekaboo.app`, `~/Library/Application Support/Peekaboo/cli-<version>` and `cli-current`, LaunchAgent `com.ccgames.peekaboo`	pinned (mac-setup); Screen Recording and Accessibility granted once at the Mac
`cloudflared`	`~/.local/bin/cloudflared` (2026.9.3; 2025.4.0 or later for `--token-file`)	pinned in `scripts/claw-master-mac-setup.sh`, `scripts/claw-master-windows-setup.sh` and `scripts/mac-ssh-connector-setup.sh` (connector setup)
Connector token	`~/.cloudflared/macbook-ssh.token`, mode 600 (from `macbook-ssh-tunnel-token`)	connector setup
Connector	`~/Library/LaunchAgents/com.ejcampbell.macbook-ssh.cloudflared.plist`, log `~/Library/Logs/macbook-ssh/cloudflared.log`	remotely managed (connector setup)
Scratch drive	`/Volumes/Scratch` (external APFS, 1.8 TB)	see below

MacBook: The scratch drive

The internal disk runs nearly full with the owner's own data, so builds go to the external drive, one folder per project: /Volumes/Scratch/<project>/. For example dolphin-maps/ holds its checkout, build/ (DerivedData, archives, result bundles: BUILD_DIR in its scripts), tours/ (screenshots) and scratch/ (logs, helpers). Simulator devices still live on the internal disk (CoreSimulator), so delete a tour's simulator after use. Never write a project's files into ~/src loose.

Temporary files follow the same rule. Anything a session creates with mktemp (a test run's DerivedData, a build's intermediates) lands in TMPDIR, which by default is on the internal disk; one week of UI-test runs left 18 GB there and stopped every build at the 12 GB floor. The managed ~/.zshenv block (scripts/mac/set-zshenv.sh) points TMPDIR at the folder named in ~/.config/mac-setup/tmpdir when that file exists and its disk is mounted, for every zsh, non-interactive SSH sessions included; with no file, or the drive unplugged, the system default stays. On this Mac: ssh mac 'mkdir -p ~/.config/mac-setup && echo /Volumes/Scratch/tmp > ~/.config/mac-setup/tmpdir'.

MacBook: Checks

# From fcvm-arm (ubuntu) or nextjs-dev (ejc3). The session comes from Remote Login: sshd-session under launchd.
ssh mac 'p=$$; while [ "$p" -gt 1 ]; do ps -o comm= -p $p; p=$(ps -o ppid= -p $p | tr -d " "); done'

# A program macOS has never seen reads the drive with no prompt ("read ok", not BLOCKED).
ssh mac 'd=$(mktemp -d); printf "#include <stdio.h>\nint main(int c,char**v){return fopen(v[1],\"r\")?puts(\"read ok\"),0:2;}\n" > $d/p.c &&
  clang -o $d/probe-$RANDOM $d/p.c && ($d/probe-* /Volumes/Scratch/.DS_Store & K=$!;
  for i in 1 2 3 4 5 6; do kill -0 $K 2>/dev/null || exit 0; sleep 1; done; echo BLOCKED; kill $K); rm -rf $d'

# Apple Events, Peekaboo, and the keychain through mac-gui.
ssh mac 'osascript -e "tell application \"System Events\" to get name of first process whose frontmost is true";
  osascript -e "tell application \"Google Chrome\" to get count of windows";
  peekaboo permissions status | head -3;
  mac-gui security show-keychain-info ~/Library/Keychains/login.keychain-db;
  mac-lease status; mac-build --status'

# Passwords are refused. This reads the methods the sshd offers, so it does not depend on the
# client: expect exactly "Authentications that can continue: publickey".
ssh mac 'ssh -v -o PubkeyAuthentication=no -o PreferredAuthentications=password,keyboard-interactive \
  -o NumberOfPasswordPrompts=0 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
  -p 22 ejcampbell@127.0.0.1 true 2>&1 | grep "Authentications that can continue"'

# Access lets a service token in and nobody else. Without credentials it answers with its login page:
curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' https://macbook-ssh.cc-games.dev
#   expect: 302 https://ejc3.cloudflareaccess.com/...

The connector's health, from a jumpbox (only cloudflare-tunnel-token may read tunnel state; it goes to curl on stdin, never argv):

{ set +x; } 2>/dev/null
T=$(aws secretsmanager get-secret-value --secret-id cloudflare-tunnel-token \
      --region us-west-1 --query SecretString --output text)
A=https://api.cloudflare.com/client/v4/accounts/12ea67fb7ced068de03f35c22688e436/cfd_tunnel
read -r ID STATUS < <(printf 'Authorization: Bearer %s\n' "$T" | curl -s -H @- "$A?name=macbook-ssh&is_deleted=false" \
  | python3 -c 'import json,sys; t=json.load(sys.stdin)["result"][0]; print(t["id"], t["status"])')
printf 'Authorization: Bearer %s\n' "$T" | curl -s -H @- "$A/$ID/connections" \
  | python3 -c 'import json,sys; print(sys.argv[1], len(json.load(sys.stdin)["result"]))' "$STATUS"
unset T

healthy 2 (or any count above zero) is a pass; down 0 or inactive 0 is not.

Symptom of a removable-volume prompt: a build step that sits at 0% CPU for minutes. sample <pid> 2 on the Mac shows the main thread in getcwd or open on /Volumes/Scratch. Check the session's ancestry (above), then "Allow full disk access for remote users" (System Settings > General > Sharing > Remote Login, (i)), before anything else. A build that sits at "mac-build: waiting for one of 2 build slots" is the cap: mac-build --status.

MacBook: Recovery

ssh mac hangs or is refused: on the Mac, launchctl kickstart -k gui/$(id -u)/com.ejcampbell.macbook-ssh.cloudflared and read ~/Library/Logs/macbook-ssh-cloudflared.log. After a reboot nothing answers until someone logs in at the Mac.
mac-gui fails with a sudo error: it needs passwordless sudo for launchctl asuser (the owner's account has it). codesign asks for the keychain password: that prompt is the owner's; report it.
A service token expired or leaked: each lasts a year (duration = "8760h"), and an expired one is not replaced by an ordinary apply. Replace the one concerned from a fresh worktree, terraform apply -replace=cloudflare_zero_trust_access_service_token.macbook_ssh_fcvm (fcvm-arm) or ...macbook_ssh_nextjs_ejc3 (nextjs-dev). The same apply rewrites that client's secret, which the proxy command reads on every connection, so nothing changes on the box. Then check ssh mac true from that client.
Reloading the connector ends every session through it, including the programs those sessions started. Start long jobs on the Mac detached (nohup … &, logging to a file on the scratch drive) or as a one-off LaunchAgent (above).
The connector token leaked: anyone holding it can run a connector for this one tunnel, and nothing more: every request still has to pass Access. Rotating it replaces cloudflare_zero_trust_tunnel_cloudflared.macbook_ssh (a new tunnel id and token), which disconnects the Mac's connector, the only route in. So the new token goes over the temporary bootstrap route in MacBook: Rebuilding from nothing (step 2), or at the Mac. Stop the connector first (launchctl bootout gui/$(id -u)/com.ejcampbell.macbook-ssh.cloudflared): Cloudflare refuses to delete a tunnel with active connections, and Terraform deletes this one before it creates the new one. Then, from a fresh worktree, terraform apply -replace=cloudflare_zero_trust_tunnel_cloudflared.macbook_ssh (its config and DNS record follow it, and macbook-ssh-tunnel-token gets the new token; delete the old tunnel in the API afterwards), install the new token as in step 3 of the rebuild (the script sees the token changed and reloads the connector), check ssh mac, and take the bootstrap route down again.

MacBook: Rebuilding from nothing

Remote Login. On the Mac: turn on Remote Login (Administrators) and "Allow full disk access for remote users". A reinstalled Mac has a new host key: replace the host_key of mac in local.mac_ssh_macs with /etc/ssh/ssh_host_ed25519_key.pub, read on the Mac.
A way to reach it the first time. Nothing reaches a wiped Mac. The Mac dials out to fcvm-arm with a tunnel-only key whose ubuntu authorized_keys line is restrict,port-forwarding,command="/bin/false",permitlisten="127.0.0.1:2223",permitopen="127.0.0.1:9" ssh-ed25519 AAAA... <comment> (permitopen pins local forwarding to a closed port), running ssh -NT -o ExitOnForwardFailure=yes -o StrictHostKeyChecking=yes -R 127.0.0.1:2223:127.0.0.1:22 -i <key> ubuntu@<fcvm-arm's Elastic IP> with fcvm-arm's host key pinned on the Mac from a machine that already trusts it, never accepted blind; and the Mac's ~/.ssh/authorized_keys holds fcvm-arm ubuntu's to-mac.pub.
Everything else on the route, from a jumpbox: scripts/mac-ssh-connector-setup.sh mac --bootstrap-port 2223. It makes Remote Login keys-only (050-keys-only.conf; when sudo needs a password it prints the command to type at the Mac), installs the pinned cloudflared, the token (macbook-ssh-tunnel-token, through ssh stdin, mode 600), the connector LaunchAgent com.ejcampbell.macbook-ssh.cloudflared, and the managed block of the clients' keys (fcvm-arm ubuntu's and nextjs-dev ejc3's to-mac.pub). Once ssh mac works, stop the bootstrap tunnel and remove its key line from fcvm-arm: Cloudflare is the only way in.
The clients: the dev-user-data.tf and nextjs-user-data.tf setups install Host mac (fcvm-arm at its next boot through dev-selfupdate, nextjs-dev through setup-sync).
The agent tools: scripts/mac-setup.sh from fcvm-arm. Then the first osascript to System Events and to Chrome from ssh mac each raise an Automation prompt for sshd-keygen-wrapper: Allow.
Run the checks above.

History. Until 2026-10-18 the Mac also kept reverse SSH tunnels to fcvm-arm (127.0.0.1:2223) and to nextjs-dev (127.0.0.1:2222); the owner chose Cloudflare as the only route and both were retired. The same day the route moved from Remote Login to a private desktop-session sshd (127.0.0.1:22222, for the keychain and Apple Events) and back (issue #505): under it every new binary stopped at a removable-volume prompt, and Remote Login with Automation grants plus mac-gui does everything it did.

SSH to Colton's Mac: `ssh colton-mac`

The same shape as the MacBook (owner, 2026-10-19: "Colton and I ... from the nextjs server, and just me from fcvm"): Cloudflare Access in front of the Mac's Remote Login at colton-mac-ssh.cc-games.dev (colton-mac-ssh.tf), a connector LaunchAgent on the Mac, and the mac-ssh-client.tf installer on each client with alias colton-mac (entry colton-mac of local.mac_ssh_macs, which pins the Mac's host key under HostKeyAlias colton-mac). No inbound port and no reverse tunnel.

From	Service token (Secrets Manager)	Configured by
fcvm-arm, `ubuntu`	`colton-mac-ssh-access-fcvm` (fcvm-arm's instance only)	`dev-user-data.tf` at boot, `fcvm-mac-ssh-clients` association daily
nextjs-dev, `colton`	`colton-mac-ssh-access-nextjs-colton` (`nextjs-dev-role`)	`nextjs-user-data.tf` (`setup-sync`, within 10 minutes)
nextjs-dev, `ejc3`	`colton-mac-ssh-access-nextjs-ejc3` (`nextjs-dev-role`)	`nextjs-user-data.tf`
EJ in a browser	the browser-manager owner policy

Each account has its own key, ~/.ssh/to-colton-mac, made on its box by the installer as that account when missing (only the public half is printed); the private half never leaves the box. The connector token's secret, colton-mac-ssh-tunnel-token, is readable in AWS by administration only, but the clients can read the token on the Mac: it is a 0600 file of the one account they all log in as. A connector run with it can only take this tunnel's traffic and break connections; it cannot get in (Access gates every request) or pass for the Mac (clients pin its host key). scripts/mac-ssh-connector-setup.sh colton-mac, run on a jumpbox, puts the Mac on the route:

it checks that Remote Login offers publickey only, and otherwise installs the MacBook's keys-only drop-in (/etc/ssh/sshd_config.d/050-keys-only.conf) if sudo needs no password, or stops and prints the command to type at the Mac;
the pinned cloudflared (verified on the Mac), the token through ssh stdin into ~/.cloudflared/colton-mac-ssh.token (0600), and the LaunchAgent com.ccgames.colton-mac-ssh.cloudflared;
a managed block in the Mac's ~/.ssh/authorized_keys with the three clients' public keys (fcvm-arm's over ssh, each nextjs account's own file on the containerized box, found by its Box=nextjs-ct tag, over its own SSM command), each accepted only as exactly one ssh-ed25519 <blob>.

It reaches the Mac as ssh colton-mac from fcvm-arm; --bootstrap-port PORT points that alias at a temporary reverse tunnel on fcvm-arm's 127.0.0.1:PORT. On the containerized box (nextjs-ct), colton's and ejc3's containers have ssh colton-mac (local.nextjs_ct_mac_clients), each with its own token: colton-mac-ssh-access-nextjs-ejc3 is the owner's (denied to every member's container) and colton-mac-ssh-access-nextjs-colton is denied to every member's container but colton's (nextjs-ct-user-data.tf).

First bring-up (the Mac kept a temporary reverse tunnel to fcvm-arm's 127.0.0.1:2224, set up by hand on 2026-10-19):

Apply. The association rewrites fcvm-arm's colton-mac alias to the Cloudflare route at once (the same to-colton-mac key and host key pin); nextjs-dev makes colton's and ejc3's keys at its next setup-sync.
From a jumpbox: scripts/mac-ssh-connector-setup.sh colton-mac --bootstrap-port 2224. It stops before the token and the connector if a client's key is not there yet or Remote Login still takes passwords (type the printed command at the Mac, then run it again).
Check from each client: ssh colton-mac true (fcvm-arm ubuntu, nextjs-dev colton and ejc3).
Retire the temporary tunnel: on the Mac, launchctl bootout gui/$(id -u)/com.ccgames.fcvm-tunnel and delete ~/Library/LaunchAgents/com.ccgames.fcvm-tunnel.plist, ~/.ssh/fcvm-tunnel* and ~/.ssh/known_hosts.fcvm-tunnel (over ssh colton-mac); on fcvm-arm, delete the restrict,...permitlisten="127.0.0.1:2224"... line from ~/.ssh/authorized_keys. Cloudflare is then the only way in.

Re-run step 2 without --bootstrap-port after adding a client or when a client's key changes. The script reloads the connector only when its LaunchAgent or token changed, detached so the reload completes even though it ends the sessions through it (the first run after the LaunchAgent took the MacBook's form, with ProcessType and a stdout log, reloads it once). The agent tools are scripts/mac-setup.sh colton-mac. Keep the Mac logged in, awake and online, as for the MacBook. Rotation is the MacBook's: terraform apply -replace='cloudflare_zero_trust_access_service_token.colton_mac_ssh["nextjs-colton"]' for one client (its secret follows; nothing changes on the box), and a leaked connector token means replacing cloudflare_zero_trust_tunnel_cloudflared.colton_mac_ssh with the connector stopped, then step 2 over a temporary route.

SSH to Colin's PC, and its screen: `ssh colin-pc`, Windows-MCP

Colton's Mac's shape on a Windows PC (owner, 2026-10-20): Cloudflare Access in front of the PC's OpenSSH Server at colin-pc-ssh.cc-games.dev (colin-pc-ssh.tf), a connector on the PC, and the mac-ssh-client.tf installer on each client with alias colin-pc (entry colin-pc of local.mac_ssh_macs, which pins the PC's host key). No inbound port.

From	Service token (Secrets Manager)	Configured by
fcvm-arm, `ubuntu`	`colin-pc-ssh-access-fcvm` (fcvm-arm's instance only)	`dev-user-data.tf` at boot, `fcvm-mac-ssh-clients` association daily
nextjs-ct, `colin` (his container only)	`colin-pc-ssh-access-nextjs-colin` (`nextjs-dev-role`; denied to every other member's container)	`local.nextjs_ct_mac_clients`, at the container's provision
EJ in a browser	the browser-manager owner policy

Unlike the Macs' LaunchAgents, the connector is a SYSTEM task at boot (cloudflared colin-pc), so the PC is reachable before anyone logs in. Its token (colin-pc-ssh-tunnel-token, administration only) is in C:\ProgramData\cloudflared-colin-pc, readable by SYSTEM and Administrators only; a client logs in as an administrator, so it could read it there, with the same consequence as on the Macs (a rogue connector for this tunnel, never a way in).

The screen is Windows-MCP (pinned): screenshots, the UI tree, clicks, typing, shortcuts and app control, served by a per-user logon task in the PC user's own session on the PC's 127.0.0.1:8000, with anonymous telemetry off and a bearer key made on the PC. A client's Claude Code reaches it through scripts/windows/windows-mcp-bridge colin-pc, a stdio MCP server that opens an ssh forward for the session and relays with mcp-remote (locked in scripts/windows/mcp-remote/); the key reaches it in a 0600 header file, read once over ssh, never an argument or an environment variable. scripts/windows/windows-mcp-client-setup.sh colin-pc, run as the client account, installs both and adds the Claude entry; a nextjs-ct container's provision runs it for each PC of local.windows_mcp_pcs the account reaches. The members' own machine tokens (Colton's Mac, Colin's PC) are denied to every other member's container in one statement, MemberMachineTokensOwnOnly.

Windows-MCP is pinned to an upstream commit (WINDOWS_MCP_COMMIT, its source zip's sha256 checked on the PC), not PyPI 0.8.8. It keeps the PC user in charge: while they use the mouse or keyboard (and 10 seconds after), direct input (Click, Type, Move, Scroll, Shortcut, switching windows) is refused with USER_CONTROL, and background tools (PowerShell, screenshots, launching an app) still run. 0.8.8 instead latched in USER_CONTROL whenever its hook missed a key-up (Win+L, an elevated window in front), refusing every tool until the server restarted.

scripts/windows-pc-setup.sh colin-pc, run on a jumpbox, puts the PC on the route and installs Windows-MCP: OpenSSH keys only and on loopback only with its inbound firewall rule off, the clients' keys in a managed block of administrators_authorized_keys, the pinned cloudflared (verified on the PC), the token through ssh stdin, the connector task, and Windows-MCP. A re-run changes only what differs; a connector restart runs from a one-off SYSTEM task, since it ends the sessions through it.

Never a key of the PC's on ubuntu (AGENTS.md, "Machines outside AWS"). The first-contact reverse tunnel below ends on pcboot, a system account on fcvm-arm with no shell (pc_bootstrap_account in mac-ssh-client.tf, installed with fcvm-arm's ssh clients at boot and daily): its keys are in the root-owned /etc/ssh/pcboot/authorized_keys, each pinned to its one port and expiring after two days, and its sshd Match block refuses Unix-socket and local forwards, agent, X11 and a TTY. The tunnel is retired as soon as the Cloudflare route works. The clients' configs set ForwardAgent no and ForwardX11 no, and whatever comes back from a PC is untrusted data.

The enrollment service (pc-enroll.tf, scripts/pc-enroll.mjs) is what replaces that bootstrap: at https://enroll.cc-games.dev, behind Cloudflare Access for the owner only, a Lambda hands a PC listed in local.pc_enroll_pcs its connector token and its clients' public keys after the owner logs in to Access on the PC. The Lambda checks the Access token itself (the API Gateway origin is public; that check is the boundary), pins the PC's host key at /pc-ssh/<pc>/host-key only when none is pinned (seeded from local.mac_ssh_macs), and lets only an owner replace a pin (POST /pcs/<pc>/host-key/replace). It reads client keys from /pc-ssh/<pc>/clients/<box>-<account> and refuses a PC until every slot holds one. The PC's script, the client keys' publishers and the bootstrap's removal come in the next change; until then the steps below still apply. Revoking a PC's connector: rotate the tunnel's token first (Cloudflare's "Rotate a compromised token"), then delete its connections (cloudflared tunnel cleanup <tunnel>, or the tunnel connections DELETE API), or delete the tunnel. A rotation alone leaves a running connector connected; a cleanup alone lets it reconnect.

A new PC (Colin's was set up this way on 2026-10-20):

Add its row to local.mac_ssh_macs and a <alias>-ssh.tf like colin-pc-ssh.tf; claim a port block for it (ports claim <alias>). After apply, fcvm-arm has ~/.ssh/to-<alias>.
On fcvm-arm: scripts/windows-pc-bootstrap.sh print <alias> PORT prints the one script the person at the PC pastes into PowerShell opened with Run as administrator (scripts/windows/bootstrap.ps1 with this PC's values; publish it behind a Copy button). It installs OpenSSH closed to the network, authorizes fcvm-arm's key, and keeps a reverse tunnel, as pcboot, to fcvm-arm's 127.0.0.1:PORT. It ends by printing the PC's own key: authorize it for that tunnel only with scripts/windows-pc-bootstrap.sh allow <alias> PORT '<key line>' (it writes pcboot's key file, with sudo).
From a jumpbox: scripts/windows-pc-setup.sh <alias> --bootstrap-port PORT. Then, on each client account, ssh <alias> true and scripts/windows/windows-mcp-client-setup.sh <alias>.
Retire the tunnel: on fcvm-arm scripts/windows-pc-bootstrap.sh retire <alias>, and on the PC Unregister-ScheduledTask 'devbox tunnel' (over ssh <alias>). colin-pc was bootstrapped on 2026-10-20 before pcboot existed: its key is a line in fcvm-arm ubuntu's ~/.ssh/authorized_keys (comment windows-pc-BLD, no expiry), to be deleted there by hand once its Cloudflare route works (ejc3/aws #557).

macOS VMs on the MacBook (ejc3, skevh)

The MacBook runs two Tart macOS VMs, ejc3 for EJ and skevh for Steve, cloned from ghcr.io/cirruslabs/macos-tahoe-base. Each signs its admin account in at boot, so it always has a desktop, and runs its own cloudflared connector for its own tunnel (mac-vms.tf). Per VM there are two TCP routes:

VM	Screen Sharing	SSH	Access
`ejc3`	`ejc3-mac.cc-games.dev`	`ejc3-mac-ssh.cc-games.dev`	EJ only (Google or one-time PIN)
`skevh`	`skevh-mac.cc-games.dev`	`skevh-mac-ssh.cc-games.dev`	GitHub members of `dolphin-labs-hq`

cloudflared access tcp --hostname skevh-mac.cc-games.dev --url localhost:5901
open vnc://localhost:5901   # sign in as admin; password: secret mac-vm-<vm>-admin-password
ssh -o ProxyCommand='cloudflared access ssh --hostname %h' admin@skevh-mac-ssh.cc-games.dev

SSH accepts keys only: the MacBook's ejcampbell key plus ejc3's keys from nextjs_user_keys (minus the fleet hop key) in ejc3, and Steve's keys from nextjs_user_keys in skevh. The admin password is used only for Screen Sharing; SSH never uses it. It is the generated secret mac-vm-<vm>-admin-password: both running guests accept it and reject the image's admin/admin (checked 2026-09-26 with dscl . -authonly admin ... through tart exec -i <vm>, reading the password from stdin). Terraform only STORES the value. Nothing applies it inside a guest, so a VM created or rebuilt from the Tart image starts with the image default until someone sets the secret by hand on the MacBook and re-runs that check. Access is the real gate: both hostnames sit behind Cloudflare Access with origin enforcement. Inside each VM, the connector is a launchd daemon that reads its token from a root-only file (/etc/cloudflared/tunnel.token), never from argv.

On the MacBook, ~/Library/LaunchAgents/com.ejcampbell.tart.<vm>.plist runs tart run --no-graphics <vm> with KeepAlive while ejcampbell is logged in; Tart needs an unlocked login keychain, so the VMs start after that login, and after a reboot only once FileVault is unlocked at the machine. Local admin from the MacBook:

tart list
tart exec <vm> <command>      # through the guest agent, no network needed
tart run --vnc <vm>            # watch the screen (stop the LaunchAgent first)

Start a Codex session

Codex is installed per Unix account on the metal boxes, jumpbox-2, and nextjs-dev. Authentication is deliberately not stored in Terraform. On a fresh account, SSH in as the person who will own the sessions and complete the one-time logins. That person needs Codex access in their ChatGPT account/workspace, the current ChatGPT app with Remote available, and device-code login enabled by either their account or workspace admin.

codex --version
codex login --device-auth
codex login status

gh auth login
gh auth setup-git

Device authentication prints a URL and one-time code for a browser or phone. Use the same ChatGPT account and workspace that will open the remote session. Device-code login may first need to be enabled in the account's ChatGPT security settings or by its workspace administrator. ~/.codex/auth.json is a persistent credential: never copy it between users, print it, or commit it. See the official Codex authentication guide.

GitHub authentication is separate from Codex. The metal boxes have a limited PAT fallback for a brand-new bootstrap, but each person should run gh auth login for their own identity. On nextjs-dev, log in as colton, connor, ejc3, or skevh rather than sharing credentials, then also run:

vercel login
claude login

Claude login is also personal and persistent on each metal box. After the first login, start the already-enabled boot service once if the machine is currently running:

claude login
claude auth status
sudo systemctl start fcvm-claude-rc.service
systemctl status fcvm-claude-rc.service --no-pager

The service checks claude auth status on every boot and skips cleanly if the login is missing or expired. To seed a new repository, open it once with t-claude --remote-control; a host-local marker makes subsequent boots include it while it remains recent. All metal repositories share one tmux server and one aggregate service, so do not restart or stop this service just to refresh one repository: that can end every managed and interactive t-claude session on the host. Check the most recent aggregate result with cat ~/.local/state/fcvm-claude/last-start.

Refresh the generated instructions that seed app-created Codex sessions:

# ubuntu on a metal box or jumpbox-2
sudo codex-agents-refresh

# a personal account on nextjs-dev
sudo kid-agents-refresh "$USER"

The first command inventories the repositories under /home/ubuntu. The second points a kids' session at that user's published project and explains that its ndev service is already running. Both write ~/Documents/Codex/AGENTS.md.

Enable remote control only after codex login succeeds:

sudo systemctl enable --now "codex-rc@$USER.service"
systemctl status "codex-rc@$USER.service" --no-pager
codex remote-control start --json

Open Remote in ChatGPT while signed in to that same account and workspace. If it asks for a manual pairing code, generate one with:

codex remote-control pair

The command creates a short-lived code; see the official remote-control CLI reference. Remote control is currently marked experimental upstream.

A phone-created task starts in ~/Documents/Codex/<date>/<task>/, which is an empty scratch directory, not a checkout. The generated parent AGENTS.md tells Codex where the real repositories are. Name the repository in the first prompt; if no repository is named, the agent should ask instead of editing the scratch directory. For a terminal session that should start in this repository directly:

cd ~/aws
codex

The ChatGPT client sends the approval and sandbox mode for each remote thread, and those settings override ~/.codex/config.toml. Enable the Remote setting called dangerous mode only when full host access is intended. These EC2 development machines are designed as the external isolation boundary; the generated config supplies their real project roots when the app selects workspace-write mode.

Once enabled, codex-rc@ returns on every boot. Machine convergence also enables it when it sees an existing login. nextjs-dev refreshes the seed files, updates Codex/Claude/Vercel, and restarts enabled agent services nightly. Codex, GitHub, Vercel, and Claude remain four independent logins.

Firecracker development

The two metal boxes are the main fcvm workstations:

ARM64 uses r8gd.metal-24xl; x86 uses c5d.metal.
Both are persistent Spot instances with stop interruption behavior and static EIPs.
Their EBS roots survive a normal stop/start: 800 GB on ARM and 300 GB on x86.
Instance-store NVMe is for VM images, build output, caches, and temporary Btrfs data. It is erased by a stop, Spot interruption, or host loss.
ARM nested virtualization requires the custom fcvm kernel and patch set documented in AGENTS.md; uname -r reports <version>-fcvm-<build-sha> on a correctly installed host.
An hourly Lambda queries a 12-hour range of five-minute peak CPU points. Any returned point at or above 5% keeps the instance running; otherwise it stops the instance and sends an SNS notification.

Terraform manages the configuration of a stopped metal box, but an aws_instance resource does not itself wake an already-stopped instance. Unlike io-box, ARM and x86 currently have no aws_ec2_instance_state wake resource. A normal plan/apply therefore will not wake them. Do not hide a mutating AWS CLI call in a script: either obtain explicit authorization for the break-glass start procedure in AGENTS.md, or add and review a Terraform-modeled lifecycle first. Persistent Spot can start only after its request reaches disabled / instance-stopped-by-user.

The idle check is a cost-saving heuristic, not proof of 12 uninterrupted idle hours. The current Lambda accepts 12 returned CPU datapoints even though a complete 12-hour, five-minute series contains about 144; missing metric periods can therefore count as idle.

Shared high-speed I/O

io-box exports its local 1.25 TB NVMe disk as private NFSv4.2. ARM, x86, Next.js, and new parallel boxes receive a bounded soft automount:

/mnt/io -> 172.31.48.10:/  (the server's `/srv/io` export)

Use it for bulk scratch, caches, and reproducible artifacts. Do not put the only copy of source or results there: every stop creates a new empty filesystem. Clients idle-unmount after ten minutes so a stopped server does not block boot or hang a shell indefinitely.

NFS (2049) is admitted only from the subnets its clients live in, local.io_box_nfs_client_cidrs in io-box.tf: the dev fleet's 10.0.1.0/24 and 10.0.2.0/24, and the parallel boxes' 172.31.48.0/20 in us-west-2d. It is never a whole VPC: a security group cannot be referenced across the inter-region peer, so the source CIDR is the only filter. A new NFS client in another subnet needs that subnet added there (scripts/test-io-box-nfs.py fails until it is). The export line in /etc/exports.d is rendered from the same list by an SSM association (io_box_exports) that runs whenever the box is up, because user_data is ignored after creation: the server and the security group cannot disagree.

The I/O watchdog queries CPU, disk bytes, and network bytes over the same intended 12-hour window. A returned five-minute point with CPU at least 5%, disk I/O at least 1 MiB, or network I/O at least 64 MiB keeps it up. Missing I/O series currently count as zero, so this is also a heuristic rather than a continuity guarantee. Wake it through the modeled Terraform state resource:

terraform apply -target=aws_ec2_instance_state.io_box

A normal full terraform apply also reconciles that resource to running; the target is just the narrow wake command. The root EBS volume survives the stop; /mnt/io data does not.

On-demand parallel compute

Eight independent boxes (parallel-box, parallel-box-2 .. parallel-box-8), each with its own persistent work volume (300 GB for box 1, 100 GB for the others), so eight jobs can run at once. From ARM or x86:

pbox flavors               # the menu: vCPU, memory, local disk, what each is for, rough price
pbox status                # every box, with the flavor each runs as
pbox up compute-48         # launch box 1 as a flavor (required), then connect
pbox up 3 memory-96        # launch box 3 (independent volume and lifecycle; 1-8)
pbox up 3 kvm-192 --open   # the same, open to the internet with a public IPv6 /80 for fcvm VMs (below)
pbox up 3 kvm-gpu-96 --on-demand   # buy this one launch on-demand, not spot (about 3x the price; below)

pbox up 3 kvm-gpu-96 --lease 3h   # not terminated for idleness for 3 hours (30m to 12h; below)
pbox ssh [N]
pbox down [N]              # terminate compute; keep that box's /mnt/work

up needs a flavor (pbox-flavors.tf), and with none it prints the menu and launches nothing. A flavor is a shape and a size: compute (2 GiB per vCPU, CPU-bound work), general (4 GiB), memory (8 GiB, memory-bound work), storage (8 GiB plus local NVMe at /mnt/nvme, gone with the box: local-disk-bound work), each in 32, 48, 96 and 192 vCPU, and kvm-96 / kvm-192 (bare metal with /dev/kvm, slow to boot, the owner's slots only). Pick the smallest that fits the work: a smaller box costs less and finds Spot capacity more often.

kvm-gpu-64 is a g5g.metal (Graviton2, 64 vCPU, 128 GiB, two NVIDIA T4G) for GPU passthrough into fcvm microVMs: /dev/kvm, the SMMU on, vfio-pci loaded at boot, and every NVIDIA function bound to vfio-pci by a persistent driverctl override (no NVIDIA driver on the host). ubuntu may open /dev/vfio/* (group kvm) and has an unlimited memlock, so a rootless fcvm can use them; /etc/pbox-gpu lists each GPU and its IOMMU group, and up fails unless every one is on vfio-pci. us-east-2 does not offer the type, so it launches at a fallback site with a fresh work volume. A GPU flavor implies --fallback: home first, then every fallback region that offers the type (us-east-1 and us-west-2 for this one), best spot placement score first:

pbox up 3 kvm-gpu-64

When a region refuses on a quota, pbox names the limit (MaxSpotInstanceCountExceeded and the spot vCPU quota code, VcpuLimitExceeded and the on-demand one, or InstanceLimitExceeded) with its value, first cancels any spot request in that region still open or active after its instance is gone (they count against the spot quota for up to half an hour after a terminate) and retries once, then moves on to the next region.

kvm-gpu-96 is a g4dn.metal (x86_64 Intel, 96 vCPU, 384 GiB, eight NVIDIA T4, 1.8 TB of local NVMe at /mnt/nvme, about $4/hr spot) set up the same way. us-east-2c offers it, so pbox up 3 kvm-gpu-96 launches at home on the persistent work volume. It is the one x86 type: the launch templates boot arm64 Ubuntu, so pbox passes the x86_64 build of the same image with --image-id (var.parallel_box_ami_ohio_x86 at home, each fallback site's by name, published in /infra/parallel-box; IAM allows each by exact ARN). Its first boot puts intel_iommu=on iommu=pt in a GRUB drop-in and reboots once, and up waits through that reboot before checking the GPUs.

Both GPU flavors count against the "All G and VT Spot Instance Requests" quota (L-3819A6DF) where they launch: pbox-flavors.tf requests 96 vCPU (one box of the largest GPU flavor) in us-east-2, us-west-2 and us-east-1 at apply when the current quota is lower. The launcher tries the flavor's types in turn in the availability zone of the persistent work volumes, and IAM allows exactly the flavors' types. One shared watchdog checks every five minutes and terminates any box after 30 minutes below 5% CPU with no disk or network activity (512 MiB of EBS or instance-store reads or writes, or 64 MiB of network, in any five minutes keeps it).

pbox up N FLAVOR --on-demand buys that one launch on-demand; every other launch stays spot. Use it for work a spot reclaim would lose, or when spot capacity for the type is short, as it often is for the bare-metal GPU types. It costs about 3x spot: up prints each type's on-demand list price (kvm-gpu-96, a g4dn.metal, is about $7.82/hr on-demand in us-east-2 against about $4 spot). It counts against the region's on-demand vCPU quota, "Running On-Demand G and VT instances" (L-DB2E81BA) for the GPU flavors and "Running On-Demand Standard (A, C, D, H, I, M, R, T, Z) instances" (L-1216C47A) for the rest, not the spot quota; up names it with its value, and a VcpuLimitExceeded refusal moves on to the next type or region. The box launches from its own on-demand launch template (parallel-box-3-ondemand; parallel-box-fallback-ondemand at a fallback site), which Terraform makes from the same definition as the spot one, without spot options. pbox status shows it as ON-DEMAND, and the idle watchdog terminates it after 30 idle minutes like a spot box. It works with --fallback, --regions, --open and the GPU flavors; it is refused for other commands and for a member's pbox, whose grant allows spot only. parallel_box_spot = false still switches every launch to on-demand at once, with an apply.

Interactive and GPU work has idle stretches: between a finished setup and the next test, or while a failed test is read. pbox up N FLAVOR --lease DURATION (90m, 2h, 1h30m; 30 minutes to 12 hours) tags the box PboxLeaseUntil with the lease's end, and the watchdog does not terminate it for idleness until then; after that the normal rule applies. The watchdog ends any lease at 12 hours from launch, whatever the tag says. A leased box costs money for the whole lease, busy or not: a kvm-gpu-96 is about $4/hr spot, so a 12-hour lease is about $50. pbox status shows lease: until HH:MM UTC (Xh Ym left) or lease: ended. The lease is set at launch only; a running box keeps what it has (pbox down N, then up again with --lease). Members' boxes take no lease.

Several jobs share the eight slots, and a slot can hold a different instance minutes after a script looked it up. pbox up ends with launched box=N ... id=i-... (with failed=<reason> and exit 1 when the box was launched but did not come up), so scripts carry that id and use pbox ip|ssh|down N --instance i-..., which refuse when the slot holds another instance; they never re-resolve a slot by name. pbox down N without --instance also refuses a box another caller launched in the last hour (its PboxLaunchedBy tag, set from PBOX_CALLER, by default the host) unless --force.

Use only pbox (or scripts/parallel-box.sh, which is the same script) for this lifecycle. Terraform owns the durable half -- work volumes, security groups, key pair and one launch template per box -- while the instances themselves are deliberately not in state. That removes the old hazard entirely: an unrelated full apply can no longer propose terminating a box in the middle of a live job.

A box open to the internet, with public IPv6 for fcvm VMs

pbox up 3 kvm-192 --open   # prints:  ipv6:  <prefix>/80 (fcvm --network routed --ipv6-prefix <prefix>/80)
pbox ssh 3
sudo fcvm podman run --name web --network routed --ipv6-prefix <prefix>/80 nginx:alpine

--open gives the box a second security group, parallel-box-open (ohio-pbox.tf), that lets every protocol and port in from 0.0.0.0/0 and ::/0, and asks EC2 for one delegated IPv6 /80 on its primary interface. fcvm's routed mode gives each VM an address in that /80, so each VM can be reached from the internet on its own public IPv6. Everything is exposed, not only the VMs: the group applies to the box's own public IPv4 and IPv6 addresses, so any port anything on the box listens on is reachable by anyone. A plain pbox up never attaches the group. up prints the /80 and adds ipv6_prefix=<the /80> to its last line; pbox status shows it while the box runs.

--open launches only in the home AZ: the fallback sites' VPCs have no IPv6, so it is refused with --fallback and --regions (and when home has no capacity, exits 3 without offering them). It is refused for every command but up, and for a member's pbox. A box already running without it is kept as it is: pbox down N, then pbox up N FLAVOR --open.

On-demand GPU box

Small NVIDIA instances for measuring browser games on real graphics hardware; every dev box renders WebGL in software. There are four independent slots, so four GPU jobs can run at once. Slot 1 is gpu-box and is what every command means without a slot; slot N (2-4) is gpu-box-N. From nextjs-dev:

gbox up [N]       # launch slot N from its launch template: tries g4dn.xlarge, g5.xlarge,
                  # g6.xlarge, g4dn.2xlarge in each us-west-2 AZ until one has capacity
gbox status [N]   # type, GPU, and its on-box shutdown timer
gbox status all   # every slot
gbox ssh [N]
gbox ip [N]
gbox down [N]     # terminate that slot only; nothing on it persists

Each slot has its own launch template of the same name, which writes the slot's Name tag, so gbox never tags anything itself. Terraform owns the four launch templates, the security group (SSH in from nextjs-dev's Elastic IP only; out only to web, DNS and NTP, so the box cannot reach the fleet's NFS scratch) and the tag-scoped managed policy gpu-box-control on nextjs-dev-role; the instance is never in state. The grant launches only these templates, only those four types, only the template's root disk shape, and only terminates the four tagged slots. Each costs money only while it runs: the parallel-box watchdog terminates it after 30 minutes below 5% CPU (and without disk or network activity) and at gpu_box_max_hours (4) from launch, and the box arms its own shutdown timer as a backup. All slots share the us-west-2 "Running On-Demand G and VT instances" quota (L-DB2E81BA): four g4dn.xlarge need 16 vCPUs, and 32 covers four of the largest type (g4dn.2xlarge).

Temporary EBS for agents

Permanent fleet infrastructure remains Terraform-managed. As a deliberate runtime exception, development boxes have a narrow IAM policy for temporary working volumes:

region must be us-west-1 or us-west-2;
volume type must be encrypted gp3;
creation must include the tag DevEBS=true;
attach/detach is allowed only between tagged volumes and tagged dev instances;
deletion is allowed only for tagged volumes.

EBS volumes are availability-zone scoped. An agent must create the volume in the same AZ as its target instance, mount it safely by volume ID rather than a guessed NVMe name, and delete it when no longer needed. This permission does not make /mnt/io persistent.

Kids' Next.js environment

nextjs-dev hosts separate Unix, GitHub, Vercel, Claude, and Codex identities for Colton and Connor. It is intentionally on-demand after repeated Spot reclamations made the URLs unavailable.

Inside a project:

ndev

ndev derives a stable port, registers the hostname, and enables a systemd service. cloudflared dials out to Cloudflare, so there are no inbound web ports and next dev remains bound to 127.0.0.1.

Deleting a checkout unpublishes it. ndev@<label> has an ExecCondition that skips the start when the published directory is gone, leaving the unit inactive instead of restarting it. ndev-prune (root) then removes every project whose directory no longer exists: it disables the unit, deletes its env file and drop-in, and drops every registry row that carries its hostname or its directory (so a pinned alias in the other zone goes too), then rebuilds each affected zone, restarting a tunnel only if its config changed. A project whose directory still exists is never touched, and a pinned route whose directory is gone is not re-added at boot. ndev-register, ndev-prune and the setup script's registry writes share one lock (/var/lib/ndev/.lock). If a zone's rebuild or restart fails, prune keeps that project's records and retries the zone on its next run; if the directory has come back by then, the project is published again. It runs on every setup run, in setup-sync before the health check, and nightly from ndev-prune.timer; sudo ndev-prune runs it immediately. DNS needs nothing because each zone has a single wildcard record.

The zone follows the account, not a flag. colton and connor publish to https://<name>.cc-games.dev; ejc3 and skevh publish to https://<name>.dolphin-labs.dev. /usr/local/bin/ndev-zone is the single source of truth for user -> zone -> tunnel, and ndev-register refuses a hostname outside the caller's zone -- it runs as root through sudoers, so that check is a privilege boundary, not a convenience.

Each zone has its own tunnel, its own credentials, its own registry, and its own cloudflared@<zone>.service, so a bad ingress rewrite on one project cannot take the other project's URLs down. Access gates both: cc-games.dev through Google or one-time PIN against an email allowlist, dolphin-labs.dev through GitHub against membership of the dolphin-labs-hq organization. A separate service token supports non-interactive checks. One host there is public on purpose: maps.dolphin-labs.dev is Dolphin Maps production, served by the Worker dolphin-maps-prod, with its own record and an Access application that bypasses the gate for that host only (dolphin-labs-zone.tf). ndev cannot serve that name.

Two deliberate exceptions, both for the family board (ejc3/family, which shows the household's day on the living-room TV, an e-ink tablet and phones):

A pinned cross-zone hostname. family.cc-games.dev routes to 127.0.0.1:3722, which is ejc3's ndev@ejc3-family service. It is the one hostname on cc-games.dev that is not a colton/connor project: the board is built under ejc3 but its audience is the kids and the wall screens, who can pass Google sign-in and not the GitHub-org gate. It is declared in local.nextjs_pinned_routes (nextjs-user-data.tf) and written into the zone registry at boot, not published with ndev, so the zone-follows-account rule above still holds for everything anyone can type. Add to that list only for the same reason.
Wall-screen paths with their own Access application. On family.cc-games.dev only, /tv, /ink and /api/device/* are not behind the Google allowlist but behind cloudflare_zero_trust_access_application.family_wall_screens (cloudflare.tf), because a TV cannot sit through a Google login. Cloudflare lets a request through only from a device enrolled in this account's WARP client by someone on the allowlist (signed in through WARP, so the screen shows no login page), or from the home screen proxy, which adds the family-wall-screen-proxy service token (Secrets Manager family-wall-screen-proxy-access; it works on these paths and nowhere else). Behind that, the app still pairs each screen: an unpaired screen shows a short code that a signed-in person approves at /pair (which stays behind Google). /_next/*, the site's compiled assets, bypasses Access in a separate application because the phone pages share it. Everything else on the hostname, and every other cc-games.dev hostname, is still behind the Google allowlist. To audit: curl -I https://family.cc-games.dev/ and curl -I https://family.cc-games.dev/tv must both be a 302 to the Access login; /_next/static/... is expected to be a 200.

Once credentials exist and their units have been enabled, services and remote-control agents start at boot. A reboot restores the published URLs without another interactive login.

The containerized replacement (nextjs-ct)

nextjs-ct (nextjs-ct.tf, nextjs-ct-user-data.tf, scripts/nextjs-ct/) is being built next to nextjs-dev and takes its Elastic IP at the cutover; until then it serves nothing. On it every account, ejc3 included, lives in a container of its own and nobody works on the host:

Getting in is unchanged: ssh <user>@<box>. The host's sshd checks the person's own ~/.ssh/authorized_keys and relays the session into their container (nextjs-ct-enter), so shells, commands, scp, sftp, rsync, git and agent forwarding work as before. -L reaches only the person's own container (ssh -L 3000:10.88.N.2:3000 ...). An admin enters a container from the ubuntu account with sudo machinectl shell <user>@<user>.
Inside, the person is root (sudo, no password), installs what they like, and runs the same ndev, Claude, Codex and t-claude as before, plus the jumpboxes' pinned Terraform (for dolphin-aws's scripts/apply.sh). Their home is the host's /home/<user>, mounted in. Ports are the container's own: two people can both use 3000.
Publishing: ndev records the project in the container; nextjs-ct-ndev-sync on the host checks it and routes <label>.<zone> from that zone's cloudflared to <container address>:<port>.
AWS: aws and the SDKs work inside with no setup. The container's 169.254.169.254 is the host's nextjs-ct-imds, which hands out a session of nextjs-dev-role narrowed by a session policy: no container gets the tunnel credentials, the setup scripts or the SSM agent channel; only the owner's gets other roles (the Dolphin apply role), the hop key, the Google Docs token and the owner's boxes. A member's session also carries that member's managed policy (nextjs-ct-member-<user>), which admits their own wbox, pbox and gbox (member-boxes.tf) and nothing of anyone else's: their own two secrets, launches only from their own launch templates with their own Name tags, start, stop and terminate only their own, attach only their own volumes, pass only their own wbox role. Nor can a member read the output of the owner's wbox run (the wbox-output-<account> bucket).
The boxes: wbox, pbox and gbox are in every container and act on that person's own boxes. A member may forward a port to their own wbox or gbox through ssh (ssh -L 8443:<own wbox address>:8443 ...), and their container is refused the private addresses of every other member's wbox and gbox. A pbox is different: it is in Ohio, reached at its public address, and its security group admits SSH from anywhere, so another member CAN reach its SSH port. The boundary there is the per-member SSH key (member-box/<person>/ssh-key), which every other member's session is denied. Both the keys and the member wbox passwords are a boundary only once they have been rotated at the cutover (below): on nextjs-dev every account has sudo and the role reads all of them, so any member could have saved another member's.
The Macs: ssh mac (the MacBook) from ejc3's container only, and ssh colton-mac from colton's and ejc3's: the same Cloudflare route as every other box (mac-ssh-client.tf), each with that account's own Access token, which every other member's AWS session is denied (local.nextjs_ct_mac_clients).

What carries over at the cutover, and how: scripts/nextjs-ct/host/nextjs-ct-migrate (homes, published projects, crontabs, then the SSH host keys at the final pass). Some secrets are rotated at the cutover rather than before it, because on nextjs-dev every account has sudo and the role reads them, and on nextjs-ct a member reads only their own:

The members' box keys and wbox passwords, after the final copy and before anyone is let in (in the same apply as nextjs_ct_serving = true): with no member box running, terraform apply with -replace='tls_private_key.member_box["<person>"]' and -replace='random_password.member_wbox_admin["<person>"]' for each of colton, connor, colin and skevh, and on nextjs-ct sudo rm -f /home/<person>/.ssh/member-box (the copied homes hold the old private key, which pbox and gbox would otherwise keep using). Nothing is pushed to a box: every member box is a fresh instance per launch, takes the key and the password from the launch template and Secrets Manager at its first boot, and the templates update their default version.
The owner's wbox password (wbox/administrator-password), once everyone is in a container: terraform apply -replace=random_password.wbox_admin and then, from ejc3's container, wbox up and wbox run 'net user Administrator (Get-SECSecretValue -SecretId wbox/administrator-password -Region us-west-1).SecretString' (the owner's box keeps its disk and sets the password only at its first boot).

The cutover plan is in the pull request that added the container layer.

Dolphin SSH through Cloudflare

ssh.dolphin-labs.dev reaches this host's existing SSH daemon on 127.0.0.1:22 through the Dolphin tunnel. The existing wildcard DNS and GitHub Access application protect it: first sign in as a member of dolphin-labs-hq, then authenticate to SSH with your existing key. This does not grant any new Unix accounts or change direct public SSH/ET access. For the owner's Dolphin checkout, use the ejc3 account.

On a Mac or Linux client with cloudflared installed:

ssh -o 'ProxyCommand=cloudflared access ssh --hostname %h' \
  ejc3@ssh.dolphin-labs.dev

Add -i /path/to/your/private-key if the key is not already in your SSH agent/config. Cloudflare opens a browser for GitHub sign-in when your Access session needs renewal. The server's connector credential is never needed on the client. Plain SSH to this hostname without the proxy command will not work. Phone SSH apps that cannot run cloudflared need a separately configured supported network/client path; this route does not configure WARP or an iOS VPN. See the Cloudflare SSH client guide.

ndev-rebuild always retains this static Dolphin-only SSH route, and ndev-register reserves its hostname so a project cannot replace it. Other project routes and the kids' tunnel are unchanged. Local cloudflared ingress requires a connector restart, not SIGHUP; only a zone whose config changes is restarted, briefly disconnecting its existing tunnel sessions. An unchanged rebuild does not restart either connector.

Colton Games Cloudflare staging and previews

The colton-games-stage Worker is deployed from CoderColton/colton-games with Wrangler and OpenNext. Terraform owns the Worker envelope and its Worker-native Access boundary; Workers Builds owns versions, application assets, bindings, and deployments. This prevents an infrastructure apply from replacing an application bundle.

The existing Worker is adopted through the checked-in import block with both workers.dev and preview URLs still disabled, matching its current remote state. Read the plan and stop if it proposes creating, replacing, or deleting the Worker or changing its deployed code.

The checked-in Worker-native Access application uses a worker destination with no domain. That single application follows every request routed to this Worker, including its workers.dev hostname once enabled, immutable and branch previews, future Custom Domains, and future zone routes. It reuses the family email allowlist and the non-interactive Access service-token policy. The URL switches intentionally remain off until the application has been applied and a second clean plan confirms the policy attachment. Do not use hostname wildcard applications or a generic REST/local-exec bridge as a substitute.

The pinned fork manages the account's cc-games.workers.dev label, a dedicated deployment API token, the GitHub repository connection, separate staging and preview triggers, and their complete build-time environment maps. The ordinary cloudflare-tunnel-token is account-owned and cannot call Workers Builds. The aliased provider instead reads the raw user-owned control token ephemerally from cloudflare-workers-builds-control-token, so its value is not persisted in Terraform state; do not reuse, copy, or replace the ordinary Cloudflare credential.

Open Create Additional Tokens and choose Use template for that template, not the Custom Token builder: API Tokens Edit is unavailable in the Custom Token permission list. Keep the template's User → API Tokens → Edit permission (called API Tokens Write by the API), then add account-level Workers Builds Configuration Edit and Workers Scripts Read scoped only to account 12ea67fb7ced068de03f35c22688e436. Those permissions let Terraform configure Builds and mint the narrower Workers Scripts Write token that build jobs receive. Store the raw control-token string in the existing Secrets Manager container without JSON wrapping. Terraform creates the deploy token; do not make or paste a second deploy credential.

The other one-time prerequisite starts in Cloudflare: Workers & Pages → colton-games-stage → Settings → Builds → Connect → GitHub. The CoderColton repository owner must then authorize the Cloudflare Workers and Pages GitHub App for only CoderColton/colton-games; ejc3 has write access, not administration access, and cannot make the grant. The GitHub App page is a reference, not the primary setup entry point because a bare install can miss the Cloudflare account connection context. When authorization returns to Cloudflare, stop: do not select, save, or connect the repository and do not create build settings in the dashboard. Terraform owns those objects. It cannot perform the interactive GitHub grant, and the repository-connection API has no read/list endpoint or import path, so an out-of-band connection cannot be adopted safely. The managed resource is protected from destroy and its last confirmed UUID exists only in versioned, encrypted Terraform state.

The managed builds are:

Deployment	Branches	Build command	Deploy command
Production staging	`main`	`npm run cf:build`	`npm run cf:deploy:built`
Pull-request preview	all except `main`	`npm run cf:build`	`npm run cf:upload:built`

Use / as the root directory and Node 24 from the application's .nvmrc. Preview builds must remain limited to trusted branches because they inherit the staging Worker's bindings and secrets. stage.colton-games.com and the production Vercel serving path remain outside this change until the domain is deliberately moved to Cloudflare.

Roll this out in order; do not collapse the safety gates into one apply:

After signed provider 5.24.0 is visible in the Terraform Registry, update only its stale lock selection, then initialize without allowing any further lock changes:
```
terraform providers lock \
  -platform=linux_arm64 -platform=linux_amd64 \
  registry.terraform.io/ejc3/cloudflare
git diff -- .terraform.lock.hcl
terraform init -lockfile=readonly
terraform validate
```
The diff must change only the ejc3/cloudflare block to signed 5.24.0 hashes for both platforms. Commit that lock update. Do not use broad terraform init -upgrade; it can advance unrelated providers allowed by their ~> constraints.

Leave both colton_games_workers_builds_enabled and colton_games_worker_urls_enabled false. Save and review a targeted backend-versioning plan, then apply that exact plan:

terraform plan \
  -target=aws_s3_bucket_versioning.terraform_state \
  -out=/tmp/colton-games-backend-versioning.tfplan
terraform show -no-color /tmp/colton-games-backend-versioning.tfplan
terraform apply /tmp/colton-games-backend-versioning.tfplan
aws s3api get-bucket-versioning --bucket ejc3-terraform-state \
  --region us-west-1 --query Status --output text

Require the saved plan to contain only the versioning resource, require Enabled after applying it, then wait at least 15 minutes for S3 versioning to propagate.

After that wait, apply only the control-token container. This legitimate Terraform state write is also the versioning probe. Again, save and review the targeted plan, then apply that exact plan:
```
terraform plan \
  -target=aws_secretsmanager_secret.cloudflare_workers_builds_control_token \
  -out=/tmp/colton-games-control-token-container.tfplan
terraform show -no-color /tmp/colton-games-control-token-container.tfplan
terraform apply /tmp/colton-games-control-token-container.tfplan
aws s3api head-object --bucket ejc3-terraform-state \
  --key aws-infrastructure/terraform.tfstate --region us-west-1 \
  --query VersionId --output text
```
Require the saved plan to contain only the empty secret container. The backend object's VersionId must then be non-empty and not null or None. Stop if it is not: do not create a deployment token or repository connection without a verified versioned state write.
Keep both gates false and return to a full saved plan:
```
terraform plan -out=/tmp/colton-games-base.tfplan
terraform apply /tmp/colton-games-base.tfplan
terraform plan
```
The saved plan should contain only the account Workers subdomain and any other already reviewed base configuration from this change. It must not create Builds credentials, connections, triggers, or environment maps; update the Worker; change Access; or destroy anything. Apply the saved plan immediately and require the follow-up plan to be empty.
Create and store the raw control token, then have the CoderColton repository owner complete the repository-only GitHub App authorization.
Change only colton_games_workers_builds_enabled to true. Re-plan immediately. This stage may create only the narrow deploy API token, its Builds registration, and the repository connection. It must not create either trigger or environment map, update the Worker, or change Access. Apply, then require an empty follow-up plan.
Change only colton_games_worker_urls_enabled to true. The plan should contain one in-place Worker update plus the two triggers and their two environment maps. The trigger dependency guarantees the already-protected URL surfaces are enabled before a GitHub event can deploy. Apply, immediately run a fresh plan, and require it to be empty. Only then verify an unauthenticated request is denied and the existing Access service token succeeds.
While pull request CoderColton/colton-games#26 is still open, cause a synchronize event with a new commit and verify both its protected immutable preview and branch alias. A preview cannot exist while previews_enabled is false, so do not perform this check before step 7 or after merging the pull request.
Only after that preview succeeds, merge pull request #26 to main. Verify the resulting staging build and protected workers.dev deployment.

Every gate change is a reviewed, committed Terraform change. Do not disable either gate after its protected resources exist; prevent_destroy is intended to stop that rollback.

Games Worth Watching: the App Store Connect key for TestFlight

Terraform creates only the container games-worth-watching/app-store-connect (us-west-1, dev-ai-services.tf); TestFlight uploads from nextjs-dev stay unavailable until it holds a key. Once, after the apply:

In App Store Connect, Users and Access, Integrations, App Store Connect API, create a team key with the Admin role (cloud signing creates the distribution certificate, which App Manager may not). Download its .p8 file (Apple names it AuthKey_<KEY_ID>.p8) into the current directory, and note the Issuer ID, the Key ID and the developer Team ID.

On a jumpbox, put the four values as one JSON object without echoing them, then delete the file:

read -rs ISSUER; read -rs KEY_ID; read -rs TEAM
set -o pipefail
jq -n --arg i "$ISSUER" --arg k "$KEY_ID" --arg t "$TEAM" --rawfile p "AuthKey_${KEY_ID}.p8" \
  '{ISSUER_ID: $i, KEY_ID: $k, TEAM_ID: $t, PRIVATE_KEY: $p}' \
  | aws secretsmanager put-secret-value --region us-west-1 \
      --secret-id games-worth-watching/app-store-connect --secret-string file:///dev/stdin \
  && rm "AuthKey_${KEY_ID}.p8"
unset ISSUER KEY_ID TEAM

The .p8 file is the only copy (App Store Connect does not offer it again), so it is removed only after the upload succeeds; on a failure, fix the cause and run the pipeline again.

Only administration, nextjs-dev's role, fcvm-metal-arm's instance (dev-server-role pinned by ec2:SourceInstanceARN, so fcvm-metal-x86 cannot) and dolphin-labs-testflight (dolphin-labs CI's TestFlight uploads: GitHub OIDC for testflight.yml on main only, dolphin-labs-testflight.tf) can read it. Every account on nextjs-dev has sudo, so treat the key as readable by that whole box (accepted by the owner, 2026-10-17). Revoke it in App Store Connect and put a new one the same way to rotate.

Astryx Swift: self-serve TestFlight invitations

https://astryx-join.dolphin-labs.dev (astryx-join.tf) is a public form: anyone types an @meta.com address, passes Cloudflare Turnstile, and the Worker adds the address to the Astryx Swift beta group "Meta (self-serve)" through the App Store Connect API. There is no sign-in (an Access gate would cost a Zero Trust seat per person), so an address is not proven when it is typed; the invitation can only be accepted from that inbox, and 5 requests per IP an hour and 500 a day bound abuse. Nothing is built until the key's secret exists; the first apply after that builds the whole host. Once:

In App Store Connect, Users and Access, Integrations, App Store Connect API, Team Keys, create a key named for this use with the App Manager role (the lowest that manages testers; not Admin, and never the Games Worth Watching key). Download its .p8 file (AuthKey_<KEY_ID>.p8) into the current directory and note the Issuer ID and Key ID.

With credentials for the astryx account (from the owner's container, the terraform_role in SSM /infra/astryx; from a jumpbox, the account's OrganizationAccountAccessRole), create the secret with the three values as one JSON object without echoing them, then delete the file:

read -rs ISSUER; read -rs KEY_ID
set -o pipefail
jq -n --arg i "$ISSUER" --arg k "$KEY_ID" --rawfile p "AuthKey_${KEY_ID}.p8" \
  '{ISSUER_ID: $i, KEY_ID: $k, PRIVATE_KEY: $p}' \
  | aws secretsmanager create-secret --region us-west-1 --name astryx/testflight-asc-key \
      --description "App Manager key for Astryx Swift TestFlight invitations (astryx-join.tf)" \
      --secret-string file:///dev/stdin >/dev/null \
  && rm "AuthKey_${KEY_ID}.p8"
unset ISSUER KEY_ID

Apply. Terraform puts an administration-only read policy on the secret and builds, in order, the record, the KV namespace for the limits, the Turnstile widget (managed, this host only), the Worker with the key and the widget's secret as its secrets, the route, and last the Access bypass for exactly this host. The widget needs Turnstile write on the provider token cloudflare-tunnel-token (AGENTS.md, "Astryx Swift TestFlight invitations").

Rotate: create a new App Manager key, put-secret-value the same JSON to astryx/testflight-asc-key (astryx account), apply, then revoke the old key. Stop: revoke the key in App Store Connect (invitations fail at once); to remove the host too, aws secretsmanager delete-secret --force-delete-without-recovery --secret-id astryx/testflight-asc-key and apply.

Limits: Apple allows 10,000 external testers per app; a new build version needs Beta App Review before the group's testers get it.

Colton Games accounts: sign-in, site admins and push

colton-games-accounts.tf holds what the site's accounts features need outside the games repository. Sign-in is optional and stays off in an environment until that environment has AUTH_GOOGLE_ID, AUTH_GOOGLE_SECRET and AUTH_SECRET. Production and non-production share no credential, so a session, an admin grant or a push subscription made in one means nothing in the other.

Four JSON secrets in us-west-1 hold the values that come from outside. Terraform creates the containers; the owner puts the values.

Secret	JSON keys (all strings)	Readers besides administration	Written to Vercel
`colton-games/prod/auth`	`AUTH_GOOGLE_ID`, `AUTH_GOOGLE_SECRET`	none	Production
`colton-games/prod/push`	`NEXT_PUBLIC_VAPID_PUBLIC_KEY`, `VAPID_PRIVATE_KEY`, `VAPID_SUBJECT`	none	Production
`colton-games/nonprod/auth`	as `prod/auth`	dev-server-role, nextjs-dev-role	Preview
`colton-games/nonprod/push`	as `prod/push`	dev-server-role, nextjs-dev-role	Preview

AUTH_SECRET is in none of them: Terraform generates one per environment and writes it to Vercel only, as it does for MP_COOKIE_SECRET. SITE_ADMIN_EMAILS is in none of them either (see "Site admins" below). Every variable is sensitive in Vercel, each has exactly one target, and nothing goes to Development. NEXT_PUBLIC_VAPID_PUBLIC_KEY is the one name the browser sees, because a push subscription needs the public key in the page.

colton_games_accounts_ready in the same file is the gate. Reading a secret that has no value fails every plan of this repository, so Terraform reads only the secrets the gate names and writes only their variables. It starts empty. Change its default in a commit, not with -var: a later plan without the flag would propose deleting the variables.

Site admins. SITE_ADMIN_EMAILS, the Google addresses of the site's admins, comes from the colton_games_site_admins key of the people/addresses secret (people.tf): one list for prod (written to Production) and one for nonprod (written to Preview), with that environment's auth secret and under the same gate. The repository holds no address. To change them, put the secret's JSON again with the new lists under that key (the same put-secret-value command as for people/addresses above); a plan on any jumpbox reads it, so there is no per-machine file. From there the addresses exist in Terraform state and in Vercel, and nowhere else: not in a repository, and not in a colton-games secret's JSON. The rules:

Write each address in lower case, one per entry, with no space. That is how the site compares them, and it silently drops an entry it cannot read; here the plan refuses it.
No duplicates, at most 20 per environment.
[] means that environment has no site admins. No variable is written for it.
A placeholder is refused where it would be written. If an environment's auth secret is in the gate and its list still holds an address at a reserved example domain, the plan fails. On the site a non-empty list switches off the admin panel's password, so a list nobody owns would leave nobody able to open the panel.
Every machine that plans needs the same entry. One without it sees the placeholders and is refused by the rule above, so it cannot replace the real list by accident.
A refusal names the variable and never an address.

ACCOUNT_SAVES is the site's switch for keeping a signed-in player's saved games in their account. The site reads exactly on; unset is off. It is not a secret, so it has no container and is not sensitive. A second gate, colton_games_account_saves_targets, names the Vercel targets that get it, each on its own. It started empty and its default now names production, the one to name: the site trusts no preview address for saves, so the switch does nothing on Preview. A target cannot be named before its auth secret is in the first gate.

The owner's steps, in order:

Merge with the gate empty, then plan and apply on a jumpbox. Expect 13 to add (four secrets, four secret policies, one IAM policy with two attachments, two generated AUTH_SECRETs), none in Vercel, and an empty follow-up plan.
Create two Google OAuth clients of type "Web application", one per environment. Give each the redirect URI https://<hostname>/api/auth/callback/google for every hostname that serves its environment:
- production: cc-games.org, and colton-games.vercel.app because Vercel's own name also serves. The other production names redirect to cc-games.org before sign-in starts.
- non-production: each dev-box site (<name>.cc-games.dev) and http://localhost:<port> for a local run. A preview needs none. Google accepts no wildcard and a preview's address is new with every deploy, so on a Vercel deployment the site offers sign-in only at its registered hostnames (the two production ones above): a preview shows no sign-in button and its /api/auth/* routes answer 404.
A Google Cloud project whose consent screen is in testing admits only its listed test users, so production's must be published.
Put each auth value from a JSON file only you can read, then remove the file. The JSON has exactly two keys, AUTH_GOOGLE_ID and AUTH_GOOGLE_SECRET; one that still has a SITE_ADMIN_EMAILS key is refused by name at step 5.
```
aws secretsmanager put-secret-value --region us-west-1 --secret-id colton-games/<env>/auth \
  --secret-string file://<a 0600 JSON file> && shred -u <that file>
```
Check the colton_games_site_admins key of the people/addresses secret holds the real addresses for prod and for nonprod, or [] for an environment without admins.
Set the gate's default to ["prod/auth", "nonprod/auth"] in a commit, then plan and apply. Expect eight vercel_project_environment_variable.colton_games_accounts to add (AUTH_SECRET, AUTH_GOOGLE_ID, AUTH_GOOGLE_SECRET and SITE_ADMIN_EMAILS, once for production and once for preview; one fewer for an environment whose list is []) and nothing else. The plan refuses, naming the secret and key or the variable and never the value, when a key is missing, a value is in the wrong key, both environments were given the same OAuth client, or an admin list is malformed or still a placeholder.
Redeploy production. A deployment reads its environment when it is built; previews get theirs on their next build.
Verify:
- terraform output colton_games_accounts_vercel_env lists the four names under each target, and a new plan is empty.
- curl -s -o /dev/null -w '%{http_code}\n' https://cc-games.org/api/auth/providers prints 200. It prints 404 while sign-in is off.
- Signing in on production works. An account on the list is a site admin there, and any other account is not.
- On a dev box, aws secretsmanager get-secret-value --region us-west-1 --secret-id colton-games/prod/auth --query Name --output text fails with AccessDeniedException, and the same command for colton-games/nonprod/auth prints the name.
Account saves, when the site has them and only after step 7 passes for production:
- Nothing is applied by hand. The saves migration (supabase/migrations/20261004000000_account_saves.sql in the games repository) is applied by games-mp-migrate when the games change merges to main. Before the switch, check it arrived, or the site offers saves with no tables behind them: aws dynamodb get-item --region us-west-1 --table-name games-mp-releases --key '{"id":{"S":"schema#main"}}' shows siteNewest at that file or a later one.
- The default of colton_games_account_saves_targets is ["production"] (set in a commit, as any later change to it is). Plan and apply. Until it has been applied, expect one to add, vercel_project_environment_variable.colton_games_account_saves["production"], and nothing else.
- Redeploy production.
- Verify: terraform output colton_games_accounts_vercel_env now lists ACCOUNT_SAVES under production only, and curl -s -o /dev/null -w '%{http_code}\n' https://cc-games.org/storage-bridge prints 200. It prints 404 while saves are off.
To switch saves off again, take the target out of the default and apply, then redeploy.
Push, when the site has it: run npx web-push generate-vapid-keys once per environment, put colton-games/prod/push and colton-games/nonprod/push (VAPID_SUBJECT is an https:// address of the site), add both names to the gate and repeat steps 5 to 7.

Later changes: a new admin list is an edit of terraform.tfvars, one apply and a redeploy. A new OAuth client is one put-secret-value of the whole JSON, one apply and a redeploy. terraform apply -replace='random_password.colton_games_auth_secret["production"]' makes a new AUTH_SECRET and signs everyone out of that environment. The values pass through Terraform state, which only administration can read, like the other games secrets.

Not managed here: the Google OAuth clients (Google has no API for them), a stable hostname for preview sign-in, and the database. The Supabase project came from the Vercel integration; Production and Preview already hold its URL, its server key and SKYHOOK_LEADERBOARD_ENVIRONMENT. The accounts migrations in the games repository need no step here: games-mp-migrate applies every file under its supabase/migrations when it merges to main, before that release is promoted (docs/games-multiplayer.md, "Site migrations"). A dev box reads colton-games/nonprod/* itself and makes its own AUTH_SECRET.

GitHub Actions and package infrastructure

The workflow_job webhook launches one-time Spot runners from prebuilt ARM64 or x86 AMIs. Labels select the architecture; the launcher tries several metal families when capacity is scarce, moving a family that just failed for capacity to the back of that order. Each new instance carries RunnerRegistrationProtocol=ddb-v2. After GitHub configuration, bootstrap validates the exact identity in .runner, conditionally records State=registered under its instance ARN in DynamoDB, and starts the service only after that identity is confirmed. Registrations are ephemeral, one job each, but a host whose job succeeded can take another: it waits three minutes, and a completed event with queued work behind it, or a later queued event, claims the host through a conditional update of its row and brokers it a fresh instance-bound credential. A host whose job failed, or that nobody claims, powers off.

Cleanup runs every five minutes. A registered ddb-v1 or ddb-v2 runner is checked by its exact GitHub runner id, and one with no row is reaped after its lease through a conditional State=reaping claim on that row, so a bootstrap arriving at the same moment finds the row taken and does not start the service. A missing row beside evidence that the runner did register (GitHub still lists it, a RunnerSeenAt stamp, or a renewed lease) is a lost row, and the instance is held to the age ceiling rather than reaped. Instances launched before the tag existed keep the roster-based rules: a renewed lease or a RunnerSeenAt stamp holds them on roster absence, and a readable roster that has never listed one expires its lease. Cleanup also reaps stalled launches, terminates jobs running for more than three hours, counts queued jobs to retry scale-up, and deregisters, then terminates, an idle registered runner whose architecture has nothing queued.

A runner instance lives at most 13 hours 30 minutes, checked on the five-minute poll, so the observed maximum is that plus one interval. Past 12 hours it drains: GitHub-idle means terminate now, and a job already in flight is left to finish. 13h30m is the hard ceiling, computed from the instance's launch time alone and applied whatever the runner is doing and whatever GitHub says, so a wedged host cannot outlive it. A ceiling termination that never saw the runner idle logs HARD-CEILING KILL; that job will appear on GitHub as "the self-hosted runner lost communication with the server", which is otherwise indistinguishable from an AWS Spot reclaim.

The EC2 hard-ceiling pass completes before cleanup reads the PAT or calls GitHub, so GitHub latency cannot defer the bound.

Terraform owns both halves of that webhook: it generates the HMAC secret, sets it on the ejc3/fcvm hook through the integrations/github provider, and passes the same value to the Lambda. There is no operator-held copy to fall out of sync -- which it did, silently and completely, until 2026-08-07. That ownership takes effect at the one-time hook import and the first apply after it (see Prerequisites); a state that has not run them yet still has the pre-Terraform secret live on the hook.

The maximum is eight arm64 and four x86 runners (local.runner_max_by_arch), counted as runners GitHub can actually hand a job to rather than instances that merely exist, so a wedged box cannot hold a slot. Each scale-up decision emits a GitHubRunners metric and a structured log line. Two alarms watch them: runner-scale-up-starved fires when queued work goes unserved for two hours (a wedged pool, or one genuinely that far behind — a single poll cannot tell them apart, because a host that wedges mid-job keeps reporting busy to GitHub), and runner-zero-online fires within ten minutes when jobs are queued and nothing is online or booting to take them. Runner roots and instance-store data are disposable. See GITHUB-RUNNERS.md for the webhook, AMI, lease, health, and cleanup internals.

Runner-image publication is a separate privilege from running CI. The replacement github-actions-ami-builder role trusts only the ejc3/fcvm environment runner-ami-publish. That environment permits only the main branch, requires approval from ejc3, and disables administrator bypass. Collaborators can still launch ordinary CI; publishing a shared runner image requires the owner's explicit approval of the exact commit. ejc3 may approve their own dispatch. Do not approve a publication without reviewing the workflow and all build inputs at that commit.

The environment settings are versioned in .github/runner-ami-environment.json. They are applied with the operator's existing GitHub CLI login; Terraform's webhook-only PAT cannot administer repository environments, and personal GitHub credentials must not be copied into Terraform state or fleet secrets. To recover this GitHub prerequisite, create/update the environment, then add its sole branch policy only if the read-only policy listing does not already contain it:

gh api --method PUT repos/ejc3/fcvm/environments/runner-ami-publish \
  --input .github/runner-ami-environment.json
gh api repos/ejc3/fcvm/environments/runner-ami-publish/deployment-branch-policies
# Only when the main branch policy is absent:
gh api --method POST repos/ejc3/fcvm/environments/runner-ami-publish/deployment-branch-policies \
  -f name=main -f type=branch

Read back the environment and branch policies before creating/enabling the AWS role. There must be no other allowed branch/tag, reviewer, or bypass. Credential migration has separate deployment gates; merging source is not evidence that a gate is live:

Create the replacement builder role/profile and own-instance bootstrap grant.
Deploy the backward-compatible runner controller and expiry cleanup. Its iam:PassRole grant is independently restricted to the existing runner role and EC2, with explicit denies for other roles/services. Verify the deployed code and full IAM allow/deny cases before changing bootstrap. The controller-only stage keeps the PAT-reading document and its instance-role permissions intact.
Publish this source's instance-bound bootstrap only after controller acceptance, then verify a real trusted CI registration/job and drain instances booted with the old script. Non-secret IAM fixtures test authorization, not registration or job execution. New hosts register for one job, delete their bootstrap credential before service startup, and power off when the service ends; EC2 then terminates them. At this intermediate stage the old PAT grant and broad SSM attachment remained; the separately verified cutoff below removes them. The runner release and asset checksums are pinned and automatic updates disabled; review each release bump before GitHub's 30-day update deadline (immediately for required critical fixes), following the runner update procedure. A daily main-only Runner Release Freshness workflow fails when a newer stable version exists; it never upgrades or deploys automatically. Keep its scheduled runs and GitHub failure notifications enabled.
Retire runner PAT reads and the broad SSM attachment only after those tests. Narrow the remaining controller EC2 launch resources after all launched resources carry the required tags. The old CI authority is retired separately, only after the protected fcvm AMI workflow has adopted the replacement builder role/profile.

Keep each gate in a reviewed Terraform change with a fresh plan. The optional cleanup scans only bootstrap metadata/tags after every hard-ceiling termination attempt and deletes only positively identified, expired controller credentials. Its work is bounded; repeated truncation or held/error records require investigation, not a claim that every orphan has expired out of AWS. See controller-first migration for the protocol and failure behavior. An additive apply alone does not close the old PAT or CI escalation paths.

This source removes the September 12 temporary runner IAM fixtures. Apply cleanup only after the before/after IAM checks and real post-cutoff trusted job pass, including own-token deletion before job startup and automatic host termination. Require a fresh full plan destroying only the two small canary hosts and their two non-credential parameters, then verify both instances, their disposable roots, and parameters are gone. No live runner, role/profile, Lambda, network, backup, or real credential is removed. RemoveAfter=2026-09-13 was a reminder, not automatic expiry; a merge does not stop billing. See the exact fixture gates and cleanup scope. The checker and result tests remain for later reviewed acceptance windows. Cleanup was applied on September 12 and independently verified at 19:09 UTC: both fixture instances terminated, their two disposable roots and interfaces disappeared, and both non-credential parameters were absent. No development disks were removed.

The repository also manages:

a private CodeArtifact npm repository in us-west-2;
an owner-approved GitHub OIDC role for runner AMI builds;
retired shared main/staging CI identities with explicit denial of all AWS actions;
credential-free Terraform validation on pull requests, main pushes, a daily schedule, and manual dispatch.

The historical drift.yml now runs Terraform Validation: formatting, locked provider installation with -backend=false, configuration validation, and CI boundary tests. It does not request an OIDC token, assume an AWS role, or read state or secret payloads. Full Terraform plans run only on an administration jumpbox. Even a read-only plan loads credential-bearing state and configures privileged Cloudflare/GitHub providers; calling that permission set read-only would not prevent administrator access through those secrets. A green validation check is not evidence of zero live drift.

CI identity retirement was separate from runner bootstrap and controller hardening. The September 12, 2026 IAM deployment (2968911, PRs #104 and #105) removed the runner's reusable PAT access and broad SSM attachment, narrowed controller launch and runner IPv6 permissions, closed metal-host account-wide command-output reads, and retired the unused predecessor dev role with an explicit Deny-all. The exact rendered policies passed AWS allow/deny simulations; real before/after EC2-role canaries proved own-versus-other SSM and DynamoDB access, and the post-cutoff PAT simulation changed from allowed to explicit Deny. Dev SSH/SSM remained healthy. The real post-cutoff trusted ARM64 job then passed at 18:43:15 UTC: the controller launched it under the narrowed policy, its own role assigned IPv6 and deleted the one-use credential at 18:28:49 before job startup at 18:28:52. The guest powered itself off; EC2 reported Client.InstanceInitiatedShutdown and terminated it by 18:49:37, with its exact root volume and ENI absent. This is actual post-cutoff acceptance, not only simulation.

Bootstrap, authentication, and convergence

Large instance setup scripts are rendered by Terraform and versioned in S3. Metal boxes run dev-selfupdate.service at boot: it compares the architecture script checksum and replays the complete bootstrap only when the published script changed. Inspect:

cat /var/lib/dev-selfupdate/status
sudo less /var/log/dev-selfupdate.log

Do not casually run dev-selfupdate.sh to deploy one small edit; it reapplies the whole machine configuration. Existing personal gh auth login state is preserved. The github-pat-ejc3 secret provides a fresh-box fallback and a separately scoped credential for claude-code-sync; it must never overwrite a person's GitHub login.

Next.js users authenticate independently. Jumpboxes intentionally leave interactive Claude/Codex login to the operator rather than storing those device credentials in Terraform.

The distributed accounts' agent instructions are agents-md.tf (AGENTS.md from ejc3/agents-md) and agent-local-md.tf (~/.claude/local.md, which is kept out of every repository). Each local.md is the administration-only secret agent-local-md/<machine>/<account>, gated by local.agent_local_md_ready. On a cold start, apply once with its gate false (that creates the secret), put its value from a 0600 file on a jumpbox, then set the gate to true and apply again. To change one later, put the new value and apply:

# from the box's live file, copied to a 0600 file here
umask 077; f=$(mktemp); ssh ... 'cat ~/.claude/local.md' > "$f"
aws secretsmanager put-secret-value --region us-west-1 --secret-id agent-local-md/fcvm-metal-arm/ubuntu \
  --secret-string "file://$f"

Codex's ~/.agents/skills mirrors ~/.claude/skills by a manual step (AGENTS.md, "Agent instructions come from ejc3/agents-md"): nothing here installs the skill checkouts, so nothing installs the mirror either.

The administration agent's own credentials (github-admin-tokens.tf) are containers that only the jumpbox role can read. Terraform creates them empty and they are filled once by hand, in the owner's browser session, because neither GitHub nor Vercel creates the first such credential by API. On a cold start, after the containers exist:

github-app/aws-admin: create the GitHub App from a manifest (GitHub's app-manifest flow: post the manifest to https://github.com/settings/apps/new, click Create, exchange the returned code with POST /app-manifests/{code}/conversions), store {app_id, slug, client_id, private_key, webhook_secret, installations} as JSON, install the app on each owner with all repositories, and record the installation ids (GET /app/installations with the app's JWT). scripts/github-app-token.py OWNER then mints tokens.
github-admin-token/<owner>: a fine-grained token per owner, all repositories, with the permissions listed in AGENTS.md; the longest expiry the owner allows.
vercel-admin-token: a full-account token from the owner's Vercel account settings (Tokens, scope Full Account).

Put each value from stdin or a 0600 file, never on a command line:

read -rs T; printf %s "$T" | aws secretsmanager put-secret-value --region us-west-1 \
  --secret-id vercel-admin-token --secret-string file:///dev/stdin; unset T

dolphin-maps' loops in Lambda (dolphin-maps-loop.tf) need their own GitHub App before they can run, then a switch-over in a fixed order so the loops never run in both places:

The app. Create dolphin-maps-loop from a manifest the same way: no webhook, repository permission contents: write only, installed on dolphin-labs-hq with only dolphin-maps selected. The conversion answers with id, slug and pem; the secret takes them renamed, as dolphin-maps-loop.tf documents: jq '{app_id: .id, slug: .slug, private_key: .pem, installations: {"dolphin-labs-hq": <installation id>}}' over the 0600 file holding the conversion, piped into put-secret-value for github-app/dolphin-maps-loop (stdin, never a command line). Stored as returned, the loop cannot mint a token and every pass fails. The loops commit straight to dolphin-maps' main, which has no ruleset today; if it ever gets one, add this app as a bypass actor first or every pass fails.
The first deploy. Set dolphin-maps' repository variables from output dolphin_maps_loop and run its deploy-loop.yml on main; it uploads the bundle and updates the three functions' code (the handler stays index.handler). Invoke each once by hand and read its journal: {"mode": "sweep"} any time, {"mode": "refit"} any time (it reads no page; its journal record is the refit's log, and its duration is the number to check against the 12-minute guard deadline), {"mode": "live"} only inside a route's morning window (weekdays 06:00-08:55 Los Angeles time): outside one the live handler returns before reading a secret or writing a record, so it proves nothing.
The switch-over, back to back: set dolphin-maps' repository variable LOOPS_RUN_IN_AWS to true (its scrape-live.yml and scrape-weekly.yml then stand down), then set dolphin_maps_loop_enabled = true here and apply (the three schedules and the not-running alarms come on). To go back, the reverse order: the gate false and apply first (the schedules stop), then the variable false (the GitHub loops resume).

Every setup script also gives an account that has no user-level agent instructions a starting file (user-agents.tf): ~/.codex/AGENTS.md, which Codex reads, with ~/.claude/CLAUDE.md, which Claude Code reads, as a symlink to it. The text is scripts/user-agents.md and nowhere else. user-agents-seed only creates: an account that already has either file is left exactly as it is, so a re-run changes nothing and an edit made on the box is kept. Changing scripts/user-agents.md therefore reaches new accounts only. It applies to ubuntu on the metal boxes and admin boxes and to every account on nextjs-dev, and arrives with the setup script: at first boot on a fresh box, through setup-sync on nextjs-dev, at the next boot on a metal box, and on an admin box only when its setup script is re-run. Once a box has the script, an account can also seed itself by running user-agents-seed.

Backups, safety, and cost controls

AWS Backup covers the ARM and x86 roots, the Next.js root, the original jumpbox home volume, and the jumpbox-2 root with daily, weekly, and monthly retention.
The parallel box's persistent /mnt/work volume is protected from Terraform destroy but is not backed up. prevent_destroy is not a backup; keep important results elsewhere.
The legacy primary vault keeps its governance Vault Lock. Existing primary and original us-east-1 history retain their lifecycles; neither receives new scheduled fleet copies. The active recovery pipeline below replaces the old staging-copy path without replacing live disks or deleting that existing history.
Daily cost reports, AWS Budget notifications (daily-cost-alert, all services, $200/day; ec2-daily, EC2 compute + EC2-Other, $150/day), runner-count/age alarms (from the GitHubRunners metrics the runner cleanup publishes every 5 minutes, with runner-cleanup-silent when it stops), the per-repo too-many-app-runners-<label> alarms, the runner-scale-up-starved alarm, and instance-status alarms publish through SNS/email. Account billing alerts are off, so nothing here reads AWS/Billing.
The original jumpbox's protected home volume and the fully reproducible jumpbox-2 provide two administration recovery paths.
Terraform state is encrypted in S3 and locked with DynamoDB.

The active backup pipeline keeps one new long-term history, in the recovery account's ejc3-backup-recovery logically air-gapped vault in us-east-1, separate from the source account and region. It is immediately compliance-locked and uses an AWS-owned key. The already-created empty west1 air-gapped vault remains untouched and receives no new copies. Empty vaults have no backup-storage charge.

fleet-backup-recovery copies only the five configured fleet volumes. Unencrypted ARM, x86 and jumpbox-home snapshots go directly to the final vault. The aws/ebs-encrypted Next.js and jumpbox-2 roots first copy to ejc3-backup-dr-cmk in the main account's us-east-1 region, then across accounts within that region. Live disks are not migrated. Final retention uses the original UTC backup date: the first of a month gets 365 days, other Sundays 30 days, and other days 7 days. This does not depend on temporary-copy TTLs.

Daily backup plans write to the unlocked ejc3-backup-processing vault with no expiration while a copy is pending. Only confirmed final copies permit source cleanup. The controller keeps the latest CMK checkpoint for each encrypted root until a verified successor exists: deleting every checkpoint would force expensive full transfers again. Older checkpoints are removed only with complete copy lineage. Recovery-point deletion is granted only by those two temporary-vault policies. AWS Backup also forwards the underlying EC2 deletion using the controller's credentials; that permission requires Backup in aws:CalledVia, our snapshot owner and pipeline tags. Direct EC2 deletion remains explicitly denied. The two untagged bootstrap CMK seeds were removed after verification; their exact-snapshot exceptions are disabled by the completed cutover. The controller cannot delete legacy history or final recovery points. Copy failures retain pending data and raise alarms rather than silently expiring it. Stalled cleanup costs storage.

The September 8, 2026 cutover followed five successful detached-volume restore tests (including validation and test-volume removal), verified final-copy lineage for all five new captures, and actual removal of all seven disposable processing/bootstrap snapshots. Both current CMK checkpoints, ten final points, and all 97 legacy points and their lifecycles were preserved. The two fleet selections now use the daily processing plans; the old plans remain for history but no longer select fleet volumes. Both rollout gates stay true. Monthly restore tests remain scheduled for the eighth at 12:00 UTC; the past one-off definitions are retained only for audit.

For a genuinely new deployment, cold bootstrap must start with both cleanup and selection-cutover gates false. Do not reset the gates on this deployed pipeline. Bootstrap leaves the existing daily/weekly/monthly plans and their selections unchanged; it seeds from their latest recovery points and has no recovery-point deletion permission. Roll out with fresh full Terraform plans in this order:

Apply the new destination/controller with both gates false. Require recent COMPLETED final points for all five exact source ARNs, both required CMK checkpoints encrypted with alias/fleet-backup-dr, and completed job-to-point lineage.
Set backup_initial_capture_at in backup-restore-canary.tf to a future UTC minute and apply the one-off processing capture. It backs up the same five volumes into the processing vault with no expiration and the required provenance tag, without moving either existing backup selection. Keep both cleanup and cutover gates false.
Set backup_initial_restore_at in backup-restore-canary.tf to a UTC minute at least 30 minutes in the future. Once step 1 has verified all five initial final copies, steps 2 and 3 may be applied together and run in parallel: restoring from the final vault does not depend on the new processing capture finishing. These are date/year crons, not recurring tests. The restore plan selects the latest completed point per volume, which may be a newer copy arriving during capture; record the actual recovery-point ARNs used by its jobs and verify their copy lineage.
Before enabling any recovery-point cleanup, require both independent checks: the five new processing captures have exact COMPLETED final copies and both CMK successor checkpoints; and all five initial restore jobs are COMPLETED, validation SUCCESSFUL, with test resources successfully deleted. Metadata verification does not prove guest data or boot integrity. Inspect copy/restore logs; do not infer success from Terraform apply. If a restore attempt fails, repair the evidenced cause, retain its job history, and set a new future backup_initial_restore_at for the same one-off plan. The controller checks a current-attempt job for every configured source volume. Rescheduling does not clear a preceding failure: the new job must complete, validate, and delete its test resource first. Missing or unfinished current jobs alarm after the one-hour start window plus one hour of propagation grace. Older jobs still receive validation/cleanup, and their failures remain in AWS job history and logs. Require all five jobs from the new attempt to pass.
Set only backup_recovery_cleanup_enabled=true and apply the two temporary-vault cleanup grants and controller update. Existing selections must remain unchanged. Verify the five processing captures and superseded CMK checkpoints actually disappear, not merely that deletion returned HTTP 200; keep each newest checkpoint. A permission failure must retain data, not lead to a broad identity-policy deletion grant. AWS can return HTTP 200 yet leave a point EXPIRED when its forwarded EC2 call fails authorization. Inspect the recovery point's status message and CloudTrail; after repair, the controller retries only expired points with the same complete final-copy lineage (and a strictly newer verified successor for CMK checkpoints). An expired processing retry also checks its provenance tag. Errors remain visible until the point disappears. Verify underlying snapshot absence as well.
Only after that cleanup acceptance succeeds, set backup_recovery_cutover_enabled=true and apply the two changed selections plus the controller's acceptance handoff. Their preconditions reject cutover while cleanup is disabled; replacement creates the new selection before removing the old one. This verified handoff retires the one-off health check so eventual AWS job-history expiration cannot create a false alarm; monthly per-volume restore checks and historical test-volume cleanup continue. Verify an EBS copySnapshot event reports incremental copying on a subsequent cycle before claiming measured incremental-transfer savings. Keep the past one-off capture and restore definitions for audit; their explicit years prevent recurring test jobs.

Main-account us-east-1 EBS copySnapshot completion events are retained for seven days in CloudWatch Logs /aws/events/fleet-backup-copy-snapshots. Correlate each event's source and destination snapshot IDs with Backup copy-job lineage before using its detail.incremental value as evidence for the CMK checkpoint leg. This log does not recover old events or prove the service-owned final air-gapped storage leg.

Existing recovery points keep their original lifecycles and age out normally; switching selections does not shorten them. Never bulk-delete old history. Preserve the unmanaged fcvm-backups vault's two indefinitely retained snapshots. Do not add compliance locks to the temporary processing/checkpoint vaults. Final-vault immutability is not absolute protection against AWS account closure.

The recovery plan fleet_monthly_detached_ebs runs on the eighth at 12:00 UTC. It restores encrypted, detached test volumes only: no instance launch, volume attach, or administrator role. Both initial and monthly tests explicitly use the recovery account's us-east-1 alias/aws/ebs key: copied restore metadata can still reference the original source-account key. This changes only test volumes, not live disks or the final vault's AWS-owned encryption. Validation checks metadata/isolation, not filesystem contents or bootability. The restore role also needs kms:ReEncryptFrom on the final vault's exact service-owned source key, through EC2 in the recovery region. Without it, EC2 initially returns a volume ID but the volume disappears when asynchronous key authorization fails. The pinned provider does not expose the air-gapped key, and its standard-vault data source rejects this vault type. backup_recovery_source_key_arn therefore pins the exact public key ARN verified with DescribeBackupVault; the vault has prevent_destroy. Recheck the pin against live vault metadata after any deliberate vault recovery/replacement. It does not grant access to every key in other accounts. AWS Backup starts cleanup after successful validation (or the two-hour validation window); the controller checks expected per-volume jobs and backs up cleanup after four hours. Require a real successful restore/validation/cleanup cycle before claiming recovery testing is proven. Five east1 tests cost $7.50 plus roughly $0.094 per hour retaining all restored volumes. At September 7 pricing and measured written snapshot sizes, east1 air-gapped storage starts around $39.85/month (693 GiB at $0.0575), plus roughly $5.78 for the two rolling CMK checkpoints (115.5 GiB at $0.05). These are baseline estimates, not a fixed bill: changed blocks, temporary-source lifetime, old history aging out, transfer, KMS/API usage and restore tests add cost. A one-day source TTL would retain nearly a continuous extra baseline; cleanup after verified completion avoids that steady-state design, but slow or failed copies still accrue storage.

The jumpbox's gp3 volumes are capped at 125 MB/s. Never run broad recursive searches across /home/ubuntu or /tmp; one such scan saturated both disks and made SSH unusable. Scope searches to the repository, prefer rg, and move large scans/builds to ARM or the parallel box.

Regional security defaults

security-defaults.tf manages 136 account settings: four defaults across 17 enabled regions in both the main and recovery accounts (34 account/region combinations). Coverage is ap-south-1, ap-northeast-{1,2,3}, ap-southeast-{1,2}, ca-central-1, eu-central-1, eu-north-1, eu-west-{1,2,3}, sa-east-1, us-east-{1,2} and us-west-{1,2}. Newly enabled regions must be added explicitly; this does not enable opt-in regions. security-regions.tf reuses the existing primary/DR providers and adds only the missing regional aliases, with the existing recovery-account admin role.

EBS encryption by default protects new volumes and new snapshot copies. Existing disks and snapshots are not re-encrypted; snapshots of an existing volume still inherit that volume's encryption. No KMS key is created or changed. The deferred existing-disk migration is not part of this rollout. See EBS encryption defaults.
Snapshot block-all-sharing prevents new public sharing and also blocks public access to already-public owned snapshots. Private sharing and cross-account backups remain allowed. It does not erase the underlying public permission: disabling the block could re-expose old public snapshots. EBS-backed AMI public sharing is a separate control. See snapshot public-access blocking.
IMDS HttpTokens=required is the default for future launches, not hard enforcement: explicit launch options can override it. Existing instances are not changed. The pinned provider also writes no-preference for the regional endpoint/tag defaults and -1 for the hop limit, matching the September 8 inventory's unset values. Review any non-default regional metadata setting before applying. See regional IMDS defaults.
SSM document public sharing is blocked by the regional service setting /ssm/documents/console/public-sharing-permission = Disable. This prevents new public shares; it does not revoke existing public shares or affect private account sharing, use of AWS-owned documents, SSH, or Session Manager. September 12 preflight found public sharing permitted in all 34 pairs, with two owned documents and no public shares. Require get-service-setting to report Disable everywhere after apply. The Terraform resource has prevent_destroy because removing it would reset the service default and permit public sharing again. See SSM document sharing.

These settings have no recurring service subscription and create no detectors, log pipelines, keys or compute. Ordinary future storage, copy and KMS usage still costs money; in particular copying with a different key can require a full snapshot copy. This is separate from the approved paid monitoring foundation, not evidence that security logging/detection is active. Public SSH/ET, Cloudflare Access service-token access and the verified backup plans, recovery history and cleanup permissions remain unchanged.

Before applying, read the full fresh plan: expect only these regional account settings, no instance, volume, backup or access-policy changes. Inventory owned publicly restorable snapshots in both accounts using describe-snapshots --owner-ids <account-id> --restorable-by-user-ids all in every listed region; investigate any result before changing its public visibility. After applying, require an empty follow-up plan and read back get-ebs-encryption-by-default, get-snapshot-block-public-access-state and get-instance-metadata-defaults in all 34 combinations. Expect true, block-all-sharing and AccountLevel.HttpTokens=required, respectively. Account defaults do not prove existing hosts or disks have been remediated.

Security monitoring rollout

The owner approved the monitoring foundation on September 8, 2026, at approximately $9/month fixed plus metered usage. Deployment requires the reviewed-plan, regional compatibility and delivery gates below; approval is not evidence of live delivery. The owner separately approved paid scanning on September 12, 2026. Live readback now confirms the five-account/region Config, Security Hub CSPM and Inspector stage described below is enabled; the known alert's actual mailbox receipt was independently verified before rollout. Service enablement is not a claim that all findings are remediated.

security-monitoring.tf and modules/security-region/main.tf define the monitoring foundation in both accounts across all 17 currently enabled regions (34 account/region pairs). The foundation records organization-wide management events and object access to the Terraform-state and dev-script buckets, enables base GuardDuty, and records traffic for the three active VPCs. Regional EBS, IMDS and snapshot-sharing defaults and the free external-access analyzers have independent owners; these monitoring modules do not recreate them. Monitoring does not migrate existing disks, restrict the required public SSH/ET access, remove Cloudflare service-token access, or change the verified backup pipeline.

For a new deployment, roll out the foundation first with local.security_posture_enabled = false. After delivery acceptance, a usage/cost review and separate owner approval, enable that single bootstrap gate for Config, Security Hub CSPM foundational checks and Inspector EC2/Lambda scanning: main account us-west-1, us-west-2, us-east-1, and recovery account us-west-1, us-east-1. The final recovery vault is in us-east-1 and must not be omitted. Config records global IAM once per account; EC2 instances, interfaces and volumes use daily recording to bound runner churn, so those posture checks can lag by 24 hours. Daily recording produces a configuration item only when the last recorded state has changed; it is not a daily charge for every unchanged resource. Other supported resources are recorded continuously. This uses the existing CSPM API, not the newer Security Hub Essentials bundle. See Config recording and CSPM pricing.

GuardDuty uses the pinned provider's aws_cloudcontrolapi_resource for the single AWS::GuardDuty::Detector, because its typed detector feature enum predates AI_PROTECTION and AI_ANALYST. This is still Terraform-managed, without a second CloudFormation stack or provisioning script. The initial request explicitly disables every regionally supported optional feature, including AI Protection and runtime agents. AI_ANALYST is specified only in the ten regions where AWS currently offers GuardDuty Investigation; the explicit regional list lives in the module. The September 8 single-region compatibility check confirmed that requesting this unavailable feature in us-west-1 is rejected even with status DISABLED; no detector was created. See GuardDuty Investigation availability. Supported features must be present and disabled in live readback. A documented unavailable feature may be absent or disabled, but any unexpected enabled feature or nested agent still fails the postconditions. Missing fields are never a blanket substitute for disabling supported features. Regional handler support must still be verified during deployment: schema validation is not proof that each regional service accepts every feature. Do not silently omit an unsupported feature or accept an unreviewed provider upgrade. See CreateDetector defaults and Cloud Control resource. The pinned generic resource has no import handler. Preserve versioned Terraform state; if its state entry is lost, recover the reviewed state version rather than creating a second detector. A failed live-property postcondition requires an administrator to diagnose and review a repair or future typed-resource migration; it does not perform automatic remediation. Never remove prevent_destroy merely to make a plan pass.

After the separate 136 regional defaults, 36 external-analysis resources and two global S3 controls, the monitoring foundation adds 293 managed Terraform resource instances plus one existing SNS topic-policy update; the posture gate adds 45 more. These are source counts, not a substitute for the fresh plan. Many are free control settings or permissions, not individually billed workloads. No existing compute, volume or backup resource should be replaced by this rollout.

Security findings, root-account activity (including read-only management calls), and high-risk IAM, network, KMS and backup changes reach the cost-alerts SNS topic through the digest Lambda in security-alert-digest.tf (since 2026-10-19, when one email per finding update reached 3,608 in a day and SNS disabled email on the topic): each finding is mailed once, immediate sources at once and Security Hub and Inspector in a digest, at most one message a minute. Inspector and GuardDuty use their native finding events only; their Security Hub imports are excluded from forwarding to avoid duplicate notifications. Foundational checks and other Security Hub products retain their existing notification path. Normal processing/checkpoint cleanup does not page the owner; deletion attempts against legacy/final backup history do. Every region has two disjoint forwarding rules sharing one encrypted 14-day dead-letter queue; each event pattern stays within AWS's default 2,048-character limit. A five-minute read-only watchdog verifies enabled rules, exact event filters and targets (destination, execution role, retries and DLQ), and queued events in all 34 pairs, plus the digest pipeline behind the central rule (its event source, its topic, its flush schedule and its DLQ). It alternates regional failure-metric queries between the two rules: each rule is sampled every ten minutes with a fifteen-minute lookback, preserving the approved CloudWatch retrieval budget. Central delivery metrics are still sampled every five minutes. Independent missing-heartbeat and Lambda-error alarms cover observer failure. It never consumes or deletes queued events. Both the regional audit rule and the central notification rule use ENABLED_WITH_ALL_CLOUDTRAIL_MANAGEMENT_EVENTS: forwarding to a custom bus does not provide a documented exemption from the read-only management-event rule state. Their event patterns stay unchanged, and the watchdog checks both states. See AWS read-only management-event routing. Regional event-bus forwarding targets retain their execution roles and regional DLQs, but omit retry_policy: AWS rejects that setting for event-bus targets. The central SNS target still has a one-day/185-attempt retry policy; the watchdog's Lambda target retains its five-minute/two-attempt policy. Target readback checks this distinction exactly rather than ignoring retry settings. The S3 audit archive has 90-day governance retention and one-year expiry; privileged administrators can bypass governance, unlike the recovery vault's compliance lock. Config/Session Manager records use a separate versioned bucket. Only standard SSM shell sessions are transcribed; SSH, Eternal Terminal and SSM port forwarding are not. The builder instance role receives only bucket-location/encryption metadata reads and transcript-prefix writes, not transcript reads or broader S3 access. Runner/builder and shared jumpbox role grants follow their existing optional-resource gates.

An apply is not delivery proof. Require a fresh empty plan, then read back trail logging status and recent digest/log objects, all three flow-log delivery statuses/objects, regional detector/feature/analyzer configuration, event targets and DLQ policies, three clean watchdog executions, and five healthy delivery alarms. Confirm SNS metrics and the actual email for a known event. A GuardDuty sample finding is a synthetic service write and sends a real notification; use it only as an explicitly authorized acceptance canary, not as a read-only check. Without an observed matching event, record the end-to-end alert path as unproven. A later posture rollout also requires Config recording/delivery, enabled CSPM standards and Inspector coverage checks; managed/agentless scan coverage is not established merely by enabling the service.

September 8, 2026 deployment receipt: infrastructure revision 80bbd3a was applied and a fresh full Terraform plan reported no changes. Readback verified 34 base-only GuardDuty detectors, 70 routes (68 regional forwarding targets, the central SNS target and the watchdog schedule target), three clean watchdog executions and five healthy delivery alarms. The RSA-2048 signature of one initial main-account us-west-1 CloudTrail digest was verified; it referenced zero log files, and no event-log bodies were read. Event-log hashes/full-chain integrity, actual email receipt, an SSM session transcript and a flow-log object from the quiet us-west-2 VPC were then unproven. The September 12 checks below close those delivery gaps.

September 12, 2026 acceptance: the owner-authorized GuardDuty high-severity sample 570453a9836e4b60be3faecb4ec18612 (Backdoor:EC2/C&CActivity.B!DNS, severity 8, sample=true, fictitious instance i-99999999) was created at 18:06 UTC. The regional forwarder and central SNS target each recorded one successful invocation, and SNS recorded one publication, two deliveries to its existing email/Lambda subscriptions, and zero failures. A preceding severity-2 SSH sample was below the intended notification threshold. Read-only access to the owner's configured mailbox verified the exact positive sample message received at 18:06:30 UTC; this closes the delivery gate without inferring mailbox receipt from SNS metrics or claiming the owner had read the message. Four clean 34-region watchdog executions and all five healthy alarms were read back. A bounded standard SSM shell session on fcvm produced a 426-byte encrypted, versioned transcript whose unique test marker was verified after the session terminated. Retained September 9 S3 flow-log objects match the exact active west2 flow-log ID; west2 compute was stopped during this check and no extra instance was launched. All three flow logs report successful delivery and have matching objects. CloudTrail digest signatures, chain links and referenced log hashes were verified with the official CLI for a bounded 15:00–17:00 UTC window across all 34 account/region pairs (102/102 digests and 1,074/1,074 logs valid). This window is not a claim of integrity for all historical log objects; audit log bodies were not printed or sent to development hosts.

September 12 paid-posture acceptance: all five Config recorders report recording and successful snapshot delivery. Exact snapshot objects from every pair are nonempty, AES256-encrypted and versioned in the separate records bucket. Four pending channels received one owner-authorized DeliverConfigSnapshot acceptance trigger each; the recovery account's west1 channel delivered naturally. All five AWS Foundational Security Best Practices subscriptions report READY; Inspector EC2 and Lambda report ENABLED. The first enrollment outlasted three provider create waiters but completed in AWS; only independently verified resources may have their timeout taints cleared, never disable/recreate working scanners to recover from a waiter timeout. Creation waits are now explicitly bounded at 20 minutes for these two resource types.

Initial coverage verified successful package scans for ten Lambda functions and all six running EC2 instances, including both temporary runner acceptance hosts. The stopped x86 and I/O boxes had inactive coverage in this initial inventory. The other three posture pairs had no EC2 instances or Lambda functions to scan. Inspector uses its hybrid EC2 mode, so agentless fallback may incur temporary snapshot charges. ECR, Lambda code and code repository scanning remain disabled; GuardDuty optional protection and runtime agents remain outside the approved base-only scope. Following the deduplication rule rollout, three consecutive 34-pair watchdog runs at 18:53, 18:58 and 19:03 UTC were clean, and all five delivery alarms were healthy.

Initial findings are a remediation backlog, not a failed deployment or a clean audit. The bounded 19:03 UTC inventory found active high/critical package findings; exact resource, package and remediation evidence is retained privately. Fixes require package-owner, vendor-version and Ubuntu ESM entitlement checks; not every flagged package has a currently available vendor fix. Do not replace fcvm's custom nested-virtualization kernel or restart shared development sessions as a side effect of a scanner rollout. Security Hub also reports intentional public SSH/ET and disabled optional-service recommendations, alongside genuine control gaps requiring separate review. No findings were suppressed or archived. Config's daily EC2 inventory and initial control evaluation can lag; freshly enabled services may temporarily retain stale failed controls. Recheck the actual service and resource before treating every failed control as a new outage or authorizing extra paid services.

Threat model and package-remediation priority

These are trusted-admin development hosts: owner/root access is intentional, not an isolation failure. Application origins remain private behind Cloudflare Access, while public SSH on 22 and Eternal Terminal on 2022 remain explicit requirements. This is restricted application ingress, not zero public attack surface.

The intended impact boundary is explicit: a development-host compromise is bad; a jumpbox compromise is catastrophic because it exposes AWS administrator authority, Terraform state and control-plane credentials. Assume an attacker who compromises a dev host can become root and take its instance-role credentials. Even then, that host must have no path to jumpbox authority: no admin SSH key, forced-command key, SSM command permission, or queue/other privileged delegation through a jumpbox. Dev-side automation must use narrow, resource/tag-scoped IAM directly, never an admin-host hop. This is an authorization boundary, not a claim that public jumpbox SSH is unreachable. Jumpbox-facing vulnerabilities, agent/tool inputs and credential protection merit stricter treatment than routine development-host package hygiene.

Prioritize exploitable public-facing vulnerabilities, authentication bypass, exposed credentials, unauthorized AWS privileges, and unexpected public routes. Also prioritize malicious webpages, repositories or build inputs gaining host execution, and escapes across intended VM/sandbox boundaries. Outbound access and Cloudflare authentication do not eliminate these risks.

Local-only privilege escalation affecting users already trusted with root is routine maintenance unless it crosses another meaningful boundary. Scanner severity alone does not justify purchasing Ubuntu Pro, rebooting shared hosts, removing development tools or replacing fcvm's custom kernel. Review exploitability and actual exposure, then schedule targeted maintenance. Under the owner's September 12 direction, ordinary development-package findings are lower priority than these access and credential boundaries; Ubuntu ESM enrollment is deferred, not an outstanding deployment gate. Dolphin's installed kernel update still awaits a separately scheduled reboot; this policy does not authorize interrupting shared sessions. Vendor fixes for every reported issue are not yet available, and this risk acceptance does not mean the packages are fixed.

Keep scanners, alerts and the remediation backlog enabled. Preserve the dev-to-admin credential boundary, credential-free ordinary CI, protected backups and Access policies.

Outside users. Two surfaces now let people outside the owner reach AWS resources this repo runs. Unlike the dev hosts, they are not behind Cloudflare Access, and their users are not trusted: multiplayer players get no shell anywhere, while a runner job gets root on its own disposable VM (the runner account has passwordless sudo), so the boundary there is the VM, its instance role and its network, not the account:

Games multiplayer. The public lobby at cc-games.org (canonical; cc-games.net and cc-games.app, which is blocked on a school network, redirect to it) launches Fargate match engines, and anyone on the internet can connect to play.cc-games.org, play.cc-games.net or play.cc-games.app, one internet-facing ALB. A WAF (per-IP rate limit, AWS managed rules), join tokens, an origin allowlist, per-IP limits in the router and router-only engine ingress protect it. The engines' task role has no permissions and their egress is HTTPS-only. Their images deploy automatically: a push to colton-games main reaches production engines (and the router, and every migration) with no human step, and any branch's code runs in preview engines only. The lobby's launch limits are pinned by Terraform. The lobby holds no ECS permission: it can only invoke a launch function that builds every RunTask from fixed settings and refuses above AWS's own ceiling on concurrent engines (smaller for preview). The sweeper enforces that ceiling and each task's lifetime (about 4h15) every minute as a backstop, and only while its sweeps and StopTask calls succeed. See the threat model in docs/games-multiplayer.md.
Runners for other repos. Every writer on CoderColton/colton-games and dolphin-labs-hq/dolphin-labs can run any code on our x86 spot runner VMs. They are ephemeral and have no inbound access, and the tokens that register them are out of reach. See "Threat model: runners for other repos" in GITHUB-RUNNERS.md.

For both, prioritize cost abuse (launches and VMs someone else can start), anything that reaches an admin credential or another tenant's work, and anything that turns a public request into host execution. Each document lists its open gaps and how to shut it off.

Incremental monitoring cost

September 8, 2026 rate checks put the watchdog's CloudWatch component around $8.06 per 30-day month, not the whole monitoring bill: 622,080 metric queries (including the higher São Paulo rate), four custom metrics and five standard alarms. The trail's rotating customer key adds about $1/month. The first management-event trail copy has no CloudTrail event charge; selected S3 data events cost $0.10 per 100,000 events. Lambda, SQS, SNS, KMS requests, S3 storage/requests and flow-log delivery are additional. See CloudWatch pricing, CloudTrail pricing and KMS pricing.

Usage-based rates in the primary region include GuardDuty management analysis at $4.40/million events and flow/DNS analysis at $1.10/GB, plus flow-log delivery to S3 at $0.335/GB. The approved posture stage adds Config at $0.003/continuous or $0.012/daily configuration item and CSPM at $0.001/security check. Security Hub service-linked Config rule evaluations are not separately charged. Recent measured EC2 usage (777.46 instance-hours over September 1–7, both accounts) and ten covered functions including the new watchdog imply roughly $10–13/month for Inspector if that usage continues; agentless scanning can also create billable temporary EBS snapshots. See GuardDuty, Config, CSPM and Inspector pricing.

For scale only, an assumed month with one million GuardDuty events, 5 GB analyzed logs, 5 GB flow-log delivery, 1,000 daily and 2,000 continuous Config items, and 10,000 CSPM checks would be about $59–62 including the above fixed components and projected Inspector, before storage/API charges. Those event/configuration volumes are examples, not measured forecasts or a spending cap. Review Cost Explorer usage types and GuardDuty usage statistics after the first complete week; initial Config inventory, rapid runner churn or unusually noisy network activity can raise the bill. The existing account-wide cost alerts remain useful, but do not enforce a monitoring-only budget limit.

Account-wide S3 public-access blocking

security-s3-account.tf manages two global account controls, one each for main and recovery, with all four S3 Block Public Access flags enabled. They cover current and future buckets/access points in every region; unlike the future-only EBS defaults, these controls can also block existing public access. They do not rewrite bucket policies/ACLs, migrate disks, create paid monitoring services, or change public SSH, Cloudflare service-token access or the verified backup pipeline.

Before deployment, the September 8, 2026 16:41 UTC read-only inventory found six main account buckets and no recovery-account buckets. All six already had all four bucket-level blocks, BucketOwnerEnforced, no bucket policies, no effective public ACL grants and no S3 website configuration. All 34 regional access-point listings and both multi-region listings were empty. No objects were enumerated or read. Private operator evidence is in /tmp/aws-guardrails-review.inJgVqJ5/s3-public-metadata.json and s3-access-point-metadata.json; these ephemeral files are not cold-bootstrap inputs. Recheck metadata if inventory changes before apply.

S3 combines account, bucket and access-point settings using the most restrictive values. Fixed-principal private sharing remains possible; a public bucket policy can cause RestrictPublicBuckets to block even its otherwise private cross-account grants. Review intended S3 websites/sharing before changing these controls, and do not disable the account block to make an unreviewed public policy pass. See AWS Block Public Access semantics. Expect a fresh full plan with exactly two account-setting creates and no bucket, object, policy or backup changes. After applying, require an empty follow-up plan and read back s3control get-public-access-block --account-id <account-id> under each account's verified credentials: all four flags must be true. Account-wide settings propagate globally but not necessarily simultaneously across regions.

External access findings

security-external-access.tf creates 34 external-access analyzers: one ACCOUNT analyzer named security-external-access in each account/region covered by the regional defaults above. It reuses the same providers and does not enable regions. AWS lists external analysis at no additional charge; paid internal-access/unused-access analyzers and paid custom policy checks are not enabled. Terraform also owns the two required AWSServiceRoleForAccessAnalyzer roles, one per account (36 resources total). Each account's analyzers depend on its own role to avoid racing first-time service-role creation. Only Access Analyzer can assume this read-only, service-owned resource-metadata role; its trust/permissions are defined by AWS, not custom grants to operators or workloads. No workload role gains permissions. See service-role boundary.

This reports supported resource-policy access from outside the account, including public and cross-account sharing. It is not an access-control enforcement or proof that all identity-policy escalation paths are closed. Intentional sharing between the main and recovery accounts can generate findings because each account is its own trust boundary. One finding shape is archived automatically: AWS Backup's own temporary share of an EBS snapshot with its service account while it copies into the air-gapped vault (us-west-1 and us-east-1; #577). No other finding is archived; inspect their resource, principal and policy before deciding whether access is intended. Creating an analyzer does not change public SSH, Cloudflare service-token access, disks or backup sharing.

This free stage does not add email delivery. Findings are available through the Access Analyzer API; the separately reviewed monitoring stage can forward finding events later. After the fresh full plan/apply, verify all 34 analyzers have exactly type=ACCOUNT and status=ACTIVE using list-analyzers/get-analyzer, then inspect active findings with list-findings. Allow the documented analysis delay (policy changes can take up to 30 minutes); an immediately empty list is not a completed security audit. Do not silently create a second analyzer or replace an existing one if preflight finds account-level analysis already configured.

Host access audit

access-audit.tf and scripts/access-audit.py report two kinds of activity to the operator:

Registered paths opened on a host. The agent watches a private list of files and reports each open by another process, with the reader's executable, pid, user, command line, its process ancestry (pid and start time of each parent) and, when the reader's environment carries one, its SSH_CONNECTION. That variable is whatever the reader's environment says, so treat it as a lead, not proof of the source. It also reports when a registered file is modified, replaced or removed, and when its own sensor degrades.
Registered principals used anywhere. Each registered host has an IAM principal whose policy denies everything. CloudTrail still records every call it makes, and a rule in each of the 17 enabled Regions forwards those calls to the access-audit bus in us-west-1. This covers every management event and the S3 data events the account trail already selects. Data-plane calls the trail does not record (an S3 object read in another bucket, a Lambda invoke) are not seen.

Everything that says which hosts, files and principals are registered is private. It is held only in the access-audit/config secret and in Terraform state. Nothing in this repository, its tests or its pull requests names it, and nothing should.

What fires

Signal	Where it shows	Alarm
Registered path opened	operator email (per report, at most 20 per 10 minutes per host), `/security/access-audit` log stream for the host, `AccessAudit/FileRead`	`access-audit-file-read`
Registered path changed, sensor degraded or silent	the same three	`access-audit-tamper`
Registered principal made an API call	operator email naming caller, action, source IP and user agent; `AWS/Events TriggeredRules` for `access-audit-notify`	`access-audit-principal-use`
Agent stopped reporting (stopped, killed, host down)	no heartbeat for 15 minutes	`access-audit-heartbeat-<host>`
Principal notification could not be delivered	`AWS/Events FailedInvocations`	`access-audit-delivery`

The access-audit CloudWatch dashboard shows all of them and the latest host reports.

After a delivery failure, empty its dead-letter queue. A forward or notification that fails lands in a dead-letter queue (access-audit-forward-dlq in the Region of the failed rule, or security-central-delivery-dlq in us-west-1 for the notification) and is kept 14 days. While it is there, the security observer's security-delivery-QueuedEvents alarm stays in ALARM, so a later, unrelated failure raises no new alarm. Once the cause is fixed, drain the queue into a private file (the events carry the call or report that was missed, so keep them), deleting each batch only after it is saved, until the queue is empty; the alarm then returns to OK:

( umask 077; set -euo pipefail
  Q=$(aws sqs get-queue-url --region <region> --queue-name <queue> --query QueueUrl --output text)
  while :; do
    batch=$(aws sqs receive-message --region <region> --queue-url "$Q" --max-number-of-messages 10 \
      --wait-time-seconds 5 --visibility-timeout 120 --attribute-names All)
    [ -n "$batch" ] && [ "$(jq '.Messages | length' <<<"$batch")" -gt 0 ] || break
    jq -c '.Messages[]' <<<"$batch" >> dead-letters.jsonl
    jq -r '.Messages[].ReceiptHandle' <<<"$batch" | while read -r handle; do
      aws sqs delete-message --region <region> --queue-url "$Q" --receipt-handle "$handle"
    done
  done
  aws sqs get-queue-attributes --region <region> --queue-url "$Q" --attribute-names \
    ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible ApproximateNumberOfMessagesDelayed )

It is empty only when all three counts are 0. A message a failed run left in flight stays invisible for the 120-second visibility timeout: if any count is not 0, wait two minutes and run it again (the file is appended to, so nothing already saved is lost). Read dead-letters.jsonl, act on it, then shred it.

This is the same for the other security dead-letter queues.

Sensors

Linux: fanotify permission events. The opener is held only while the agent reads its /proc entry, then always allowed, so even a reader that exits at once is identified. Opens are matched by inode, so a read through a hard link or another mount of the same file is reported under the registered path, with opened_path naming the one used. Files replaced or recreated are re-marked by the integrity check.
macOS: eslogger open runs as its own launchd job (<label>.helper; the label is local.<service name> unless the registry sets launchd_label) and writes into a root-only FIFO the agent reads, so only /usr/bin/eslogger needs Full Disk Access. Give it that once: System Settings, Privacy & Security, Full Disk Access, +, Cmd-Shift-G, /usr/bin/eslogger. Until it has it, or whenever it stops, atime polling still reports reads (without the reader's identity) and the agent sends one sensor silent report. eslogger does not report processes in its own login session, which matters only when testing by hand. eslogger events are matched by path, so a read through a hard link under another name is caught by the atime poll (atime is per inode) without the reader's identity. Executables in ignore_executables are ignored by both.

Rollout

The secrets are containers first, then values, then the full build, the same order as other private values in this repository.

Merge, then on a jumpbox apply with the default enable_access_audit = false. This creates only the two empty secret containers and the always-present bus, roles, alarms and dashboard.
Write the registry. Read it from a file you then delete; never put it on a command line:
```
aws secretsmanager put-secret-value --region us-west-1 --secret-id access-audit/config \
  --secret-string file://registry.json && shred -u registry.json
```
Shape (placeholders):
```
{"hosts": {"example-host": {
  "principal": "example-principal-name",
  "files": [
    {"path": "/home/example/.config/example/settings", "owner": "example", "mode": "0600",
     "content": "key={{principal_access_key_id}}\nsecret={{principal_secret_access_key}}\n"},
    {"path": "/home/example/.example/notes", "owner": "example", "watch": false,
     "content": "{{random_hex:32}}\n"}
  ],
  "ignore_executables": [],
  "service_name": "example-name"
}}}
```
Templates: {{principal_access_key_id}}, {{principal_secret_access_key}}, {{ssh_private_key}} and {{ssh_public_key}} (one key pair generated per install), and {{random_hex:N}}. service_name (optional, default hostmetricsd) names the installed service and its paths; launchd_label (optional, local. prefix) overrides the macOS label. watch: false places a file without watching opens of it; use it for files that ordinary tools read on every run. age_days (default 120) back-dates each file. Choose paths no routine job reads: backup agents, indexers and file-sync clients would page you. The installer refuses to overwrite any file it did not place, and refuses a path whose directories include a symlink the file's owner could have made.
Set enable_access_audit to true in a commit, plan, read the plan, apply. Creating the principals and their keys trips the existing IAM-change security alerts; that is expected.

Install on each host. On the jumpbox, take that host's bundle from the generated secret and pipe it with the agent to the host (directly, or through the dev box that reaches it):

# umask 077 in a subshell: bundle.json holds the reporter key, the planted key and any generated SSH private key,
# so it is created 0600 and never readable by another account, even before the shred.
# set -e and the EXIT trap: a failed copy or install fails the whole command, and the bundle is shredded either way.
( umask 077
  set -euo pipefail
  trap 'status=$?; shred -u bundle.json 2>/dev/null || true; exit "$status"' EXIT
  aws secretsmanager get-secret-value --region us-west-1 --secret-id access-audit/bundles \
    --query SecretString --output text | jq -c '."example-host"' > bundle.json
  scp scripts/access-audit.py example-host:/tmp/access-audit.py
  ssh example-host 'sudo python3 -I /tmp/access-audit.py install && rm /tmp/access-audit.py' < bundle.json )

Prove the delivery path without touching a registered file: sudo python3 -I /usr/local/libexec/hostmetricsd/hostmetricsd test-alert sends one email and log line labelled TEST, and exits nonzero naming any destination (log, metrics, email) that did not take it. With a service_name, the path is /usr/local/libexec/<name>/<name>.

Re-running the install replaces only files the agent placed and that are unchanged since. It stops the running agent first, so the replaced files do not page as tamper, and starts it again at the end; an install that fails in between leaves the agent stopped, which access-audit-heartbeat-<host> reports within 15 minutes, and re-running the install starts it. sudo python3 -I /usr/local/libexec/hostmetricsd/hostmetricsd uninstall stops the service and removes only those files. If an install fails part-way, fix the bundle and re-run the install: it records each file as it is placed, so the retry takes over the ones already there. Removing a host from the registry and applying deletes its principal, reporter, stream and heartbeat alarm.

Tests

python3 -S -B scripts/test-access-audit.py: offline (signing against the AWS-published SigV4 example, bundle validation, file placement and its refusal to overwrite, the eslogger stream and FIFO reader, reporting limits, integrity, and the Terraform boundaries).
sudo python3 -S -B scripts/test-access-audit-live.py: the live Linux fanotify sensor.

dolphin-labs news pass

dolphin-labs' team-news pass and its X-posts step run in the Dolphin account, from dolphin-labs-hq/dolphin-aws (news.tf): the functions dolphin-labs-news-pass and dolphin-labs-news-x, their journal bucket, secrets, deploy role and alarms. This account held the first copies until 2026-10-22; nothing of them is left here.

Private browser manager

The application, its browserctl CLI and the CDP bridge for scripted jobs live in the public repository ejc3/browser-manager: installation, usage, the viewer, the security model and development are documented there. This repository provisions what it runs behind: the dedicated tunnels, DNS records, owner-only Access applications and service tokens for browsers.cc-games.dev (fcvm-metal-arm) and mac-browsers.cc-games.dev (the personal Mac), and the secrets that carry their credentials (browser-manager.tf, browser-manager-mac.tf). It does not modify Dolphin Labs or its tunnel. Only the owner has Access permission.

1. Apply on your administration host

Use this repository's Terraform 1.10.3 and existing backend/provider setup. Review the plan for the whole stack; do not use -target to bypass unrelated changes. No Terraform plan or apply runs on the browser/dev host, and it does not delegate one to a jumpbox.

umask 077
bm_handoff=$(mktemp -d)
terraform init -lockfile=readonly
terraform plan -out="$bm_handoff/plan"
terraform apply "$bm_handoff/plan"
terraform output -raw browser_manager_env > "$bm_handoff/browser-manager.env"

The hostname is fixed at browsers.cc-games.dev. Terraform creates a dedicated tunnel, DNS record, and owner-only Access application using the existing Google/one-time-PIN providers. It does not reuse the kids' family policy or Dolphin's GitHub policy. The browser host needs no Terraform, AWS administration credentials, or Cloudflare account API token.

Terraform stores the raw connector token in AWS Secrets Manager as browser-manager-tunnel-token in us-west-1. The shared ARM/x86 dev-server instance role can read only this secret through its dedicated policy. Fetching it requires no additional AWS login; the Next.js, runner, and temporary-compute roles do not receive this grant.

Transfer the public environment file privately to the browser host, owned by the browser user with mode 0600. On the ARM/x86 host, retrieve the connector token as ubuntu:

umask 077
install -d -m 0700 /home/ubuntu/.config/browser-manager
bm_token_tmp=$(mktemp /home/ubuntu/.config/browser-manager/.tunnel-token.XXXXXX)
if aws secretsmanager get-secret-value --region us-west-1 \
  --secret-id browser-manager-tunnel-token --query SecretString --output text > "$bm_token_tmp" \
  && test -s "$bm_token_tmp"; then
  mv -f "$bm_token_tmp" /home/ubuntu/.config/browser-manager/tunnel-token
else
  rm -f "$bm_token_tmp"
  echo 'Tunnel token retrieval failed; the existing token file was preserved.' >&2
fi

The connector token and saved Terraform plan are sensitive. Do not print or check them in. The application receives the public Access settings; only cloudflared receives the token file path. A public URL cannot work until both this apply and the host installation are done.

Terraform also creates separate Cloudflare Access service tokens for OpenClaw viewers of the AWS and Mac browser hosts. Each is accepted only by its exact Access application, and the origin independently verifies that the signed JWT's common_name equals that token's client ID. The client ID is public application configuration. The secrets are stored in AWS Secrets Manager as browser-manager-openclaw-access and browser-manager-mac-openclaw-access, and also in sensitive Terraform state. Resource policies deny direct reads by dev and CI roles. Account administrators can still change those policies; this is not an immutable boundary against an administrator.

The local Mac uses its existing AWS SSO login to assume browser-manager-openclaw-client. That role can read only those two viewer secrets; it has no Cloudflare administration, tunnel, or other Secrets Manager access. Configure the local profile after apply. Obtain browser_manager_openclaw_client_role_arn with Terraform on the administration jumpbox and hand that public ARN to the Mac operator; do not run Terraform on the Mac. For this account the commands are (no credential value):

aws configure set profile.browser-manager-openclaw role_arn \
  arn:aws:iam::928413605543:role/browser-manager-openclaw-client
aws configure set profile.browser-manager-openclaw source_profile AdministratorAccess-928413605543
aws configure set profile.browser-manager-openclaw region us-west-1

Both service-token resources have prevent_destroy: replacing one changes the client ID pinned in the browser-manager environment. Do not remove the guard for an ordinary apply. A deliberate replacement requires a coordinated maintenance deployment: save the corresponding browser_manager_env or browser_manager_mac_env output, reinstall/restart that origin with its environment, refresh the local viewer session, and verify both machine access and unauthenticated denial. The existing owner-login policy stays attached throughout. A stale origin must fail closed, not accept an arbitrary replacement service identity.

2. Install on the browser host

On fcvm-metal-arm, as ubuntu, the checkout is ~/browser-manager (a clone of the public repository) and the installer gets the transferred environment file and the tunnel token retrieved above:

git clone https://github.com/ejc3/browser-manager ~/browser-manager && cd ~/browser-manager
npm ci --ignore-scripts && npm run build
./scripts/install.sh /home/ubuntu/.config/browser-manager/browser-manager.env /home/ubuntu/.config/browser-manager/tunnel-token

To update: git pull, npm ci --ignore-scripts && npm run build, and rerun the installer with only the environment file (the tunnel unit is left as it is). The installer restarts browser-manager.service, which restarts every desktop (profiles and logins are kept); ask before doing it while someone may be using one. A desktop on an external profile (--profile) can come back in an error state after that restart, because Chrome leaves its Singleton* lock links behind: once pgrep -f <profile path> shows nothing, unlink the three links and browserctl start <name>.

OpenClaw viewers

The OpenClaw viewer credentials are the service tokens above. On the Mac, from a clone of the public repository, with the allowed hosts and the Access team in the environment (the helper takes them from nowhere else):

export BM_VIEWER_PROFILES='{"https://browsers.cc-games.dev":"secure-browser-viewer","https://mac-browsers.cc-games.dev":"secure-mac-browser-viewer"}'
export BM_ACCESS_ISSUER=https://ejc3.cloudflareaccess.com
aws --profile browser-manager-openclaw secretsmanager get-secret-value \
  --secret-id browser-manager-openclaw-access --query SecretString --output text \
  | node scripts/configure-openclaw-viewer.mjs secure-browser-viewer
aws --profile browser-manager-openclaw secretsmanager get-secret-value \
  --secret-id browser-manager-mac-openclaw-access --query SecretString --output text \
  | node scripts/configure-openclaw-viewer.mjs secure-mac-browser-viewer

An expired AWS SSO session must be renewed with aws sso login --profile AdministratorAccess-928413605543 first; the helper does not copy administrator credentials or bypass a failed login.

Connector rotation

The AWS browser connector secret is Terraform-managed. For an intentional rotation, bump random_bytes.browser_manager_tunnel_secret.keepers.rotation, review a fresh full plan, and apply the in-place tunnel update and new Secrets Manager version. Never replace the tunnel, DNS, Access application, or Mac connector. Read the replacement secret directly on ARM using its instance role into a private temporary file, verify it, and atomically install it as ~/.config/browser-manager/tunnel-token (owner ubuntu, mode 0600). Restart only browser-manager-tunnel.service; the browser manager and its desktops stay running. After a suspected exposure, also invalidate existing connections for this exact tunnel through Cloudflare's connections API, then verify the new connector is ready. Rotation alone prevents new connections with the old credential but does not disconnect old ones. Never print either token or place it in command arguments, Git, or a dev-host AWS admin session. Installed connector units read a local file; they do not automatically refresh from Secrets Manager. A stopped AWS browser host (including x86) is not verified by an ARM rotation: leave it stopped, and refresh its token through its own instance role before restarting its connector on the next authorized boot. Do not start an extra replica against a different browser manager, because Cloudflare would distribute traffic between unrelated desktops. See Cloudflare token rotation and connection invalidation.

September 12, 2026 receipt: revision cc87713 rotated the AWS tunnel in place and published a new Secrets Manager version. ARM fetched it using its own instance role; the installed file was verified ubuntu:ubuntu, mode 0600. Existing connections were invalidated and a new connector became healthy. The browser-manager process stayed running; unauthenticated access returned the Access login redirect and the dedicated service-token request reached the application with HTTP 200. The Mac tunnel was not changed, and x86 remained stopped with the next-boot refresh requirement above.

The personal Mac

https://mac-browsers.cc-games.dev is an independent browser manager on the personal Mac (native Chrome; see the public repository), not the optional AWS Mac instance, with its own tunnel and Access application. Never run its connector with the AWS browser tunnel token: Cloudflare would load-balance between two machines' browser lists.

Terraform runs only on the jumpbox. After reviewing/applying the full plan, export terraform output -raw browser_manager_mac_env into a private environment file. The Mac's connector credential is stored in Secrets Manager as browser-manager-mac-tunnel-token; it is also sensitive Terraform state. Transfer that token and public environment file over the existing authenticated SSH connection, owned by the Mac user and mode 0600. Do not print tokens or place them in Git, command arguments, browser profiles or LaunchAgent XML. For later credential retrieval, the Mac's existing SSO administrator session can assume browser-manager-mac-connector, which can read only that connector secret. No AWS credentials are required by the running browser-manager or tunnel process.

The Mac still runs the application from an old checkout of this repository, project path /Users/ejcampbell/src/aws-browser-manager/browser-manager. Do not git pull it: this repository no longer contains the application, and pulling would delete the code its LaunchAgents run. The macOS installer keeps its installation tied to that exact project path ("Another checkout owns this installation; keep its fixed path"), so move to the public repository by putting its clone at the same path, in one sitting (the LaunchAgents keep running the old code until the installer restarts them):

mv ~/src/aws-browser-manager ~/src/aws-browser-manager.before-public
mkdir ~/src/aws-browser-manager
git clone https://github.com/ejc3/browser-manager ~/src/aws-browser-manager/browser-manager
cd ~/src/aws-browser-manager/browser-manager && npm ci --ignore-scripts && npm run build
node scripts/install-macos.mjs /absolute/private/browser-manager.env

Remove ~/src/aws-browser-manager.before-public once the Mac URL works.

File map

Area	Main files
Backend, providers, VPC, routing	`main.tf`, `.terraform.lock.hcl`, `variables.tf`, `runner-vpc.tf`
Admin hosts	`jumpbox.tf`, `jumpbox2.tf`, `jumpbox2-user-data.tf`
Metal dev boxes	`firecracker-dev.tf`, `x86-dev.tf`, `dev-user-data.tf`
Shared dev setup	`dev-instance-common.tf`, `dev-selfupdate.tf`, `dev-hop-key.tf`, `dev-ebs.tf`
Kids' environment	`nextjs-dev.tf`, `nextjs-user-data.tf`, `cloudflare.tf`
Shared I/O and burst compute	`io-box.tf`, `parallel-box.tf`, `parallel-box-watchdog.tf`, `scripts/parallel-box.sh`
GitHub runners and OIDC	`runner-autoscale.tf`, `runner-webhook-front.tf`, `runner-repos.tf` (per-repo tokens for the other repos' runners), `github-actions.tf`, `GITHUB-RUNNERS.md`
Recovery and monitoring	`backups.tf`, `backup-security.tf`, `security-monitoring.tf`, `security-alert-digest.tf` (security alerts reach `cost-alerts` through a digest Lambda: each finding mailed once, GuardDuty, CloudTrail audit and Access Analyzer at once, Security Hub and Inspector in a digest at most every 10 minutes, never more than one message a minute), `security-hub-automation.tf` (main-account control findings that are true by design, suppressed with the reason: no-NAT public IPs, GuardDuty and Inspector plans off for cost, the custom Config role, the nextjs-ct session policies, and key-only SSH on exactly the named groups until the Cloudflare VPN replaces it), `default-security-groups.tf` (every VPC's default security group, emptied), `modules/security-region/main.tf`, `cost-alerts.tf`, `fcvm-ec2-key-backup.tf`
Regional account defaults	`security-defaults.tf`, `security-regions.tf`, `modules/security-defaults/main.tf`
Free external-access findings	`security-external-access.tf`
Global S3 public-access defaults	`security-s3-account.tf`
The Dolphin member account: account and OU, its Terraform state bucket and the `dolphin-terraform` apply role that only nextjs-dev may assume (box-wide), Identity Center sign-in (owner admin, Steve read-only). Its own resources live in `dolphin-labs-hq/dolphin-aws`	`dolphin-account.tf`, `dolphin-bootstrap.tf`, `dolphin-identity-center.tf`
The astryx member account (Astryx sync): account and OU with a us-west-1 SCP, its Terraform state bucket, the `astryx-terraform` apply role (fcvm-arm pinned, and the owner's nextjs-ct container), `/infra/astryx`, and staging's hostname, tunnel and Access on `/native/login`. Its own resources live in `ejc3/astryx-infra`	`astryx-account.tf`, `astryx-stage.tf`
Private browser desktops (AWS and personal Mac)	`browser-manager.tf`, `browser-manager-mac.tf` (the app: github.com/ejc3/browser-manager)
Optional Mac	`mac-dev.tf`, `mac-dev-secrets.tf`, `mac-dev-teardown.tf`
SSH to EJ's MacBook and Colton's Mac through Cloudflare Access (`ssh mac`, `ssh colton-mac`): tunnels, Access, per-client service tokens, the clients' installer and the Mac-side connector setup	`macbook-ssh.tf`, `colton-mac-ssh.tf`, `mac-ssh-client.tf`, `scripts/mac-ssh-connector-setup.sh`
The metal boxes' box tools (pbox, gbox, wbox, codex-restart) kept equal to main daily, not only at boot	`metal-box-tools.tf`
The Macs' agent tools and the pinned Peekaboo	`scripts/mac-setup.sh`, `scripts/mac/` (`peekaboo-install.sh` holds the pin)
The dev accounts' agent instructions: AGENTS.md from ejc3/agents-md, and `~/.claude/local.md` from administration-only secrets (never in git)	`agents-md.tf`, `agent-local-md.tf`
Colton Games' production domains on Vercel (`cc-games.org`, canonical; `cc-games.net`, `cc-games.app`, `ccgames.app` and `colton-games.com` redirecting to it)	`vercel.tf`, `vercel-cc-games.tf`
Colton Games accounts (Google sign-in, site admins, browser push): per-environment secret containers, the generated `AUTH_SECRET`s and the Vercel variables written from them	`colton-games-accounts.tf` (keys, gate and bring-up order in its header)
tmux-scroll release pin, tag + sha256 (every aarch64 box; `fcvm-metal-x86` keeps its copy until the release has an x86_64 asset). t-claude is pinned (`local.tclaude_ref`) only on the jumpboxes; the metal boxes and nextjs-dev follow its `main`	`tmux-scroll.tf`, `scripts/admin-tmux-tclaude.sh`
Windows playtest box (g4dn.xlarge, Server 2025, persistent D: game disk, reachable from the dev boxes only, `wbox up` / `wbox down` / `wbox run` / `wbox launch`, stops after 1 idle hour)	`wbox.tf`, `scripts/wbox.sh`
Each member's own wbox, pbox and gbox (fixed private IPs, per-person keys, passwords and instance roles; `wbox` / `pbox` / `gbox` act on the member's own box when run from their account)	`member-boxes.tf`, `member-wbox.tf`, `member-pbox.tf`, `member-gbox.tf`, `scripts/wbox-member.sh`
Shared claw-master server (t4g.micro at 10.0.1.50 holding the Claude subscription logins; boxes authenticate with certificates, Macs and a Windows PC through a Cloudflare tunnel, each with an Access service token; no password, nothing inbound)	`claw-master-server.tf`, `claw-master-tunnel.tf`, `scripts/claw-master-login.sh --server`, `scripts/claw-master-enroll.sh`, `scripts/claw-master-mac-bundle.sh`, `scripts/claw-master-windows-setup.sh`
Codex refresh (weekly SSM association on the jumpboxes and metal boxes) and `codex-restart` (see running vs installed, restart deliberately)	`codex-update.tf`, `scripts/codex-restart.sh`
Starter user-level agent instructions (`~/.codex/AGENTS.md` with `~/.claude/CLAUDE.md` linked to it), created only for an account that has neither	`user-agents.tf`, `scripts/user-agents.md` (the text), `scripts/user-agents-seed.sh`
Games multiplayer (ECS match engines, `play.cc-games.org`, `play.cc-games.net` and `play.cc-games.app`)	`games-multiplayer.tf`, `games-multiplayer-bringup.tf`, `games-multiplayer-deploy.tf` (automatic deploys: poller, builds, releases, migrations), `games-multiplayer-edge.tf` (WAF, access logs, router autoscaling, health alarms), `games-multiplayer/` (launch, poller, release and sweeper functions, bring-up steps, buildspecs), `docs/games-multiplayer.md`
Imagine (`ejc3/imagine`: a collaborative editor's backend on ECS, zero tasks while unused, behind a host rule on the games load balancer at `imagine.play.cc-games.app`)	`imagine.tf` (bring-up order in its header), `imagine/scale.py` (the function that wakes, sleeps and rolls the service)
dolphin-films (`dolphin-labs-hq/dolphin-films`: a Next.js site on Vercel with Google sign-in and a Turso store; its credentials per environment, and its CI runners)	`dolphin-films.tf` (secret containers and readers; bring-up order in its header), `runner-repos.tf` and `runner-app.tf` (served under the `dolphin` label)
Dolphin Maps production (`maps.dolphin-labs.dev`, public, served by the Worker `dolphin-maps-prod`; its secrets are `workers-prod/dolphin-maps` in the Dolphin account)	`dolphin-labs-zone.tf`, `scripts/workers-prod-secrets.sh dolphin-maps`
Host access audit (reports opens of registered paths on hosts and any use of registered principals; the registry is private; rollout in "Host access audit" above)	`access-audit.tf`, `scripts/access-audit.py`
Staging and packages	`dev-staging-account.tf`, `dev-staging-bootstrap.tf`, `codeartifact.tf`

AGENTS.md contains the deeper operational constraints, nested-virtualization details, persistent-Spot recovery procedure, and live-system safety notes that an automation agent must read before changing the repository.

Name		Name	Last commit message	Last commit date
.claude		.claude
.github		.github
.greptile		.greptile
docs		docs
dolphin-maps-loop		dolphin-maps-loop
fcvm-coordinator-release		fcvm-coordinator-release
games-multiplayer		games-multiplayer
imagine		imagine
modules		modules
runner-app		runner-app
scripts		scripts
.gitignore		.gitignore
.terraform.lock.hcl		.terraform.lock.hcl
AGENTS.md		AGENTS.md
CLAUDE.md		CLAUDE.md
GITHUB-RUNNERS.md		GITHUB-RUNNERS.md
LICENSE		LICENSE
README.md		README.md
access-audit.tf		access-audit.tf
account-tags.tf		account-tags.tf
agent-local-md.tf		agent-local-md.tf
agent-session-sync.tf		agent-session-sync.tf
agents-md.tf		agents-md.tf
applied-status.tf		applied-status.tf
astryx-account.tf		astryx-account.tf
astryx-join.tf		astryx-join.tf
astryx-stage.tf		astryx-stage.tf
auto-reboot.tf		auto-reboot.tf
backup-restore-canary.tf		backup-restore-canary.tf
backup-security.tf		backup-security.tf
backups.tf		backups.tf
bedrock-haiku.tf		bedrock-haiku.tf
browser-manager-mac.tf		browser-manager-mac.tf
browser-manager.tf		browser-manager.tf
claude-remote-control.tf		claude-remote-control.tf
claw-master-cert-renew.tf		claw-master-cert-renew.tf
claw-master-client.tf		claw-master-client.tf
claw-master-dashboard-site.tf		claw-master-dashboard-site.tf
claw-master-dashboard.tf		claw-master-dashboard.tf
claw-master-moved.tf		claw-master-moved.tf
claw-master-server.tf		claw-master-server.tf
claw-master-tunnel.tf		claw-master-tunnel.tf
cloudflare-observability.tf		cloudflare-observability.tf
cloudflare-production.tf		cloudflare-production.tf
cloudflare.tf		cloudflare.tf
codeartifact.tf		codeartifact.tf
codex-remote-control.tf		codex-remote-control.tf
codex-update.tf		codex-update.tf
colin-pc-ssh.tf		colin-pc-ssh.tf
colton-games-accounts.tf		colton-games-accounts.tf
colton-games-moderation.tf		colton-games-moderation.tf
colton-mac-ssh.tf		colton-mac-ssh.tf
constellation-account.tf		constellation-account.tf
constellation-lattice-stage.tf		constellation-lattice-stage.tf
constellation-mcp-stage.tf		constellation-mcp-stage.tf
cost-alerts.tf		cost-alerts.tf
default-security-groups.tf		default-security-groups.tf
dev-ai-services.tf		dev-ai-services.tf
dev-auto-stop-lambda.tf		dev-auto-stop-lambda.tf
dev-diagnostics.tf		dev-diagnostics.tf
dev-ebs.tf		dev-ebs.tf
dev-hop-key.tf		dev-hop-key.tf
dev-instance-common.tf		dev-instance-common.tf
dev-selfupdate.tf		dev-selfupdate.tf
dev-staging-account.tf		dev-staging-account.tf
dev-staging-bootstrap.tf		dev-staging-bootstrap.tf
dev-user-data.tf		dev-user-data.tf
dolphin-account.tf		dolphin-account.tf
dolphin-alerts.tf		dolphin-alerts.tf
dolphin-bootstrap.tf		dolphin-bootstrap.tf
dolphin-family.tf		dolphin-family.tf
dolphin-films.tf		dolphin-films.tf
dolphin-identity-center.tf		dolphin-identity-center.tf
dolphin-labs-ci-logs.tf		dolphin-labs-ci-logs.tf
dolphin-labs-email.tf		dolphin-labs-email.tf
dolphin-labs-testflight.tf		dolphin-labs-testflight.tf
dolphin-labs-zone.tf		dolphin-labs-zone.tf
dolphin-maps-loop.tf		dolphin-maps-loop.tf
dolphin-maps-moderation.tf		dolphin-maps-moderation.tf
dolphin-redis.tf		dolphin-redis.tf
dolphin-region-scp.tf		dolphin-region-scp.tf
dolphin-relay.tf		dolphin-relay.tf
dolphin-security.tf		dolphin-security.tf
fcvm-dev.tf		fcvm-dev.tf
fcvm-dns-test.tf		fcvm-dns-test.tf
fcvm-ec2-key-backup.tf		fcvm-ec2-key-backup.tf
fcvm-fleet-account.tf		fcvm-fleet-account.tf
fcvm-fleet-ci.tf		fcvm-fleet-ci.tf
fcvm-fleet.tf		fcvm-fleet.tf
firecracker-dev.tf		firecracker-dev.tf
games-multiplayer-bringup.tf		games-multiplayer-bringup.tf
games-multiplayer-deploy.tf		games-multiplayer-deploy.tf
games-multiplayer-edge.tf		games-multiplayer-edge.tf
games-multiplayer-observe.tf		games-multiplayer-observe.tf
games-multiplayer.tf		games-multiplayer.tf
gamesworthwatching-zone.tf		gamesworthwatching-zone.tf
github-actions.tf		github-actions.tf
github-admin-tokens.tf		github-admin-tokens.tf
github-ami-builder.tf		github-ami-builder.tf
gpu-box.tf		gpu-box.tf

Latest commit

History

Folders and files

Repository files navigation