Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/lambda-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,9 @@ jobs:
- name: Metal secondary IPv4 persists across boots without restarting networking
run: python3 -S -B scripts/test-fcvm-secondary-ipv4.py

- name: ARM public address cutover and operator-supplied script destinations
run: python3 -S -B scripts/test-fcvm-arm-address.py

- name: Jumpbox native Claude installation boundaries
run: python3 -S -B scripts/test-jumpbox-native-claude.py

Expand Down
4 changes: 2 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -2246,7 +2246,7 @@ aws ec2 start-instances --instance-ids $INSTANCE_ID --region us-west-1
aws ec2 delete-volume --volume-id $CURRENT_VOL --region us-west-1

# 6. Re-associate EIP (terraform loses the association on recreate)
# ARM: eipalloc-034a515771765d101, x86: eipalloc-0173c9b5e3d294cc5
# ARM: Terraform owns its address (aws_eip_association.firecracker_dev_next), x86: eipalloc-0173c9b5e3d294cc5
EIP_ALLOC="eipalloc-0173c9b5e3d294cc5" # x86
aws ec2 associate-address --instance-id $INSTANCE_ID --allocation-id $EIP_ALLOC --region us-west-1

Expand All @@ -2260,7 +2260,7 @@ echo "Done!"

**Persistent Volume IDs** (don't delete these!):
- ARM (fcvm-metal-arm): root volume is built by terraform and changes on every move
(`terraform state show 'aws_instance.firecracker_dev[0]'`); EIP: `184.72.40.255` (`eipalloc-034a515771765d101`)
(`terraform state show 'aws_instance.firecracker_dev[0]'`); address: `terraform output firecracker_dev_public_ip`
- x86 (fcvm-metal-x86): `vol-071f114b67441e776`, EIP: `50.18.109.164` (`eipalloc-0173c9b5e3d294cc5`)

**When to run the manual swap** (x86): After `terraform apply` creates a new instance (you'll see a new instance ID in the output). Check if data is missing, then run the swap.
Expand Down
25 changes: 19 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -406,10 +406,22 @@ ssh nextjs
ssh io
```

The ARM host's second inbound address is reported by
`firecracker_dev_public_ip_next`. Terraform persists its secondary private
address in netplan and adds it live without restarting networking. The original
address remains the outbound address until a separate cutover.
The ARM host's inbound and outbound public address is reported by
`firecracker_dev_public_ip`. Terraform retains the secondary private address in
netplan after the public-address cutover.

Before releasing a previous ARM Elastic IP, update the active SSH aliases on every
running dev host that carries them. Terraform publishes new setup scripts but does
not rewrite an already-running host's `/home/ubuntu/.ssh/config.d-devhop`. Back up
that file, then atomically replace only the `HostName` for `fcvm-arm fcvm-metal-arm`
with the new address from the reviewed plan, preserving its owner and `0600` mode.
Verify `ssh -G fcvm-arm` and `ssh -G fcvm-metal-arm` both select the new address,
then verify an actual `ssh fcvm-arm hostname` from each running host before the old
address is released. This step updates the alias file only; running the full
`dev-selfupdate` script can restart unrelated services. Keep stopped dev hosts
stopped until the apply has published the updated S3 setup scripts; their next boot
then installs the new aliases. I/O and parallel boxes normally carry only inbound
hop authorization, so check whether they have an alias file before including them.


The aliases use Elastic IPs for the three long-lived `us-west-1` boxes and fixed private
Expand Down Expand Up @@ -893,8 +905,9 @@ alone lets it reconnect.

1. Add its row to `local.mac_ssh_macs` and a `<alias>-ssh.tf` like `colin-pc-ssh.tf`; claim a port block for it
(`ports claim <alias>`). After apply, fcvm-arm has `~/.ssh/to-<alias>`.
2. On fcvm-arm: `scripts/windows-pc-bootstrap.sh print <alias> PORT` prints the one script the person at the PC pastes
into PowerShell opened with **Run as administrator** (`scripts/windows/bootstrap.ps1` with this PC's values; publish
2. On fcvm-arm: `FCVM_HOST=<current-arm-public-address> scripts/windows-pc-bootstrap.sh print <alias> PORT` prints
the one script the person at the PC pastes. Use the `firecracker_dev_public_ip` Terraform output for `FCVM_HOST`.
Paste into PowerShell opened with **Run as administrator** (`scripts/windows/bootstrap.ps1` with this PC's values; publish
it behind a Copy button). It installs OpenSSH closed to the network, authorizes fcvm-arm's key, and keeps a reverse
tunnel, as `pcboot`, to fcvm-arm's `127.0.0.1:PORT`. It ends by printing the PC's own key: authorize it for that
tunnel only with `scripts/windows-pc-bootstrap.sh allow <alias> PORT '<key line>'` (it writes pcboot's key file, with sudo).
Expand Down
2 changes: 1 addition & 1 deletion dev-hop-key.tf
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ resource "aws_secretsmanager_secret_version" "dev_hop" {
locals {
dev_hop_targets = {
for k, v in {
fcvm-arm = { tag = "fcvm-metal-arm", user = "ubuntu", enabled = var.enable_firecracker_instance, ip = var.enable_firecracker_instance ? aws_eip.firecracker_dev[0].public_ip : "" }
fcvm-arm = { tag = "fcvm-metal-arm", user = "ubuntu", enabled = var.enable_firecracker_instance, ip = var.enable_firecracker_instance ? aws_eip.firecracker_dev_next[0].public_ip : "" }
Comment thread
ejc3 marked this conversation as resolved.
fcvm-x86 = { tag = "fcvm-metal-x86", user = "ubuntu", enabled = var.enable_x86_dev_instance, ip = var.enable_x86_dev_instance ? aws_eip.x86_dev[0].public_ip : "" }
nextjs = { tag = "nextjs-dev", user = "ubuntu", enabled = var.enable_nextjs_dev, ip = var.enable_nextjs_dev ? aws_eip.nextjs_dev[0].public_ip : "" }
# Same box as nextjs, but the dolphin-labs checkout lives in ejc3's account, so this hop
Expand Down
45 changes: 12 additions & 33 deletions firecracker-dev.tf
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,8 @@ resource "aws_network_interface" "firecracker_dev" {
subnet_id = local.subnet_ids_by_az[var.firecracker_availability_zone]
security_groups = [aws_security_group.firecracker_dev[0].id]
ipv6_prefix_count = 1
# One secondary private IPv4 address, for the second public address (aws_eip.firecracker_dev_next).
# One secondary private IPv4 address, which held the second public address during the switch-over.
# Nothing public maps to it now; it and its State Manager association are removed in a follow-up.
private_ips_count = 1

# The host also gets its own IPv6 address, not just the /80 its VMs route, but CreateNetworkInterface
Expand Down Expand Up @@ -288,26 +289,10 @@ resource "aws_instance" "firecracker_dev" {
# Elastic IP for static address
# ============================================

resource "aws_eip" "firecracker_dev" {
count = var.enable_firecracker_instance ? 1 : 0
domain = "vpc"

tags = {
Name = "fcvm-metal-arm-eip"
}
}

resource "aws_eip_association" "firecracker_dev" {
count = var.enable_firecracker_instance ? 1 : 0
instance_id = aws_instance.firecracker_dev[0].id
allocation_id = aws_eip.firecracker_dev[0].id
}

# A second public address, to move the box to (owner, 2026-10-25: "add a second ip to this box and we will
# switch over"). It maps to the ENI's secondary private address, so inbound traffic reaches the box on either
# address while outbound traffic still leaves from the primary private address, through aws_eip.firecracker_dev.
# Every allowlist of the box's address (fcvm-dns-test.tf, gpu-box.tf, ohio.tf, runner-vpc.tf) admits both. The switch is a
# later change: associate this address with the primary private address and release the old one.
# The box's public address (owner, 2026-10-25: "add a second ip to this box and we will switch over").
# It is the second address #666 added, now on the primary private address, so inbound and outbound
# traffic both use it; the address it replaced is released. The resource keeps its `_next` name:
# renaming it in the apply that releases the old address would allocate a new one.
resource "aws_eip" "firecracker_dev_next" {
count = var.enable_firecracker_instance ? 1 : 0
domain = "vpc"
Expand All @@ -321,7 +306,9 @@ resource "aws_eip_association" "firecracker_dev_next" {
count = var.enable_firecracker_instance ? 1 : 0
allocation_id = aws_eip.firecracker_dev_next[0].id
network_interface_id = aws_network_interface.firecracker_dev[0].id
private_ip_address = one(setsubtract(aws_network_interface.firecracker_dev[0].private_ips, [aws_network_interface.firecracker_dev[0].private_ip]))
private_ip_address = aws_network_interface.firecracker_dev[0].private_ip
# The address moves from the secondary private address in this apply.
allow_reassociation = true
}

# Persist the address in netplan for ordinary reboots and add it live without reconfiguring
Expand Down Expand Up @@ -349,11 +336,8 @@ resource "aws_ssm_association" "firecracker_dev_secondary_ipv4" {
}

locals {
# Both public addresses of fcvm-metal-arm, current first. Allowlists of the box use this list.
firecracker_dev_public_ips = concat(
[for e in aws_eip.firecracker_dev : e.public_ip],
[for e in aws_eip.firecracker_dev_next : e.public_ip],
)
# fcvm-metal-arm's public address. Allowlists of the box use this list.
firecracker_dev_public_ips = [for e in aws_eip.firecracker_dev_next : e.public_ip]
}

# Output the instance ID and connection command
Expand All @@ -369,15 +353,10 @@ output "firecracker_dev_availability_zone" {

output "firecracker_dev_public_ip" {
description = "Public IP of Firecracker dev instance (Elastic IP)"
value = var.enable_firecracker_instance ? aws_eip.firecracker_dev[0].public_ip : null
}

output "firecracker_dev_public_ip_next" {
description = "Second public IP of Firecracker dev instance (Elastic IP on the secondary private address)"
value = var.enable_firecracker_instance ? aws_eip.firecracker_dev_next[0].public_ip : null
}

output "firecracker_dev_ssh_command" {
description = "Command to connect to Firecracker dev instance via SSH"
value = var.enable_firecracker_instance ? "ssh -i ~/.ssh/${var.firecracker_key_name} ubuntu@${aws_eip.firecracker_dev[0].public_ip}" : null
value = var.enable_firecracker_instance ? "ssh -i ~/.ssh/${var.firecracker_key_name} ubuntu@${aws_eip.firecracker_dev_next[0].public_ip}" : null
}
1 change: 0 additions & 1 deletion ohio.tf
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,6 @@ resource "aws_security_group" "ohio_runner" {
self = true
cidr_blocks = [
"${aws_eip.jumpbox[0].public_ip}/32",
"${aws_eip.firecracker_dev[0].public_ip}/32",
"${aws_eip.firecracker_dev_next[0].public_ip}/32",
"${aws_eip.x86_dev[0].public_ip}/32",
]
Expand Down
3 changes: 1 addition & 2 deletions runner-vpc.tf
Original file line number Diff line number Diff line change
Expand Up @@ -141,8 +141,7 @@ resource "aws_security_group" "runner" {
self = true # fcvm runner-to-runner only
cidr_blocks = [
"${aws_eip.jumpbox[0].public_ip}/32", # jumpbox (management host)
"${aws_eip.firecracker_dev[0].public_ip}/32", # fcvm-metal-arm dev server
"${aws_eip.firecracker_dev_next[0].public_ip}/32", # fcvm-metal-arm, second address
"${aws_eip.firecracker_dev_next[0].public_ip}/32", # fcvm-metal-arm dev server
"${aws_eip.x86_dev[0].public_ip}/32", # fcvm-metal-x86 dev server
]
description = "SSH from fcvm runners (self) + operator EIPs (jumpbox, dev servers); SSM elsewhere"
Expand Down
15 changes: 14 additions & 1 deletion scripts/claw-master-login.sh
Original file line number Diff line number Diff line change
Expand Up @@ -15,12 +15,25 @@
# the native Claude login or between profiles.
set -euo pipefail

HOST=${FCVM_HOST:-184.72.40.255} # fcvm-metal-arm's Elastic IP
# fcvm-metal-arm's address is not kept in this repository: FCVM_HOST, else the HostName of the user's `fcvm-arm` ssh
# alias (ssh -G prints the alias itself for a name with no entry, which is not an address), else an error.
fcvm_host() {
local h=${FCVM_HOST:-}
if [ -z "$h" ]; then
h=$(ssh -G fcvm-arm 2>/dev/null | awk '$1 == "hostname" { print $2; exit }' || true)
[ "$h" != fcvm-arm ] || h=""
fi
[ -n "$h" ] || { echo "no address for fcvm-metal-arm: set FCVM_HOST, or give the ssh alias fcvm-arm a HostName in ~/.ssh/config" >&2; exit 2; }
printf '%s\n' "$h"
}

KEY=${FCVM_KEY:-$HOME/.ssh/fcvm-ec2}
SERVER=0
if [ "${1:-}" = "--server" ]; then
SERVER=1; shift
HOST=${CLAUDE_MASTER_SERVER_HOST:-10.0.1.50} # claw-master-server.tf: a fixed private address
else
HOST=$(fcvm_host)
fi

profiles=("$@")
Expand Down
13 changes: 12 additions & 1 deletion scripts/claw-master-mac-setup.sh
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,17 @@
set -euo pipefail

REGION=${AWS_REGION:-us-west-1}
FCVM=${FCVM_HOST:-184.72.40.255} # fcvm-metal-arm's Elastic IP
# fcvm-metal-arm's address is not kept in this repository: FCVM_HOST, else the HostName of the user's `fcvm-arm` ssh
# alias (ssh -G prints the alias itself for a name with no entry, which is not an address), else an error.
fcvm_host() {
local h=${FCVM_HOST:-}
if [ -z "$h" ]; then
h=$(ssh -G fcvm-arm 2>/dev/null | awk '$1 == "hostname" { print $2; exit }' || true)
[ "$h" != fcvm-arm ] || h=""
fi
[ -n "$h" ] || { echo "no address for fcvm-metal-arm: set FCVM_HOST, or give the ssh alias fcvm-arm a HostName in ~/.ssh/config" >&2; exit 2; }
printf '%s\n' "$h"
}
KEY=${FCVM_KEY:-$HOME/.ssh/fcvm-ec2}

# Pins. Keep CM_TAG equal to claw_master_tag in claw-master-server.tf (a test enforces it) and
Expand All @@ -36,6 +46,7 @@ CFD_SHA256_DARWIN_ARM64_BIN=5472c1a01c84bc31b3021056a73b4e5774ddddefc572124ea8fd

ALIAS=${1:-}
[[ $ALIAS =~ ^[a-z][a-z0-9-]{0,30}$ ]] || { echo "usage: claw-master-mac-setup.sh ALIAS (mac: the ssh alias on fcvm-arm)" >&2; exit 2; }
FCVM=$(fcvm_host)

# Run something on the Mac: jumpbox -> fcvm -> (Cloudflare Access) -> Mac. stdin is passed through.
on_mac() { ssh -o BatchMode=yes -o ConnectTimeout=10 -i "$KEY" "ubuntu@$FCVM" "ssh -o BatchMode=yes -o ConnectTimeout=10 $ALIAS $*"; }
Expand Down
13 changes: 12 additions & 1 deletion scripts/mac-ssh-connector-setup.sh
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,17 @@
set -euo pipefail

REGION=${AWS_REGION:-us-west-1}
FCVM=${FCVM_HOST:-184.72.40.255} # fcvm-metal-arm's Elastic IP
# fcvm-metal-arm's address is not kept in this repository: FCVM_HOST, else the HostName of the user's `fcvm-arm` ssh
# alias (ssh -G prints the alias itself for a name with no entry, which is not an address), else an error.
fcvm_host() {
local h=${FCVM_HOST:-}
if [ -z "$h" ]; then
h=$(ssh -G fcvm-arm 2>/dev/null | awk '$1 == "hostname" { print $2; exit }' || true)
[ "$h" != fcvm-arm ] || h=""
fi
[ -n "$h" ] || { echo "no address for fcvm-metal-arm: set FCVM_HOST, or give the ssh alias fcvm-arm a HostName in ~/.ssh/config" >&2; exit 2; }
printf '%s\n' "$h"
}
KEY=${FCVM_KEY:-$HOME/.ssh/fcvm-ec2}

# Pins: keep equal to scripts/claw-master-mac-setup.sh (scripts/test-mac-ssh-client.py enforces it).
Expand Down Expand Up @@ -75,6 +85,7 @@ case "$ALIAS" in
;;
*) usage ;;
esac
FCVM=$(fcvm_host)

MAC_SSH="$ALIAS"
if [ -n "$BOOTSTRAP_PORT" ]; then
Expand Down
105 changes: 105 additions & 0 deletions scripts/test-fcvm-arm-address.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
#!/usr/bin/env python3
"""fcvm-metal-arm has one public address, on its primary private address, and nothing
references the address it replaced (released, so whoever AWS gives it next gets no access)."""
import os
from pathlib import Path
import re
import subprocess
import tempfile
import unittest

ROOT = Path(__file__).resolve().parent.parent


class ArmAddress(unittest.TestCase):
def test_the_replaced_address_is_released_and_unreferenced(self):
for tf in sorted(ROOT.glob('*.tf')):
text = tf.read_text()
self.assertNotIn('resource "aws_eip" "firecracker_dev" {', text, tf.name)
self.assertNotRegex(text, r'aws_eip\.firecracker_dev(\[|\s*:)', tf.name)

def test_the_address_is_on_the_primary_private_address(self):
tf = (ROOT / 'firecracker-dev.tf').read_text()
assoc = re.search(r'resource "aws_eip_association" "firecracker_dev_next" \{.*?\n\}', tf, re.S).group()
self.assertIn('private_ip_address = aws_network_interface.firecracker_dev[0].private_ip\n', assoc)
self.assertIn('allow_reassociation = true', assoc)
self.assertIn('firecracker_dev_public_ips = [for e in aws_eip.firecracker_dev_next : e.public_ip]', tf)


IPV4 = re.compile(r'\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b')
# The scripts that reach fcvm-metal-arm by address, with arguments that get each one past its argument checks.
HOST_SCRIPTS = {
'claw-master-login.sh': ['claude-test'],
'claw-master-mac-setup.sh': ['mac'],
'mac-ssh-connector-setup.sh': ['mac'],
'windows-pc-setup.sh': ['colin-pc'],
'windows-pc-bootstrap.sh': ['print', 'test-pc', '21899'],
}


class HostComesFromTheOperator(unittest.TestCase):
"""The box's address is not in the repository: FCVM_HOST, else the `fcvm-arm` ssh alias, else an error.
A default literal would point a script at a stranger once the address is released."""

def run_script(self, home, name, alias_host=None, **env):
"""Run a script with stub ssh and aws on PATH. `ssh -G fcvm-arm` answers with alias_host as its hostname,
or with the alias itself, which is what ssh prints for a name that has no config entry."""
stubs = Path(home) / 'stubs'
stubs.mkdir(exist_ok=True)
(stubs / 'ssh').write_text(
'#!/bin/sh\n'
'printf "%s\\n" "$*" >> "$HOME/ssh.log"\n'
f'if [ "$1" = -G ]; then echo "hostname {alias_host or "fcvm-arm"}"; exit 0; fi\n'
'exit 0\n')
(stubs / 'aws').write_text('#!/bin/sh\necho aws >> "$HOME/aws.log"\nexit 99\n')
for stub in stubs.iterdir():
stub.chmod(0o755)
ssh = Path(home) / '.ssh'
ssh.mkdir(exist_ok=True)
if not (ssh / 'to-test-pc.pub').exists():
subprocess.run(['ssh-keygen', '-q', '-t', 'ed25519', '-N', '', '-f', str(ssh / 'to-test-pc')], check=True)
base = {k: v for k, v in os.environ.items() if k != 'FCVM_HOST'}
return subprocess.run(['bash', str(ROOT / 'scripts' / name), *HOST_SCRIPTS[name]], capture_output=True, text=True,
timeout=60, stdin=subprocess.DEVNULL,
env=dict(base, HOME=home, PATH=f'{stubs}:/usr/bin:/bin', FCVM_KEY='/nonexistent', **env))

def test_no_script_names_an_address_for_the_host(self):
for name in HOST_SCRIPTS:
for n, line in enumerate((ROOT / 'scripts' / name).read_text().splitlines(), 1):
if 'FCVM_HOST' in line or re.match(r'\s*(FCVM|HOST|ip)=', line):
public = [a for a in IPV4.findall(line) if not a.startswith(('10.', '127.'))]
self.assertEqual(public, [], f'{name}:{n}')

def test_without_FCVM_HOST_or_the_alias_a_script_stops_and_names_both(self):
for name in HOST_SCRIPTS:
with tempfile.TemporaryDirectory() as home:
r = self.run_script(home, name)
self.assertEqual(r.returncode, 2, f'{name}: {r.stdout}{r.stderr}')
self.assertIn('FCVM_HOST', r.stderr, name)
if name != 'windows-pc-bootstrap.sh': # it runs on the box itself, where no alias points at the box
self.assertIn('fcvm-arm', r.stderr, name)
self.assertFalse((Path(home) / 'aws.log').exists(), f'{name} reached aws without a host')
ssh_log = Path(home) / 'ssh.log'
self.assertNotIn('ubuntu@', ssh_log.read_text() if ssh_log.exists() else '', name)

def test_the_alias_hostname_is_used_and_FCVM_HOST_wins_over_it(self):
with tempfile.TemporaryDirectory() as home:
r = self.run_script(home, 'claw-master-login.sh', alias_host='203.0.113.9')
self.assertEqual(r.returncode, 0, r.stdout + r.stderr)
self.assertIn('ubuntu@203.0.113.9', (Path(home) / 'ssh.log').read_text())
with tempfile.TemporaryDirectory() as home:
r = self.run_script(home, 'claw-master-login.sh', alias_host='203.0.113.9', FCVM_HOST='203.0.113.20')
self.assertEqual(r.returncode, 0, r.stdout + r.stderr)
log = (Path(home) / 'ssh.log').read_text()
self.assertIn('ubuntu@203.0.113.20', log)
self.assertNotIn('203.0.113.9', log)

def test_the_bootstrap_paste_uses_FCVM_HOST(self):
with tempfile.TemporaryDirectory() as home:
r = self.run_script(home, 'windows-pc-bootstrap.sh', FCVM_HOST='203.0.113.7')
self.assertEqual(r.returncode, 0, r.stderr)
self.assertIn('pcboot@203.0.113.7', r.stdout)


if __name__ == '__main__':
unittest.main(buffer=True)
Loading
Loading