EC2: VM-backed instances (stage 1)

.github/workflows/ci.yml

@@ -47,6 +47,32 @@ jobs:
             GOOS=${p%/*} GOARCH=${p#*/} CGO_ENABLED=0 go build -o /dev/null .
           done
 
+  vm:
+    # Boots a real virtual machine (the Ubuntu cloud image under QEMU/KVM inside a
+    # Docker container) through the EC2 API and checks SSH, cloud-init user data,
+    # the metadata service, stop/start, security groups and cleanup. Only this
+    # test runs here; the rest of the suite is in the go job.
+    runs-on: ubuntu-latest
+    timeout-minutes: 45
+    defaults:
+      run:
+        working-directory: cli
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-go@v5
+        with:
+          go-version-file: cli/go.mod
+      - name: Enable KVM
+        run: |
+          echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm4all.rules
+          sudo udevadm control --reload-rules
+          sudo udevadm trigger --name-match=kvm
+          ls -l /dev/kvm
+      - name: VM instances
+        run: go test -count=1 -timeout 40m -v -run 'TestVMInstanceLifecycle|TestVMDisksImagesAndBackup' ./internal/svc/ec2
+        env:
+          HC_TEST_VM: "1"
+
   console:
     runs-on: ubuntu-latest
     defaults:

CHANGELOG.md

@@ -17,6 +17,14 @@
 - Cognito user pools support `USER_SRP_AUTH` (Amplify, amazon-cognito-identity-js, pycognito), `ForgotPassword`/`ConfirmForgotPassword` and `AdminResetUserPassword`. Existing users need their password set again (or one `USER_PASSWORD_AUTH` sign-in) before SRP works.
 - CloudFormation updates behave like AWS: queues, topics, parameters, functions, roles, tables, alarms and other resources change in place, replacements create the new resource before deleting the old one, and a failed update rolls back to the previous template (`UPDATE_ROLLBACK_COMPLETE`, `ContinueUpdateRollback`, `DisableRollback`). A replacement that keeps a custom resource name now fails, as in AWS.
 
+### EC2
+- VM instances: `ami-ubuntu-24-04-vm` and `ami-debian-12-vm` boot the official cloud image as a QEMU virtual machine with its own kernel, systemd and cloud-init. The VM runs inside a container attached to the instance's VPC network with its private IP, so DNS, the metadata service, security groups and published ports work as for container instances; passt gives the guest the container's address and forwards inbound ports. Key pairs, user data (scripts and `#cloud-config`), the root block device size, stop/start/reboot, console output and terminate work through the same EC2 API, CLI, console and Terraform.
+- Guests use KVM when the Docker host has `/dev/kvm` and are emulated (slower) otherwise; instances report `virtualization: "kvm"` or `"emulated"`. `DescribeImages` and `DescribeInstances` report `Hypervisor: kvm` for VM images and instances. Cloud images are pinned to a release, verified by checksum and downloaded once into a Docker volume; the runner image is built locally on first use.
+- VM disks: extra EBS volumes (`BlockDeviceMappings`, `AttachVolume`, `DetachVolume`) are virtio disks in the guest; attaching or detaching reboots the guest because Docker cannot add mounts to a running container. `CreateSnapshot` and `CreateImage` flatten a VM's root disk into a standalone copy, instances launch from such images, and `homecloud backup` archives a VM's root disk as a standalone image. After a restore, a VM instance whose container is gone is recreated from its disk and left stopped.
+- VM containers use Docker's default seccomp profile plus `unshare`, `mount`, `umount2` and `pivot_root` (what passt's sandbox needs) instead of an unconfined one; AppArmor is unconfined (`HC_VM_APPARMOR` names a profile instead).
+- VM instances have run-command (through qemu-guest-agent, installed by cloud-init), the browser terminal (the guest's serial console; log in with a password set by user data) and CloudWatch `HC/EC2` metrics measured inside the guest (CPU, memory, network, disk, process count), all under the same IAM actions as container instances.
+- CI boots a VM with KVM on every change.
+
 ## 0.3.0 (2026-09-30)
 
 Every service in the console now speaks the AWS protocols, security groups filter traffic inside a VPC, and there's a demo console on the website.

README.md

@@ -11,7 +11,7 @@
 </p>
 
 - **Your AWS code, unchanged.** HomeCloud speaks the AWS wire protocols (SigV4, awsJson, awsQuery, REST). Set `AWS_ENDPOINT_URL` and the AWS CLI, boto3 and the Terraform AWS provider work against it, with IAM policies enforced as on AWS.
-- **Real compute, not mocks.** Lambda runs on AWS's official runtime images, RDS is a real PostgreSQL/MySQL/MariaDB, S3 is MinIO, EC2 instances are containers you can shell into, load balancers are nginx, security groups are iptables rules. Your integration tests hit the same kind of thing production does.
+- **Real compute, not mocks.** Lambda runs on AWS's official runtime images, RDS is a real PostgreSQL/MySQL/MariaDB, S3 is MinIO, EC2 instances are containers you can shell into or full VMs under QEMU/KVM, load balancers are nginx, security groups are iptables rules. Your integration tests hit the same kind of thing production does.
 - **See what happened.** A web console modeled on AWS's, CloudWatch logs and metrics for every resource, and a CloudTrail record of every AWS API call, so a failed test run can be inspected instead of guessed at.
 
 Built for **developers who want a real AWS-compatible target for local development and CI**. It also suits **college labs** teaching AWS without accounts or bills, and **small teams and homelabs** that want AWS tooling on their own hardware.
@@ -93,7 +93,7 @@ Run `homecloud aws-env` to get the environment variables. For Terraform, point t
 
 | Service | AWS equivalent | Status | Notes |
 | --- | --- | --- | --- |
-| Compute | EC2, EBS, AMIs | AWS API | Instances are **containers**, not VMs (VM-backed instances are in progress, [#3](https://github.com/solinode/homecloud/issues/3)). Key pairs, user data, volumes, snapshots, launch templates, Elastic IPs (records only), IMDSv1/v2, browser shell |
+| Compute | EC2, EBS, AMIs | AWS API | Instances are **containers**; `ami-ubuntu-24-04-vm` and `ami-debian-12-vm` boot real **virtual machines** (QEMU, KVM when the host has `/dev/kvm`, emulated otherwise) on the same VPC networking, with extra volumes, snapshots, images, run-command and a serial console. Key pairs, user data, volumes, snapshots, launch templates, Elastic IPs (records only), IMDSv1/v2, browser shell |
 | Auto Scaling | EC2 Auto Scaling | AWS API | Target tracking on CPU; scheduled actions and lifecycle hooks are not implemented |
 | Networking | VPC, security groups | AWS API | Security groups enforced inside the VPC; network ACLs recorded, not enforced; no peering; IPv4 only |
 | Load balancing | ELB v2 | AWS API | Application load balancers (HTTP/HTTPS, path/host rules). No network load balancers |
@@ -184,7 +184,7 @@ HomeCloud needs the Docker socket, which is root-equivalent on the host: treat H
 
 ## Roadmap
 
-- **In progress:** VM-backed instances with QEMU/KVM ([#3](https://github.com/solinode/homecloud/issues/3)), a container image for one-command starts
+- **In progress:** a container image for one-command starts
 - **Planned:** multi-node clusters ([#55](https://github.com/solinode/homecloud/issues/55)), edge compute and hardware integrations ([#56](https://github.com/solinode/homecloud/issues/56))
 
 See the [open issues](https://github.com/solinode/homecloud/issues) for everything planned.

cli/internal/runtime/docker.go

@@ -74,27 +74,30 @@ type Mount struct {
 }
 
 type RunSpec struct {
-	Name       string
-	Image      string
-	Cmd        []string
-	Entrypoint []string
-	Env        map[string]string
-	Labels     map[string]string
-	NanoCPUs   int64
-	MemoryMB   int64
-	Ports      []Port
-	Mounts     []Mount
-	Network    string
-	IP         string
-	Aliases    []string
-	Restart    string // "", "unless-stopped", "always"
-	WorkingDir string
-	User       string // run as this user (e.g. "0"); empty uses the image's default
-	Hostname   string
-	DNS        []string // upstream servers for Docker's embedded DNS
-	ExtraHosts []string // "name:ip" entries for /etc/hosts ("host-gateway" is the Docker host)
-	CapAdd     []string // extra Linux capabilities (e.g. NET_ADMIN)
-	Start      bool
+	Name        string
+	Image       string
+	Cmd         []string
+	Entrypoint  []string
+	Env         map[string]string
+	Labels      map[string]string
+	NanoCPUs    int64
+	MemoryMB    int64
+	Ports       []Port
+	Mounts      []Mount
+	Network     string
+	IP          string
+	Aliases     []string
+	Restart     string // "", "unless-stopped", "always"
+	WorkingDir  string
+	User        string // run as this user (e.g. "0"); empty uses the image's default
+	Hostname    string
+	DNS         []string // upstream servers for Docker's embedded DNS
+	ExtraHosts  []string // "name:ip" entries for /etc/hosts ("host-gateway" is the Docker host)
+	CapAdd      []string // extra Linux capabilities (e.g. NET_ADMIN)
+	Devices     []string // host devices passed through (e.g. /dev/kvm)
+	SecurityOpt []string // e.g. seccomp=unconfined
+	NoFile      int64    // raise the open-file limit (soft and hard) to this; 0 keeps the default
+	Start       bool
 }
 
 // HostAlias makes the Docker host reachable from containers as host.docker.internal
@@ -159,6 +162,13 @@ func (d *Docker) Run(ctx context.Context, s RunSpec) (string, error) {
 		hc.CPUPeriod, hc.CPUQuota = 100000, s.NanoCPUs/10000
 	}
 	hc.CapAdd = s.CapAdd
+	hc.SecurityOpt = s.SecurityOpt
+	if s.NoFile > 0 {
+		hc.Ulimits = []docker.ULimit{{Name: "nofile", Soft: s.NoFile, Hard: s.NoFile}}
+	}
+	for _, dev := range s.Devices {
+		hc.Devices = append(hc.Devices, docker.Device{PathOnHost: dev, PathInContainer: dev, CgroupPermissions: "rwm"})
+	}
 	if s.Restart != "" {
 		hc.RestartPolicy = docker.RestartPolicy{Name: s.Restart}
 	}

cli/internal/runtime/netns.go

@@ -14,16 +14,24 @@ import (
 
 // BuildImage builds tag from a single Dockerfile unless the image already exists.
 func (d *Docker) BuildImage(ctx context.Context, tag, dockerfile string) error {
+	return d.BuildImageFiles(ctx, tag, map[string][]byte{"Dockerfile": []byte(dockerfile)})
+}
+
+// BuildImageFiles builds tag from a build context (path -> content, including
+// the Dockerfile) unless the image already exists.
+func (d *Docker) BuildImageFiles(ctx context.Context, tag string, files map[string][]byte) error {
 	if _, err := d.C.InspectImage(tag); err == nil {
 		return nil
 	}
 	var buf bytes.Buffer
 	tw := tar.NewWriter(&buf)
-	if err := tw.WriteHeader(&tar.Header{Name: "Dockerfile", Mode: 0o644, Size: int64(len(dockerfile)), ModTime: time.Now()}); err != nil {
-		return err
-	}
-	if _, err := tw.Write([]byte(dockerfile)); err != nil {
-		return err
+	for name, content := range files {
+		if err := tw.WriteHeader(&tar.Header{Name: name, Mode: 0o644, Size: int64(len(content)), ModTime: time.Now()}); err != nil {
+			return err
+		}
+		if _, err := tw.Write(content); err != nil {
+			return err
+		}
 	}
 	if err := tw.Close(); err != nil {
 		return err

cli/internal/runtime/oneshot.go

@@ -0,0 +1,34 @@
+package runtime
+
+import (
+	"context"
+	"fmt"
+	"strings"
+	"time"
+)
+
+// RunOnce runs a short-lived container to completion and returns its output.
+// A non-zero exit is an error (with the output in it). The container is removed.
+func (d *Docker) RunOnce(ctx context.Context, s RunSpec) (string, error) {
+	s.Start = false
+	if s.Labels == nil {
+		s.Labels = Labels("ec2", "helper", nil)
+	}
+	id, err := d.Run(ctx, s)
+	if err != nil {
+		return "", err
+	}
+	defer func() { _ = d.Remove(id) }()
+	if err := d.C.StartContainerWithContext(id, nil, ctx); err != nil {
+		return "", fmt.Errorf("start container: %w", err)
+	}
+	code, err := d.C.WaitContainerWithContext(id, ctx)
+	out, _ := d.Logs(id, 200, time.Time{})
+	if err != nil {
+		return out, err
+	}
+	if code != 0 {
+		return out, fmt.Errorf("exit %d: %s", code, strings.TrimSpace(out))
+	}
+	return out, nil
+}

cli/internal/server/server.go

@@ -178,6 +178,7 @@ func Run(ctx context.Context, cfg core.Config, opts Options) error {
 	}
 	ec2Svc := ec2.New(env, vpcSvc)
 	ec2Svc.Recover()
+	cw.GuestUsage = ec2Svc.GuestUsage
 	s3Svc := s3.New(env, secSvc)
 	vpcSvc.AfterCreate = func(v vpc.VPC) {
 		s3Svc.ConnectNetwork(v)
@@ -280,7 +281,7 @@ func Run(ctx context.Context, cfg core.Config, opts Options) error {
 	}
 
 	backup := &system.Backup{Cfg: cfg, Docker: dk, AccountID: account, Version: Version,
-		Snapshots: map[string]func(io.Writer) error{"dynamodb.db": ddb.Snapshot}}
+		Snapshots: map[string]func(io.Writer) error{"dynamodb.db": ddb.Snapshot}, Flatten: ec2Svc.FlattenForBackup}
 	mux := http.NewServeMux()
 	rt := &httpx.Router{Mux: mux, Auth: iamSvc, Account: account, Audit: trailSvc.Record}
 	for _, s := range []routable{iamSvc, secSvc, cw, vpcSvc, ec2Svc, s3Svc, rdsSvc, lambdaSvc, sqsSvc, snsSvc, ddb, eventsSvc, kmsSvc, ssmSvc, ecrSvc, elbSvc, ecsSvc, sfnSvc, cfnSvc, cognitoSvc, asgSvc, acmSvc, dnsSvc, trailSvc, backup} {

cli/internal/svc/cloudwatch/metrics.go

@@ -17,6 +17,7 @@ import (
 
 	"github.com/homecloudhq/homecloud/cli/internal/core"
 	"github.com/homecloudhq/homecloud/cli/internal/httpx"
+	"github.com/homecloudhq/homecloud/cli/internal/runtime"
 	"github.com/homecloudhq/homecloud/cli/internal/svc"
 )
 
@@ -110,6 +111,11 @@ type Service struct {
 	Notify Notifier
 	// Deliver sends log subscription payloads to a Lambda function ARN.
 	Deliver func(ctx context.Context, arn string, payload []byte) error
+	// GuestUsage samples the guest of a virtual-machine EC2 instance (through its
+	// guest agent), so its metrics describe the guest rather than the QEMU
+	// container. While it returns an error (agent not up yet) nothing is published.
+	GuestUsage func(ctx context.Context, instanceID string) (*runtime.Usage, error)
+	guestBusy  sync.Map // container ID -> true while a guest sample is running
 }
 
 func New(env *svc.Env) (*Service, error) {
@@ -294,6 +300,35 @@ func (s *Service) collect(ctx context.Context) {
 			continue
 		}
 		res := c.Labels[core.LabelResource]
+		publish := func(id string, u *runtime.Usage, at time.Time) {
+			dims := map[string]string{n.dim: res}
+			s.Put(n.ns, "CPUUtilization", dims, "Percent", round(u.CPUPercent), at)
+			s.Put(n.ns, "MemoryUtilization", dims, "Percent", round(u.MemoryPercent), at)
+			s.Put(n.ns, "MemoryUsed", dims, "Bytes", float64(u.MemoryBytes), at)
+			s.Put(n.ns, "NetworkIn", dims, "Bytes", s.delta(id+"rx", u.NetRxBytes), at)
+			s.Put(n.ns, "NetworkOut", dims, "Bytes", s.delta(id+"tx", u.NetTxBytes), at)
+			s.Put(n.ns, "DiskReadBytes", dims, "Bytes", s.delta(id+"br", u.BlockRead), at)
+			s.Put(n.ns, "DiskWriteBytes", dims, "Bytes", s.delta(id+"bw", u.BlockWrite), at)
+			s.Put(n.ns, "ProcessCount", dims, "Count", float64(u.Pids), at)
+		}
+		if s.GuestUsage != nil && c.Labels[core.LabelService] == "ec2" && c.Labels["homecloud.virtualization"] != "" {
+			// A guest sample takes a second or more, and much longer while the
+			// guest agent is not up yet: it runs on its own so it never holds up
+			// the tick, and one per instance at a time.
+			if _, busy := s.guestBusy.LoadOrStore(c.ID, true); busy {
+				continue
+			}
+			go func(id string) {
+				defer core.Recover("cloudwatch guest sample " + res)
+				defer s.guestBusy.Delete(id)
+				sctx, cancel := context.WithTimeout(ctx, 30*time.Second)
+				defer cancel()
+				if u, err := s.GuestUsage(sctx, res); err == nil {
+					publish(id, u, time.Now().UTC())
+				}
+			}(c.ID)
+			continue
+		}
 		wg.Add(1)
 		sem <- struct{}{}
 		go func(id string) {
@@ -305,15 +340,7 @@ func (s *Service) collect(ctx context.Context) {
 			if err != nil {
 				return
 			}
-			dims := map[string]string{n.dim: res}
-			s.Put(n.ns, "CPUUtilization", dims, "Percent", round(u.CPUPercent), now)
-			s.Put(n.ns, "MemoryUtilization", dims, "Percent", round(u.MemoryPercent), now)
-			s.Put(n.ns, "MemoryUsed", dims, "Bytes", float64(u.MemoryBytes), now)
-			s.Put(n.ns, "NetworkIn", dims, "Bytes", s.delta(id+"rx", u.NetRxBytes), now)
-			s.Put(n.ns, "NetworkOut", dims, "Bytes", s.delta(id+"tx", u.NetTxBytes), now)
-			s.Put(n.ns, "DiskReadBytes", dims, "Bytes", s.delta(id+"br", u.BlockRead), now)
-			s.Put(n.ns, "DiskWriteBytes", dims, "Bytes", s.delta(id+"bw", u.BlockWrite), now)
-			s.Put(n.ns, "ProcessCount", dims, "Count", float64(u.Pids), now)
+			publish(id, u, now)
 		}(c.ID)
 	}
 	wg.Wait()