GitHub Action and testcontainers modules for using HomeCloud in tests

.github/workflows/action-test.yml

@@ -0,0 +1,53 @@
+name: Action test
+# Exercises integrations/github-action against the latest release. Runs only when
+# the action changes (or by hand) to keep CI minutes down.
+on:
+  pull_request:
+    paths:
+      - "integrations/github-action/**"
+      - ".github/workflows/action-test.yml"
+  push:
+    branches: [main]
+    paths:
+      - "integrations/github-action/**"
+      - ".github/workflows/action-test.yml"
+  workflow_dispatch:
+    inputs:
+      version:
+        description: HomeCloud release to test (e.g. v0.3.0)
+        default: latest
+
+permissions:
+  contents: read
+
+jobs:
+  action:
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    steps:
+      - uses: actions/checkout@v4
+      - name: Input validation
+        run: bash integrations/github-action/test/inputs.sh
+      - id: homecloud
+        uses: ./integrations/github-action
+        with:
+          version: ${{ inputs.version || 'latest' }}
+          services-wait: s3
+      - name: AWS CLI against HomeCloud
+        run: |
+          set -eux
+          test "$AWS_ENDPOINT_URL" = "${{ steps.homecloud.outputs.endpoint }}"
+          aws sts get-caller-identity
+          aws s3 mb s3://action-test
+          echo hello | aws s3 cp - s3://action-test/hello.txt
+          test "$(aws s3 cp s3://action-test/hello.txt -)" = hello
+          url=$(aws sqs create-queue --queue-name action-test --query QueueUrl --output text)
+          aws sqs send-message --queue-url "$url" --message-body hi
+          test "$(aws sqs receive-message --queue-url "$url" --query 'Messages[0].Body' --output text)" = hi
+          aws dynamodb create-table --table-name action-test \
+            --attribute-definitions AttributeName=id,AttributeType=S \
+            --key-schema AttributeName=id,KeyType=HASH --billing-mode PAY_PER_REQUEST
+          aws dynamodb put-item --table-name action-test --item '{"id":{"S":"1"}}'
+          aws dynamodb get-item --table-name action-test --key '{"id":{"S":"1"}}'
+          aws lambda list-functions
+          homecloud version

CHANGELOG.md

@@ -2,6 +2,7 @@
 
 ## Unreleased
 
+- HomeCloud as a drop-in AWS for tests ([docs/integrations.md](docs/integrations.md)): a GitHub Action (`integrations/github-action`) that installs a checksum-verified release, starts the server and exports `AWS_ENDPOINT_URL` and credentials, and testcontainers modules for Go (`integrations/testcontainers-go`) and Python (`integrations/testcontainers-python`) that run the HomeCloud image and clean up every container it started.
 - KMS supports imported key material (`create-key --origin EXTERNAL`, `get-parameters-for-import`, `import-key-material` with `RSAES_OAEP_SHA_1/256` and `RSA_AES_KEY_WRAP_SHA_1/256`, `ValidTo` expiry, `delete-imported-key-material`; only the same material can be imported again) and multi-Region primary keys (`--multi-region`, `mrk-` ids). `ReplicateKey` and `UpdatePrimaryRegion` return `UnsupportedOperationException` because HomeCloud runs a single region.
 - `homecloud configure` accepts `--ca-file` and `--region`, and keeps the region and `ca_file` already in the credentials file when it rewrites it (#79).
 - With `--addr 0.0.0.0:8080` (or `[::]`) the credentials file records `127.0.0.1` (or the public URL) as the endpoint instead of `0.0.0.0`; an existing file is repaired at the next start (#79).

README.md

@@ -6,6 +6,7 @@
   <a href="https://homecloud.pages.dev/demo/"><b>Live demo console</b></a> ·
   <a href="#quick-start"><b>Quick start</b></a> ·
   <a href="docs/aws-compat.md"><b>AWS compatibility</b></a> ·
+  <a href="docs/integrations.md"><b>Testing &amp; CI</b></a> ·
   <a href="docs/comparison.md"><b>vs. LocalStack, moto, MinIO…</b></a> ·
   <a href="https://discord.gg/pemra9uaC9"><b>Discord</b></a>
 </p>
@@ -70,6 +71,8 @@ eval "$(homecloud aws-env)"
 
 S3 starts in the background on first boot; if your tests use S3 right away, wait for `s3: MinIO ready` in the log (see [`.github/workflows/ci.yml`](.github/workflows/ci.yml)).
 
+Or use the ready-made pieces in **[docs/integrations.md](docs/integrations.md)**: a GitHub Action that does the above in one step, and testcontainers modules for Go and Python.
+
 ---
 
 ## Works with

docs/integrations.md

@@ -0,0 +1,145 @@
+# Using HomeCloud in tests and CI
+
+HomeCloud speaks the AWS protocols (SigV4) on one port, so any project can use it as a
+drop-in AWS for integration tests: point `AWS_ENDPOINT_URL` at it and use the root
+credentials it creates on first start. This page covers the ready-made integrations in
+[`integrations/`](../integrations):
+
+| | |
+| --- | --- |
+| [GitHub Action](#github-actions) | `integrations/github-action`: install, start, export `AWS_*` |
+| [testcontainers-go](#testcontainers-go) | `integrations/testcontainers-go`: start HomeCloud from a Go test |
+| [testcontainers-python](#testcontainers-python) | `integrations/testcontainers-python`: start HomeCloud from pytest |
+| [Plain binary or Docker](#plain-binary-or-docker) | anything else |
+
+Everything HomeCloud runs (MinIO for S3, Lambda runtimes, databases) is a container on
+the Docker host, so all of these need Docker. Only **one HomeCloud runs per Docker host**
+at a time: its helper containers have fixed names (`homecloud-s3`, ...) and host ports
+(MinIO 9500/9501, ECR 5500, DNS 8053), and it refuses to start next to another
+installation's containers. Do not run test suites that each start HomeCloud in parallel
+on the same Docker host.
+
+## GitHub Actions
+
+```yaml
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: solinode/homecloud/integrations/github-action@main
+        with:
+          version: latest      # or v0.3.0
+          services-wait: s3    # wait for MinIO too
+      - run: |
+          aws s3 mb s3://artifacts
+          aws sqs create-queue --queue-name jobs
+          pytest               # boto3 reads AWS_ENDPOINT_URL
+```
+
+The action downloads the release, checks it against `checksums.txt`, runs `homecloud
+serve` in the background (data and log under `$RUNNER_TEMP`), waits for
+`/api/v1/health`, and exports `AWS_ENDPOINT_URL`, `AWS_ACCESS_KEY_ID`,
+`AWS_SECRET_ACCESS_KEY` (masked), `AWS_REGION` and `AWS_DEFAULT_REGION`. After the job,
+a post step prints the server log (collapsed), stops HomeCloud and removes its Docker
+resources. Inputs, outputs and boto3/Terraform examples:
+[integrations/github-action/README.md](../integrations/github-action/README.md).
+
+## testcontainers-go
+
+```go
+import homecloud "github.com/solinode/homecloud/integrations/testcontainers-go"
+
+hc, err := homecloud.Run(ctx, homecloud.DefaultImage)
+testcontainers.CleanupContainer(t, hc)
+if err != nil {
+	t.Fatal(err)
+}
+s3c := s3.NewFromConfig(hc.AWSConfig(), func(o *s3.Options) { o.UsePathStyle = true })
+```
+
+`hc.EndpointURL()`, `hc.Credentials()` and `hc.Env()` give the connection details;
+terminating the container also removes every container, network and volume HomeCloud
+created. See [integrations/testcontainers-go/README.md](../integrations/testcontainers-go/README.md).
+
+## testcontainers-python
+
+```python
+from testcontainers_homecloud import HomeCloudContainer
+
+@pytest.fixture(scope="session")
+def homecloud():
+    with HomeCloudContainer() as hc:
+        yield hc
+
+def test_upload(homecloud):
+    s3 = homecloud.get_client("s3")
+    s3.create_bucket(Bucket="test")
+```
+
+See [integrations/testcontainers-python/README.md](../integrations/testcontainers-python/README.md).
+
+### How the testcontainers modules run HomeCloud
+
+Both modules start the image (default `ghcr.io/solinode/homecloud:latest`, configurable)
+with the Docker socket mounted and **host networking**, and the API on
+`127.0.0.1:18080` of the Docker host (configurable; not 8080, which is often taken).
+HomeCloud then reaches the containers it starts on the host's loopback ports, and they
+call it back through `host.docker.internal`, exactly as when it runs on the host. This
+works on Linux (GitHub Actions included), OrbStack, and Docker Desktop with host
+networking enabled (Settings > Resources > Network). They wait for the API and S3, read
+the credentials with `homecloud aws-env` inside the container, and on stop remove
+everything labelled with the account HomeCloud created.
+
+Until the official image is published, build one from a checkout:
+
+```sh
+docker build -f integrations/testdata/Dockerfile -t homecloud:test cli
+HOMECLOUD_IMAGE=homecloud:test go test ./...      # in integrations/testcontainers-go
+HOMECLOUD_IMAGE=homecloud:test pytest             # in integrations/testcontainers-python
+```
+
+## Plain binary or Docker
+
+With the binary (any CI with Docker, or a laptop):
+
+```sh
+curl -fsSL https://homecloud.pages.dev/scripts/install.sh | sh
+export HOMECLOUD_DATA_DIR=$(mktemp -d)
+homecloud serve --data-dir "$HOMECLOUD_DATA_DIR" > homecloud.log 2>&1 &
+until curl -fs http://127.0.0.1:8080/api/v1/health; do sleep 1; done
+eval "$(homecloud aws-env)"      # AWS_ENDPOINT_URL, keys, region
+aws s3 ls
+```
+
+With Docker only, run the image the same way the testcontainers modules do. Until
+`ghcr.io/solinode/homecloud` is published, use the image built above (`homecloud:test`);
+afterwards, replace it with `ghcr.io/solinode/homecloud`:
+
+```sh
+docker build -f integrations/testdata/Dockerfile -t homecloud:test cli
+docker run -d --name homecloud --network host \
+  -v /var/run/docker.sock:/var/run/docker.sock \
+  homecloud:test serve --data-dir /data --addr 127.0.0.1:18080
+until curl -fs http://127.0.0.1:18080/api/v1/health; do sleep 1; done
+docker exec homecloud homecloud aws-env      # credentials; the endpoint is http://127.0.0.1:18080
+```
+
+To clean up afterwards, remove the HomeCloud container with its data volume (`-v`), then
+everything labelled with its account (`docker logs homecloud | grep "created account"`
+shows it):
+
+```sh
+docker rm -fv homecloud
+docker rm -fv $(docker ps -aq --filter label=homecloud.account=ACCOUNT)
+docker network rm $(docker network ls -q --filter label=homecloud.account=ACCOUNT)
+docker volume rm $(docker volume ls -q --filter label=homecloud.account=ACCOUNT)
+```
+
+## SDK notes
+
+- AWS CLI v2, boto3 1.28+, aws-sdk-go-v2, the JavaScript SDK v3 and the Terraform AWS
+  provider 5.x read `AWS_ENDPOINT_URL`.
+- Use path-style S3 addressing (`UsePathStyle`, `addressing_style: path`,
+  `s3_use_path_style = true`) when the endpoint is an IP address or `localhost`.
+- The region is `us-east-1` unless the server was started with another one.

integrations/github-action/README.md

@@ -0,0 +1,126 @@
+# Set up HomeCloud (GitHub Action)
+
+Runs [HomeCloud](https://github.com/solinode/homecloud), a self-hosted AWS, inside your
+workflow so tests can call S3, SQS, DynamoDB, Lambda and the rest of the AWS API without
+an AWS account.
+
+The action:
+
+1. downloads the HomeCloud release you ask for and verifies it against the release's `checksums.txt`,
+2. starts `homecloud serve` in the background (data in `$RUNNER_TEMP/homecloud-data`, log in `$RUNNER_TEMP/homecloud.log`),
+3. waits for `/api/v1/health` and for any services listed in `services-wait`,
+4. exports `AWS_ENDPOINT_URL`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` (masked), `AWS_REGION`, `AWS_DEFAULT_REGION` and `HOMECLOUD_DATA_DIR` for the following steps, and puts `homecloud` on `PATH`,
+5. in a post step after every job, prints the server log (collapsed), stops HomeCloud and removes the containers, networks and volumes it created (so self-hosted runners stay clean).
+
+HomeCloud runs every service on Docker, so use a runner with Docker: `ubuntu-latest` works
+as is. macOS and Windows hosted runners have no Docker.
+
+## Usage
+
+```yaml
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: solinode/homecloud/integrations/github-action@main
+        with:
+          services-wait: s3
+      - run: aws s3 mb s3://my-bucket && aws s3 ls
+```
+
+### Inputs
+
+| Input | Default | Description |
+| --- | --- | --- |
+| `version` | `latest` | Release to install, e.g. `v0.3.0` (the `v` is optional). |
+| `port` | `8080` | Port the API listens on, on `127.0.0.1`. |
+| `services-wait` | (none) | Comma or newline separated services to wait for. `s3`, `ecr` and `route53` start containers in the background; in-process services (`sqs`, `dynamodb`, `lambda`, ...) are ready with the API; an unknown name fails the step. |
+| `wait-timeout` | `180` | Seconds to wait for the API and the services. |
+
+### Outputs
+
+| Output | Description |
+| --- | --- |
+| `endpoint` | e.g. `http://127.0.0.1:8080` (same as `AWS_ENDPOINT_URL`) |
+| `access-key-id` | root access key ID |
+| `region` | region HomeCloud serves (`us-east-1`) |
+| `log-file` | path of the server log |
+
+AWS CLI v2 and current AWS SDKs read `AWS_ENDPOINT_URL`, so most tools need no configuration.
+
+## Examples
+
+### AWS CLI
+
+```yaml
+      - uses: solinode/homecloud/integrations/github-action@main
+        with:
+          services-wait: s3
+      - run: |
+          aws s3 mb s3://artifacts
+          aws sqs create-queue --queue-name jobs
+          aws dynamodb list-tables
+          aws lambda list-functions
+```
+
+### Python (boto3)
+
+boto3 1.28 and later pick up `AWS_ENDPOINT_URL`:
+
+```yaml
+      - uses: solinode/homecloud/integrations/github-action@main
+        with:
+          services-wait: s3
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: pip install boto3 pytest && pytest
+```
+
+```python
+import boto3
+
+def test_upload():
+    s3 = boto3.client("s3")  # endpoint, keys and region come from the environment
+    s3.create_bucket(Bucket="test")
+    s3.put_object(Bucket="test", Key="a.txt", Body=b"hi")
+    assert s3.get_object(Bucket="test", Key="a.txt")["Body"].read() == b"hi"
+```
+
+### Terraform
+
+The AWS provider (5.x and later) honours `AWS_ENDPOINT_URL`; tell it not to look for a real account:
+
+```yaml
+      - uses: solinode/homecloud/integrations/github-action@main
+        with:
+          services-wait: s3
+      - uses: hashicorp/setup-terraform@v3
+      - run: terraform init && terraform apply -auto-approve
+```
+
+```hcl
+provider "aws" {
+  skip_credentials_validation = true
+  skip_requesting_account_id  = true
+  s3_use_path_style           = true
+}
+```
+
+### Using the step outputs
+
+```yaml
+      - id: homecloud
+        uses: solinode/homecloud/integrations/github-action@main
+      - run: ./run-tests.sh --endpoint "${{ steps.homecloud.outputs.endpoint }}"
+```
+
+## Notes
+
+- One HomeCloud per Docker host: it names its helper containers (`homecloud-s3`, ...) and
+  publishes MinIO on 9500/9501, ECR on 5500 and DNS on 8053 on `127.0.0.1`. Those ports
+  must be free on the runner.
+- The root console password from the first-start log is masked as well.
+- This is a JavaScript action (`node24`) only because composite actions cannot declare a
+  post step; all the work is in `setup.sh` and `post.sh`, and it has no dependencies to install.