test(gcp/storage-bucket): add e2e test - #263
Merged
Merged
Conversation
Adds an `e2e/` module that builds the building block definition from source with an ephemeral backplane and orders a workspace-level building block, then asserts the run succeeded and that the bucket name, console URL and summary outputs describe the requested bucket. Two module fixes were needed to make this testable: * `backplane/provider.tf` declared its own `google` provider. A module with a local provider configuration anywhere in its tree is a "legacy module" that callers may not wrap with `count`, `for_each` or `depends_on` — which is exactly what an e2e root needs to gate the build-from-source path. The provider now comes from the caller, matching every other backplane in the hub, and `google_service_account` gained the explicit `project` it used to inherit from that provider block. * The e2e passes a per-run `workload_identity.pool_identifier`. GCP soft-deletes workload identity pools for ~30 days and refuses to reissue their identifiers in that window, so a fixed pool id makes the second run fail. Documented in the backplane readme. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
Scorecard Check
📊 meshstack-hub Module Scorecard
📋 Per-Module Category SummaryScore per category per building block.
Core Structure — ✅ all passingBasic module file structure and documentation — applies to 1 modules
Core Structure — Summary
Integration — ✅ all passingmeshstack_integration.tf conventions — applies to 1 modules
Integration — Summary
Azure Backplane — not applicableAzure UAMI-based automation principal conventions — applies to 0 modules No applicable modules. STACKIT Backplane — not applicableSTACKIT WIF-based automation principal conventions — applies to 0 modules No applicable modules. Testing — ✅ all passingEnd-to-end test coverage — applies to 1 modules
Testing — Summary
|
|
This pull request is automatically being deployed by Amplify Hosting (learn more). |
Enabling the APIs a building block depends on is cloud-side preparation, which is what a backplane is for — a project that has never used IAM answers every call the backplane makes with SERVICE_DISABLED, and one that has never used Cloud Storage does the same to the building block. Follows the precedent in modules/gcp/budget-alert/backplane, which already enables its own API. Five APIs: `iam` and `cloudresourcemanager` for what the backplane itself provisions, `sts` and `iamcredentials` for the workload identity path at run time, and `storage` for the buckets the building block creates. Unlike budget-alert this sets `disable_on_destroy = false`. The provider default is true, which is a hazard here: a backplane never owns its project exclusively and can be short-lived, so a teardown that switched storage.googleapis.com off would break every other tenant of that project. The resources that need an API also gained an explicit `depends_on`, since nothing in them references the enablement. Documents the four roles the applying identity needs — serviceUsageAdmin included, and it cannot be self-granted — in backplane/README.md. The building block's own service account deliberately does not get serviceUsageAdmin: it only creates buckets and never enables a service. Also adds .agents/references/gcp-backplane.md, which did not exist even though AWS, Azure and STACKIT all had one, so the convention this is now the second module to follow was written down nowhere. It records the WIF pattern, the API enablement rule, the required permissions and the workload identity pool soft-delete constraint, and is explicit about where the two existing GCP backplanes genuinely disagree rather than inventing a rule. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A CI run against a project with serviceusage.googleapis.com disabled failed on the *read* of google_project_service, before any enable was attempted: Error when reading or editing Project Service : ... Failed to list enabled services for project <p>: Error 403: Service Usage API has not been used in project <n> before or it is disabled. So the earlier claim that it "is enabled on every project by default" was wrong. google_project_service cannot bootstrap it either — it is the API that does the enabling. Record it as a one-time out-of-band step for the project owner in both the backplane readme and the GCP backplane reference. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…credentials
The building block run failed every time with
oauth2/google: status code 403: Permission 'iam.serviceAccounts.getAccessToken'
denied on resource (or it may not exist). ... "reason": "IAM_PERMISSION_DENIED"
which reads like a broken workload identity pool but is not. The STS token
exchange succeeds — that alone proves the pool, provider, attribute mapping and
attribute condition are all correct — and only the impersonation is refused.
Reading the live IAM policy of the backplane service account while an e2e run
was in flight settled it. The binding is created exactly as intended:
role: roles/iam.workloadIdentityUser
member: principalSet://iam.googleapis.com/projects/950267761049/locations/
global/workloadIdentityPools/hub-e2e-.../*
with the project number, the right pool, and the documented pool-wide form. But
sampling the policy across a whole run showed it only existed for about 80
seconds — empty at first, granted, then torn down — and the building block ran
within a minute of the grant. Google's guidance is to allow two to seven minutes
after adding a workloadIdentityUser binding before a denied impersonation starts
working, so the grant never had a chance to take effect.
So this is not a test-only artifact: any composition or reference architecture
that applies a backplane and orders a building block in the same run hits it.
Fix it in the module, following the eventual-consistency pattern
modules/azure/spoke-network/buildingblock already uses for Azure role
assignments: a time_sleep over both IAM grants, with the delay exposed as
iam_propagation_delay_seconds.
The wait hangs off the `credentials_json` *output* rather than sitting inertly
in the module, so a consumer that builds a building block definition from those
credentials inherits the ordering without having to know the wait exists.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The eventual-consistency explanation was accidentally left above google_project_iam_member.storage_admin as well as above the binding it describes. Comment-only; no behaviour change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
JohannesRudolph
marked this pull request as ready for review
August 20, 2026 13:33
Strip the comments this branch added and write back only the ones a reader cannot derive from the code: why an ordering is explicit, why disable_on_destroy stays false, why the pool id is per-run, and why the credentials output waits. The rest restated the code, argued the design to a reviewer, or duplicated the variable description. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds an
e2e/test module forgcp/storage-bucket, and fixes the four things standing between thatmodule and a working end-to-end deployment: a legacy provider block, a non-reusable workload identity
pool identifier, missing project API enablement, and an IAM propagation race that made every building
block run fail. Also writes down the GCP backplane conventions, which were documented nowhere.
1. The e2e test
modules/gcp/storage-bucket/e2e/follows the standard hub e2e shape: a singletest_contextgrab-bag, the integration module sourced relatively and gated on
bbd_version_ref == null, and aworkspace-level
meshstack_building_blockwithdepends_onon the integration module so the deleterun finishes before the backplane's workload identity pool is torn down.
Assertions: run status
SUCCEEDED, thenbucket_name, thebucket_urlconsole deep link, and thesummarymarkdown (location +gs://URI). The bucket name carries the per-run suffix — GCS bucketnames are globally unique, so collisions across runs would otherwise be a real failure mode.
2. The backplane now enables the project APIs it and the building block need
A project that has never used IAM answers every call the backplane makes with
SERVICE_DISABLED,and one that has never used Cloud Storage does the same to the building block. Enabling those APIs
is cloud-side preparation — which is what a backplane is for.
modules/gcp/budget-alert/backplanealready sets this precedent by enabling its own
billingbudgets.googleapis.com.Five APIs in a single
google_project_servicefor_each:iamcloudresourcemanagerstsiamcredentialsstorageserviceusage.googleapis.comcannot be in the list — it is the API that does the enabling, sogoogle_project_servicecannot bootstrap it. It is also not reliably on by default: the testproject had it disabled, which fails on the read before any enable is attempted. That is a
one-time out-of-band step for the project owner, now documented in
backplane/README.md.disable_on_destroy = false, and that is load-bearing. The provider default istrue, so ithas to be set explicitly. A backplane never owns its project exclusively and can be short-lived —
the e2e provisions and destroys one per run — so a teardown that switched
storage.googleapis.comoff would break every other tenant of the same project.
budget-alerttakes the default here; seethe note at the bottom.
The resources that need an API also gained an explicit
depends_on, since nothing in themreferences the enablement and they would otherwise race it.
No toggle.
budget-alerthas none, and an identity that may create workload identity pools andgrant project-level
roles/storage.adminis already a project admin for whomserviceUsageAdminisa small addition.
backplane/README.mddocuments the requirement and, for organizations that enableAPIs centrally, the
tofu importescape hatch — which is a better answer than a variable nobodyasked for.
serviceUsageAdminis not granted to the building block's service account.budget-alertdoesgrant it, but its building block never calls serviceusage either, so that grant is dead privilege
rather than a pattern to copy. This building block only creates buckets.
3.
.agents/references/gcp-backplane.mdAWS, Azure and STACKIT each have a backplane reference; GCP had none, and
CLAUDE.mdmentioned GCPonly in a variable-prefix example. So the convention this PR is the second module to follow was
written down nowhere. The new reference records the WIF pattern, the API-enablement rule with
disable_on_destroy = false, the required permissions for the applying identity, the workloadidentity pool soft-delete constraint, and a checklist — and is explicit about the places where the
two existing GCP backplanes genuinely disagree (WIF vs. service account key,
project_idvs.backplane_project_id) instead of retrofitting a rule onto one of them.Linked from
AGENTS.md/CLAUDE.mdnext to the other three, plus the STACKIT reference which existedbut was never linked from there either, and from the module skill's per-cloud table.
4. Two module fixes the e2e forced
backplane/provider.tfmade the backplane a "legacy module"It declared its own
googleprovider. A module with a local provider configuration anywhere in itstree cannot be wrapped by a caller that uses
count,for_eachordepends_on:That is exactly the shape an e2e root needs. The provider now comes from the caller — matching every
other backplane in the hub — and
google_service_accountgained the explicitprojectit had beeninheriting from that block.
The workload identity pool identifier must be unique per deployment
GCP soft-deletes workload identity pools and providers and keeps them ~30 days, during which their
identifiers cannot be claimed again, and the provider does not undelete them. A fixed
workload_identity.pool_identifiertherefore makes every re-apply after a destroy fail. The e2ederives a per-run identifier and the constraint is documented in
backplane/README.md.5. The building block run's 403 was IAM propagation, not a broken pool
With the APIs enabled the e2e got all the way to the building block run, which then failed — every
time, not intermittently — with:
That reads like a misconfigured workload identity pool. It is not, and it is worth recording how
that was established, because the obvious reading sends you to the wrong half of the search space.
The STS exchange succeeding is the tell. The external account flow is two hops: exchange the
runner's OIDC token at
sts.googleapis.com, then impersonate the service account atiamcredentials.googleapis.com. A wrong issuer, audience,attribute_mappingorattribute_conditionfails the first hop with a token-exchange error. This failed on the second,so the pool, provider, mapping and the
startsWith()condition were all already proven correct.Reading the live IAM policy settled the rest. Sampling the backplane service account's policy
through a run gave the ground truth:
So the binding is created exactly as intended — project number, correct pool, the documented
pool-wide form — and it existed for about 80 seconds, with the building block running inside the
first minute of it. Google's guidance is to allow two to seven minutes after adding a
roles/iam.workloadIdentityUserbinding before a denied impersonation starts working. The grantnever had a chance to take effect.
Why the fix is in the module and not the test. It would be easy to call this a test artifact and
sleep in the e2e. But nothing about it is test-specific: any composition or reference architecture
that applies a backplane and orders a building block in the same run hits exactly this. A backplane
that grants a role and publishes credentials in the same breath is publishing credentials that do
not work yet — that is the module's bug to fix.
So: a
time_sleepover both IAM grants, with the delay exposed asiam_propagation_delay_seconds(default 180s), following the pattern
modules/azure/spoke-network/buildingblockalready uses forAzure role assignments. The wait hangs off the
credentials_jsonoutput rather than sittinginertly inside the module, so a consumer that builds a BBD from those credentials inherits the
ordering without having to know the wait exists.
On the WIF trust question
The backplane is correct here: issuer, audience, subjects and the token file path are all
parameterised through
var.workload_identity_federation, andmeshstack_integration.tffills themfrom
data.meshstack_integrationsrather than hardcoding anything, so the pool provider trustswhichever meshStack instance the integration is deployed against.
One deliberate difference from the Azure/STACKIT integrations: they scope the federated subject down
to
…buildingblockdefinition.<bbd-uuid>, while this one stops at…buildingblockdefinitionandrelies on the backplane's
startsWith()attribute condition. It has to — the BBD consumesmodule.backplane.credentials_json, so referencing the BBD's uuid from the backplane inputs would bea dependency cycle. Worth revisiting, out of scope here.
Noted, not fixed:
gcp/budget-alerthas a latent teardown hazardmodules/gcp/budget-alert/backplaneenablesbillingbudgets.googleapis.comwithoutdisable_on_destroy = false. If that module ever gets an e2e test, each teardown would disable theAPI project-wide for every other consumer. Its
roles/serviceusage.serviceUsageAdmingrant to thebuilding block's service account is also unused. Left alone deliberately — separate PR.
Verification
tofu fmt,tofu validate(backplane and e2e) passgcp/storage-bucket100% overall, Testing 100%tests/gcp_storage_bucket_hub.tftest.hcl... pass,Success! 1 passed, 0 failed.Sampling the service account's IAM policy across that run shows the mechanism working: the binding
lived 3m46s (13:26:26 → 13:30:12) instead of the ~80s it survived before the fix, and the
building block ran near the end of that window rather than in the first minute of it.
Teardown is clean — no leaked buckets, no leaked service accounts, no stale
deleted:members in theproject IAM policy. The only residue is the expected one soft-deleted workload identity pool per run
(4 in the project now, purging on GCP's ~30-day schedule), which is why the module is on the
nightly-only list.
🤖 Generated with Claude Code