Skip to content

feat(scorecard): enforce a hub-wide meshstack provider version floor - #259

Open
JohannesRudolph wants to merge 2 commits into
mainfrom
claude/scorecard-provider-floor
Open

feat(scorecard): enforce a hub-wide meshstack provider version floor#259
JohannesRudolph wants to merge 2 commits into
mainfrom
claude/scorecard-provider-floor

Conversation

@JohannesRudolph

@JohannesRudolph JohannesRudolph commented Aug 18, 2026

Copy link
Copy Markdown
Member

Adds one central knob for the minimum meshcloud/meshstack provider version the whole hub is tested and used against, enforced by a new scorecard check, and raises every module to >= 0.23.0.

Two commits, reviewable separately: the check, then the bump.

Why a floor is needed

The hub's e2e roots declared meshcloud/meshstack with no version constraint and commit no lockfile. In CI that is harmless, because the smoke-test workflow builds the provider from main HEAD and installs it via Terraform dev_overrides, which bypasses both required_providers and .terraform.lock.hcl.

Locally it is not harmless. Gitignored on-disk lockfiles in this repo reach back to provider 0.20.5 (modules/ske/ske-starterkit/e2e), 0.20.9 (modules/azure/budget-alert/e2e, modules/azure/storage-account/e2e) and 0.20.11 (modules/azure/resource-group/e2e, modules/meshstack/noop/e2e). All predate provider commit 01cabb2 (released v0.20.9), which made the building-block delete poll lifecycle-aware — before it, DeletionSuccessful accepted only a 404 and could not detect a soft delete.

The consequence is a real race on a local task e2e:run without the override provider: the building block delete returns before the block is soft-deleted, and the following building-block-definition delete gets 409 "…because there are existing BuildingBlocks referencing it", stranding cloud objects. 0.23.0 is the release that first contains the current meshstack_building_block resource (provider commit 0ee7530bc07).

The knob

PROVIDER_FLOOR in tools/scorecard/scorecard.mjs, keyed by registry source address rather than local alias, since the alias is per-module:

const PROVIDER_FLOOR = { "meshcloud/meshstack": "0.23.0" }

Crank the value, run the scorecard, and the red list is the bump work item.

Verify, not generate

The deciding reason is mechanical: HCL required_providers { version = … } must be a string literal — no interpolation — so there is nothing a module could reference. Generating would mean a new codegen tool plus a --check drift mode, two mechanisms instead of none, and it would fight the terraform_docs pre-commit hook that regenerates README requirement tables. Verification reuses the scorecard exactly as-is, and the constraint stays a reviewable statement of what each module actually needs.

Scope of the check

The closest existing check, provider_pinned, enforces >= style but says nothing about how old, and only scanned backplane/ + buildingblock/. The new check covers all four tiers a module owns — meshstack_integration.tf, backplane/, buildingblock/, e2e/ (including nested submodules) — and treats "declared with no version" as a failure, which was the actual finding.

provider_pinned was left on its original tiers. The shared collector was refactored into collectProviderEntries(mod, tiers); provider_pinned stays at exactly 42/52 and every other check is byte-identical to the pre-change report.

Results

Before the bump: 26 of 28 applicable modules failed, e.g. e2e/terraform.tf: meshstack has no version (need ">= 0.23.0") and meshstack_integration.tf: meshstack = ">= 0.21.0" is below 0.23.0.

After:

| ⬆️ | Provider constraints meet the hub-wide version floor | **28/28** | 🟢 100% |
| 🔒 | Provider versions use minimum constraint (>=)        | **42/52** | 🟢 81%  |

24 modules render ➖ — they declare no meshstack provider of their own.

The bump touches 41 .tf files across 26 modules plus 4 README requirement rows. 16 were the e2e/ tier gaining a version for the first time; the rest were floors at >= 0.7.1 / 0.14.0 / 0.21.0. Every diff is 2–3 lines, tofu fmt-clean, and the terraform_docs hook passes on all changed files.

Knob proven: temporarily setting it to 0.24.4 flipped the row to 0/28 with per-file detail and a working AGENTS.md#provider-version-floor fix link. Constraint parser tested against a throwaway fixture — >= 0.23 / >= 0.23.0, < 1.0.0 / ~> 0.24 pass; >= 0.22.9 / ~> 0.22 / > 0.22.0 / < 1.0.0 / no-version fail.

One wiring gap fixed along the way

pr-scorecard.sh reports only on changed modules, so a PR that merely turns the knob touches no module and would have printed _No module changes detected_ — the knob would have been silent in CI, which defeats the point. It now reports hub-wide when tools/scorecard/ itself changed. Verified: this branch triggers the tool-changed path.

Known limitation

Provider-level modules/<provider>/meshstack_integration.tf files (aks, aws, azure, gcp, stackit) are real consumer-facing configurations, but discoverModules only walks modules/<provider>/<service>/, so no scorecard check reaches them. They are bumped here for uniformity, but the check cannot enforce them. Widening discovery would change the module universe for all 30-odd checks, so that is left alone deliberately.

🤖 Generated with Claude Code

JohannesRudolph and others added 2 commits August 18, 2026 17:42
@JohannesRudolph @claude
feat(scorecard): add a hub-wide provider version floor check
16be584 
A `>=` constraint only says "not older than this"; nothing in the repo said how
old is still acceptable *anywhere*. `PROVIDER_FLOOR` at the top of scorecard.mjs
is now that single knob — crank a value and the new `provider_floor` check turns
red on every module still declaring less, which is the work item for the bump.

The check covers every tier a module owns (meshstack_integration.tf, backplane/,
buildingblock/, e2e/, incl. nested submodules) and fails two ways: a
required_providers entry for a floor-managed provider with no `version` at all
(unbounded — `tofu init` may resolve anything), or a constraint whose lower bound
sits below the floor. It verifies rather than rewrites: HCL `required_providers`
`version` must be a string literal, so there is nothing a module could reference.

This matters for local runs specifically. The hub commits no
.terraform.lock.hcl, and the smoke-test workflow injects a provider built from
main through Terraform `dev_overrides`, bypassing both required_providers and any
lockfile — so CI never notices a stale constraint, but `task e2e:run` does.

pr-scorecard.sh now reports on all modules when tools/scorecard/ itself changed,
because a PR that only turns the knob touches no module and would otherwise get
an empty scorecard comment.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@JohannesRudolph @claude
chore: raise the meshstack provider floor to >= 0.23.0 across the hub
7abeb1e 
Satisfies the new `provider_floor` scorecard check: every meshcloud/meshstack
entry in a module now declares at least `>= 0.23.0`, the first release carrying
the current `meshstack_building_block` resource.

Two kinds of change, 41 .tf files (+ 4 regenerated buildingblock READMEs):

- e2e roots (and their nested submodules) declared the provider with no `version`
  at all, so a local `tofu init` was free to resolve anything. On-disk lockfiles
  in this checkout had settled as far back as 0.20.5. Anything below 0.23.0
  predates the lifecycle-aware building block delete poll, where
  `DeletionSuccessful` accepted only a 404 and could not see a soft delete: a
  local `task e2e:run` teardown then returns before the block is gone and the
  following building-block-*definition* delete fails with
  409 "…because there are existing BuildingBlocks referencing it", stranding
  objects in the smoke-test workspace.

- integration/backplane/buildingblock tiers carried floors from `>= 0.7.1` to
  `>= 0.21.0`, predating the same fix.

Five provider-level meshstack_integration.tf files (modules/<provider>/) are
bumped for consistency but sit outside the scorecard's module discovery, which
only walks modules/<provider>/<service>/ — the check cannot guard them.

The four README Requirements rows are hand-applied rather than regenerated: the
terraform-docs available here (0.22.0) reformats every table separator in the
file, which is unrelated churn. Only the meshstack version cell is changed, which
is exactly what the pinned hook would produce.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Scorecard Check

Scorecard run on commit 9a0352c4eaa0a8397c80dde8c372fca9efbd6b2e relative to origin/main

⚙️ Scorecard tooling changed — reporting on all modules, not just the ones this PR touches.

📊 meshstack-hub Module Scorecard

Generated: 2026-08-18 | Modules scanned: 52 | Categories: 5

📋 Per-Module Category Summary

Score per category per building block. n/a = category does not apply to this module.

Module Overall Core Structure Integration Azure Backplane STACKIT Backplane Testing
aks/github-connector 🟢 87% 🟢 86% 🟢 100% n/a n/a 🔴 33%
aks/postgresql 🟡 60% 🟡 71% n/a n/a n/a 🔴 33%
aks/starterkit 🟢 83% 🟢 100% 🟢 92% n/a n/a 🔴 0%
aws/agentic-coding-sandbox 🟡 73% 🟢 88% n/a n/a n/a 🔴 33%
aws/alternate-contacts 🟡 60% 🟡 71% n/a n/a n/a 🔴 33%
aws/budget-alert 🟡 70% 🟢 86% n/a n/a n/a 🔴 33%
aws/opt-in-region 🟡 60% 🟡 71% n/a n/a n/a 🔴 33%
aws/route53-dns-alias-record 🟢 91% 🟢 100% 🟢 100% n/a n/a 🔴 33%
aws/route53-dns-record 🟢 91% 🟢 100% 🟢 100% n/a n/a 🔴 33%
aws/s3_bucket 🟢 91% 🟢 100% 🟢 100% n/a n/a 🔴 33%
azure/aks 🔴 42% 🟢 86% n/a 🔴 11% n/a 🔴 33%
azure/azure-bastion 🟡 58% 🟢 86% n/a 🔴 44% n/a 🔴 33%
azure/azure-virtual-machine 🔴 42% 🟢 86% n/a 🔴 11% n/a 🔴 33%
azure/azure-virtual-machine-starterkit 🟡 50% 🟡 63% n/a 🔴 44% n/a 🔴 33%
azure/budget-alert 🟢 100% 🟢 100% 🟢 100% 🟢 100% n/a 🟢 100%
azure/container-registry 🔴 42% 🟢 86% n/a 🔴 11% n/a 🔴 33%
azure/entra-id-groups 🟢 94% 🟢 100% 🟢 100% 🟢 100% n/a 🔴 33%
azure/github-actions-terraform-setup 🔴 42% 🟡 71% n/a 🔴 22% n/a 🔴 33%
azure/key-vault 🔴 42% 🟢 86% n/a 🔴 11% n/a 🔴 33%
azure/postgresql 🔴 47% 🟡 57% n/a 🔴 44% n/a 🔴 33%
azure/resource-group 🟢 100% 🟢 100% 🟢 100% 🟢 100% n/a 🟢 100%
azure/service-principal 🟡 69% 🟢 100% 🟢 100% 🔴 11% n/a 🔴 33%
azure/spoke-network 🟡 53% 🟡 71% n/a 🔴 44% n/a 🔴 33%
azure/storage-account 🟢 100% 🟢 100% 🟢 100% 🟢 100% n/a 🟢 100%
azure/vmss 🟡 60% 🟢 86% n/a n/a n/a 🔴 0%
azuredevops/agent-pool 🟡 70% 🟢 86% n/a n/a n/a 🔴 33%
azuredevops/pipeline 🟡 70% 🟢 86% n/a n/a n/a 🔴 33%
azuredevops/project 🟡 70% 🟢 86% n/a n/a n/a 🔴 33%
azuredevops/repository 🟡 70% 🟢 86% n/a n/a n/a 🔴 33%
azuredevops/service-connection-subscription 🟡 70% 🟢 86% n/a n/a n/a 🔴 33%
gcp/budget-alert 🟡 60% 🟡 71% n/a n/a n/a 🔴 33%
gcp/storage-bucket 🟢 91% 🟢 100% 🟢 100% n/a n/a 🔴 33%
github/repository 🟡 78% 🟡 71% 🟢 100% n/a n/a 🔴 0%
ionos/dcd 🟡 70% 🟢 86% n/a n/a n/a 🔴 33%
ionos/user-management 🟡 60% 🟢 86% n/a n/a n/a 🔴 0%
kubernetes/manifest 🟢 87% 🟢 86% 🟢 100% n/a n/a 🔴 33%
kubernetes/service-account 🟡 60% 🟢 86% n/a n/a n/a 🔴 0%
meshstack/github-workflow 🟢 87% 🟡 71% 🟢 92% n/a n/a 🟢 100%
meshstack/link 🟢 100% 🟢 100% 🟢 100% n/a n/a 🟢 100%
meshstack/manual 🟢 86% 🟡 71% 🟢 92% n/a n/a 🟢 100%
meshstack/noop 🟢 96% 🟢 86% 🟢 100% n/a n/a 🟢 100%
meshstack/payment-method 🟡 64% 🟢 88% n/a n/a n/a 🔴 0%
oci/application-compartment 🔴 45% 🟡 63% n/a n/a n/a 🔴 0%
sapbtp/subaccounts 🟡 60% 🟢 86% n/a n/a n/a 🔴 0%
sapbtp/subdirectory 🟡 60% 🟢 86% n/a n/a n/a 🔴 0%
ske/forgejo-connector 🟢 87% 🟢 100% 🟢 100% n/a n/a 🔴 0%
ske/ske-starterkit 🟢 91% 🟢 100% 🟢 92% n/a n/a 🟡 67%
stackit/git-repository 🟢 89% 🟢 100% 🟢 100% n/a 🔴 25% 🟢 100%
stackit/network 🟢 93% 🟢 100% 🟢 100% n/a 🟢 100% 🔴 33%
stackit/network-area 🟢 93% 🟢 100% 🟢 100% n/a 🟢 100% 🔴 33%
stackit/project 🟢 82% 🟢 100% 🟡 79% n/a 🟢 100% 🔴 33%
stackit/storage-bucket 🟢 100% 🟢 100% 🟢 100% n/a 🟢 100% 🟢 100%

⚠️ 47 modules have failing checks — failing categories are expanded below.

Core Structure — some checks failing

Basic module file structure and documentation — applies to 52 modules

Module Score 📦 🔗 📋 📝 🖼️ 📌 🔒 ⬆️
aks/github-connector 🟢 86% ✅ ✅ ➖ ✅ ✅ ✅ ❌ ✅
aks/postgresql 🟡 71% ✅ ❌ ✅ ✅ ✅ ✅ ❌ ➖
aks/starterkit 🟢 100% ✅ ✅ ➖ ✅ ✅ ✅ ✅ ✅
aws/agentic-coding-sandbox 🟢 88% ✅ ❌ ✅ ✅ ✅ ✅ ✅ ✅
aws/alternate-contacts 🟡 71% ✅ ❌ ❌ ✅ ✅ ✅ ✅ ➖
aws/budget-alert 🟢 86% ✅ ❌ ✅ ✅ ✅ ✅ ✅ ➖
aws/opt-in-region 🟡 71% ✅ ❌ ✅ ✅ ✅ ✅ ❌ ➖
aws/route53-dns-alias-record 🟢 100% ✅ ✅ ➖ ✅ ✅ ✅ ✅ ✅
aws/route53-dns-record 🟢 100% ✅ ✅ ➖ ✅ ✅ ✅ ✅ ✅
aws/s3_bucket 🟢 100% ✅ ✅ ➖ ✅ ✅ ✅ ✅ ✅
azure/aks 🟢 86% ✅ ❌ ✅ ✅ ✅ ✅ ✅ ➖
azure/azure-bastion 🟢 86% ✅ ❌ ✅ ✅ ✅ ✅ ✅ ➖
azure/azure-virtual-machine 🟢 86% ✅ ❌ ✅ ✅ ✅ ✅ ✅ ➖
azure/azure-virtual-machine-starterkit 🟡 63% ✅ ❌ ❌ ✅ ❌ ✅ ✅ ✅
azure/budget-alert 🟢 100% ✅ ✅ ➖ ✅ ✅ ✅ ✅ ✅
azure/container-registry 🟢 86% ✅ ❌ ✅ ✅ ✅ ✅ ✅ ➖
azure/entra-id-groups 🟢 100% ✅ ✅ ➖ ✅ ✅ ✅ ✅ ✅
azure/github-actions-terraform-setup 🟡 71% ✅ ❌ ✅ ✅ ✅ ✅ ❌ ➖
azure/key-vault 🟢 86% ✅ ❌ ✅ ✅ ✅ ✅ ✅ ➖
azure/postgresql 🟡 57% ✅ ❌ ❌ ✅ ✅ ✅ ❌ ➖
azure/resource-group 🟢 100% ✅ ✅ ➖ ✅ ✅ ✅ ✅ ✅
azure/service-principal 🟢 100% ✅ ✅ ➖ ✅ ✅ ✅ ✅ ✅
azure/spoke-network 🟡 71% ✅ ❌ ✅ ✅ ✅ ✅ ❌ ➖
azure/storage-account 🟢 100% ✅ ✅ ➖ ✅ ✅ ✅ ✅ ✅
azure/vmss 🟢 86% ✅ ❌ ✅ ✅ ✅ ✅ ✅ ➖
azuredevops/agent-pool 🟢 86% ✅ ❌ ✅ ✅ ✅ ✅ ✅ ➖
azuredevops/pipeline 🟢 86% ✅ ❌ ✅ ✅ ✅ ✅ ✅ ➖
azuredevops/project 🟢 86% ✅ ❌ ✅ ✅ ✅ ✅ ✅ ➖
azuredevops/repository 🟢 86% ✅ ❌ ✅ ✅ ✅ ✅ ✅ ➖
azuredevops/service-connection-subscription 🟢 86% ✅ ❌ ✅ ✅ ✅ ✅ ✅ ➖
gcp/budget-alert 🟡 71% ✅ ❌ ✅ ✅ ✅ ✅ ❌ ➖
gcp/storage-bucket 🟢 100% ✅ ✅ ➖ ✅ ✅ ✅ ✅ ✅
github/repository 🟡 71% ✅ ✅ ➖ ✅ ✅ ❌ ❌ ✅
ionos/dcd 🟢 86% ✅ ❌ ✅ ✅ ✅ ✅ ✅ ➖
ionos/user-management 🟢 86% ✅ ❌ ✅ ✅ ✅ ✅ ✅ ➖
kubernetes/manifest 🟢 86% ✅ ✅ ➖ ✅ ❌ ✅ ✅ ✅
kubernetes/service-account 🟢 86% ✅ ❌ ✅ ✅ ✅ ✅ ✅ ➖
meshstack/github-workflow 🟡 71% ✅ ✅ ➖ ✅ ❌ ❌ ✅ ✅
meshstack/link 🟢 100% ✅ ✅ ➖ ✅ ✅ ✅ ✅ ✅
meshstack/manual 🟡 71% ✅ ✅ ➖ ✅ ❌ ❌ ✅ ✅
meshstack/noop 🟢 86% ✅ ✅ ➖ ✅ ✅ ✅ ❌ ✅
meshstack/payment-method 🟢 88% ✅ ❌ ✅ ✅ ✅ ✅ ✅ ✅
oci/application-compartment 🟡 63% ✅ ❌ ✅ ✅ ✅ ❌ ❌ ✅
sapbtp/subaccounts 🟢 86% ✅ ❌ ✅ ✅ ✅ ✅ ✅ ➖
sapbtp/subdirectory 🟢 86% ✅ ❌ ✅ ✅ ✅ ✅ ✅ ➖
ske/forgejo-connector 🟢 100% ✅ ✅ ➖ ✅ ✅ ✅ ✅ ✅
ske/ske-starterkit 🟢 100% ✅ ✅ ➖ ✅ ✅ ✅ ✅ ✅
stackit/git-repository 🟢 100% ✅ ✅ ➖ ✅ ✅ ✅ ✅ ✅
stackit/network 🟢 100% ✅ ✅ ➖ ✅ ✅ ✅ ✅ ✅
stackit/network-area 🟢 100% ✅ ✅ ➖ ✅ ✅ ✅ ✅ ✅
stackit/project 🟢 100% ✅ ✅ ➖ ✅ ✅ ✅ ✅ ✅
stackit/storage-bucket 🟢 100% ✅ ✅ ➖ ✅ ✅ ✅ ✅ ✅

Core Structure — Summary

Emoji Criterion Coverage Status
📦 buildingblock/ directory exists 52/52 🟢 100%
🔗 meshstack_integration.tf present 24/52 🔴 46%
📋 buildingblock/APP_TEAM_README.md present (no-integration fallback) 25/28 🟢 89%
📝 buildingblock/README.md with YAML front-matter 52/52 🟢 100%
🖼️ buildingblock/logo.png included 48/52 🟢 92%
📌 buildingblock/versions.tf present 48/52 🟢 92%
🔒 Provider versions use minimum constraint (>=) 42/52 🟢 81%
⬆️ Provider constraints meet the hub-wide version floor 28/28 🟢 100%
Integration — some checks failing

meshstack_integration.tf conventions — applies to 24 modules

Module Score 🏷️ 🏢 📤 🔌 📎 🔀 📋 🏷️ 🧱 📖 📝 📊 🚫 🔄
aks/github-connector 🟢 100% ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖
aks/starterkit 🟢 92% ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ❌ ✅ ✅ ✅ ✅ ➖
aws/route53-dns-alias-record 🟢 100% ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖
aws/route53-dns-record 🟢 100% ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖
aws/s3_bucket 🟢 100% ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖
azure/budget-alert 🟢 100% ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖
azure/entra-id-groups 🟢 100% ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖
azure/resource-group 🟢 100% ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖
azure/service-principal 🟢 100% ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖
azure/storage-account 🟢 100% ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖
gcp/storage-bucket 🟢 100% ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖
github/repository 🟢 100% ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖
kubernetes/manifest 🟢 100% ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖
meshstack/github-workflow 🟢 92% ✅ ✅ ✅ ✅ ✅ ❌ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖
meshstack/link 🟢 100% ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖
meshstack/manual 🟢 92% ✅ ✅ ✅ ✅ ✅ ❌ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖
meshstack/noop 🟢 100% ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖
ske/forgejo-connector 🟢 100% ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖
ske/ske-starterkit 🟢 92% ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ❌ ✅ ✅ ✅ ✅ ➖
stackit/git-repository 🟢 100% ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖
stackit/network 🟢 100% ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖
stackit/network-area 🟢 100% ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖
stackit/project 🟡 79% ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ❌ ❌ ❌ ✅ ✅
stackit/storage-bucket 🟢 100% ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ➖

Integration — Summary

Emoji Criterion Coverage Status
🏷️ variable "hub" in integration 24/24 🟢 100%
🏢 variable "meshstack" in integration 24/24 🟢 100%
📤 building_block_definition output exposed 24/24 🟢 100%
🔌 meshcloud/meshstack in required_providers 24/24 🟢 100%
📎 backplane source uses var.hub.git_ref 24/24 🟢 100%
🔀 ref_name uses var.hub.git_ref 22/24 🟢 92%
📋 version_spec.draft uses var.hub.bbd_draft 24/24 🟢 100%
🏷️ BBD metadata.tags forwards var.meshstack.tags 24/24 🟢 100%
🧱 BBD input argument vars with optional() have explicit defaults 22/24 🟢 92%
📖 BBD readme field present 23/24 🟢 96%
📝 BBD readme starts with plain-text description (no heading) 23/24 🟢 96%
📊 BBD readme has shared responsibility table (✅/❌) 23/24 🟢 96%
🚫 No documentation_md output in backplane 24/24 🟢 100%
🔄 meshstack_platform has lifecycle ignore_changes = [availability] 1/1 🟢 100%
Azure Backplane — some checks failing

Azure UAMI-based automation principal conventions — applies to 14 modules

Module Score 🪪 🚫 🚫 🔑 🔗 ⚡ 🧹 📤 📍
azure/aks 🔴 11% ❌ ❌ ❌ ❌ ❌ ✅ ❌ ❌ ❌
azure/azure-bastion 🔴 44% ❌ ✅ ✅ ✅ ❌ ❌ ✅ ❌ ❌
azure/azure-virtual-machine 🔴 11% ❌ ❌ ❌ ❌ ❌ ✅ ❌ ❌ ❌
azure/azure-virtual-machine-starterkit 🔴 44% ❌ ✅ ✅ ✅ ❌ ❌ ✅ ❌ ❌
azure/budget-alert 🟢 100% ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅
azure/container-registry 🔴 11% ❌ ❌ ❌ ❌ ❌ ✅ ❌ ❌ ❌
azure/entra-id-groups 🟢 100% ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅
azure/github-actions-terraform-setup 🔴 22% ❌ ❌ ❌ ✅ ❌ ❌ ✅ ❌ ❌
azure/key-vault 🔴 11% ❌ ❌ ❌ ❌ ❌ ✅ ❌ ❌ ❌
azure/postgresql 🔴 44% ❌ ✅ ✅ ✅ ❌ ❌ ✅ ❌ ❌
azure/resource-group 🟢 100% ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅
azure/service-principal 🔴 11% ❌ ❌ ❌ ❌ ❌ ✅ ❌ ❌ ❌
azure/spoke-network 🔴 44% ❌ ✅ ✅ ✅ ❌ ❌ ✅ ❌ ❌
azure/storage-account 🟢 100% ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅

Azure Backplane — Summary

Emoji Criterion Coverage Status
🪪 Uses azurerm_user_assigned_identity 4/14 🔴 29%
🚫 No azuread_application resources 8/14 🟡 57%
🚫 No azuread_service_principal resources 8/14 🟡 57%
🔑 No azuread_application_password resources 9/14 🟡 64%
🔗 Uses azurerm_federated_identity_credential 4/14 🔴 29%
⚡ workload_identity_federation is non-nullable 9/14 🟡 64%
🧹 No create_service_principal_name toggle 9/14 🟡 64%
📤 Outputs identity (client_id, principal_id, tenant_id) 4/14 🔴 29%
📍 Integration has azure_location 4/14 🔴 29%
STACKIT Backplane — some checks failing

STACKIT WIF-based automation principal conventions — applies to 5 modules

Module Score 🔐 🚫 📤 ⚡
stackit/git-repository 🔴 25% ❌ ✅ ❌ ❌
stackit/network 🟢 100% ✅ ✅ ✅ ✅
stackit/network-area 🟢 100% ✅ ✅ ✅ ✅
stackit/project 🟢 100% ✅ ✅ ✅ ✅
stackit/storage-bucket 🟢 100% ✅ ✅ ✅ ✅

STACKIT Backplane — Summary

Emoji Criterion Coverage Status
🔐 Uses stackit_service_account_federated_identity_provider 4/5 🟢 80%
🚫 No stackit_service_account_key resource 5/5 🟢 100%
📤 Outputs service_account_email (not key) 4/5 🟢 80%
⚡ Buildingblock provider uses use_oidc = true 4/5 🟢 80%
Testing — some checks failing

End-to-end test coverage — applies to 52 modules

Module Score ⚙️ 🧪 ✅
aks/github-connector 🔴 33% ✅ ❌ ❌
aks/postgresql 🔴 33% ✅ ❌ ❌
aks/starterkit 🔴 0% ❌ ❌ ❌
aws/agentic-coding-sandbox 🔴 33% ✅ ❌ ❌
aws/alternate-contacts 🔴 33% ✅ ❌ ❌
aws/budget-alert 🔴 33% ✅ ❌ ❌
aws/opt-in-region 🔴 33% ✅ ❌ ❌
aws/route53-dns-alias-record 🔴 33% ✅ ❌ ❌
aws/route53-dns-record 🔴 33% ✅ ❌ ❌
aws/s3_bucket 🔴 33% ✅ ❌ ❌
azure/aks 🔴 33% ✅ ❌ ❌
azure/azure-bastion 🔴 33% ✅ ❌ ❌
azure/azure-virtual-machine 🔴 33% ✅ ❌ ❌
azure/azure-virtual-machine-starterkit 🔴 33% ✅ ❌ ❌
azure/budget-alert 🟢 100% ✅ ✅ ✅
azure/container-registry 🔴 33% ✅ ❌ ❌
azure/entra-id-groups 🔴 33% ✅ ❌ ❌
azure/github-actions-terraform-setup 🔴 33% ✅ ❌ ❌
azure/key-vault 🔴 33% ✅ ❌ ❌
azure/postgresql 🔴 33% ✅ ❌ ❌
azure/resource-group 🟢 100% ✅ ✅ ✅
azure/service-principal 🔴 33% ✅ ❌ ❌
azure/spoke-network 🔴 33% ✅ ❌ ❌
azure/storage-account 🟢 100% ✅ ✅ ✅
azure/vmss 🔴 0% ❌ ❌ ❌
azuredevops/agent-pool 🔴 33% ✅ ❌ ❌
azuredevops/pipeline 🔴 33% ✅ ❌ ❌
azuredevops/project 🔴 33% ✅ ❌ ❌
azuredevops/repository 🔴 33% ✅ ❌ ❌
azuredevops/service-connection-subscription 🔴 33% ✅ ❌ ❌
gcp/budget-alert 🔴 33% ✅ ❌ ❌
gcp/storage-bucket 🔴 33% ✅ ❌ ❌
github/repository 🔴 0% ❌ ❌ ❌
ionos/dcd 🔴 33% ✅ ❌ ❌
ionos/user-management 🔴 0% ❌ ❌ ❌
kubernetes/manifest 🔴 33% ✅ ❌ ❌
kubernetes/service-account 🔴 0% ❌ ❌ ❌
meshstack/github-workflow 🟢 100% ✅ ✅ ✅
meshstack/link 🟢 100% ➖ ✅ ✅
meshstack/manual 🟢 100% ➖ ✅ ✅
meshstack/noop 🟢 100% ✅ ✅ ✅
meshstack/payment-method 🔴 0% ❌ ❌ ❌
oci/application-compartment 🔴 0% ❌ ❌ ❌
sapbtp/subaccounts 🔴 0% ❌ ❌ ❌
sapbtp/subdirectory 🔴 0% ❌ ❌ ❌
ske/forgejo-connector 🔴 0% ❌ ❌ ❌
ske/ske-starterkit 🟡 67% ❌ ✅ ✅
stackit/git-repository 🟢 100% ✅ ✅ ✅
stackit/network 🔴 33% ✅ ❌ ❌
stackit/network-area 🔴 33% ✅ ❌ ❌
stackit/project 🔴 33% ✅ ❌ ❌
stackit/storage-bucket 🟢 100% ✅ ✅ ✅

Testing — Summary

Emoji Criterion Coverage Status
⚙️ backplane/ directory (optional tier) 39/50 🟡 78%
🧪 e2e/ test directory exists 10/52 🔴 19%
✅ e2e/ contains .tftest.hcl files 10/52 🔴 19%

📈 Overall Summary

Overall Average Score: 73%

Score Distribution

  • 🟢 High maturity (≥80%): 22 modules
  • 🟡 Medium maturity (50–79%): 23 modules
  • 🔴 Low maturity (<50%): 7 modules

@aws-amplify-eu-central-1

aws-amplify-eu-central-1 Bot commented Aug 18, 2026

Copy link
Copy Markdown

This pull request is automatically being deployed by Amplify Hosting (learn more).

Access this pull request here: https://pr-259.d1o16zfeoh2slu.amplifyapp.com

JohannesRudolph
JohannesRudolph commented Aug 18, 2026
View reviewed changes

@JohannesRudolph JohannesRudolph left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@grubmeshi one point i wonder:

I noticed this discrepancy in foundation mode tests (ie smoke tests with a deployed BBD)

  • smoke tests pinned v0.21
  • hub declared >= 0.21
  • hub used features of >= 0.23

Nothing broken in smoke-tests (because it uses dev overrides).
I see two solution for this

  • run foundation mode tests for all hub modules (difficult to achieve, though we have decent coverage across demo + try meshstacks)
  • run smoke-test workflow as a matrix once with and once without the dev override. This means we have a true CI tests for all meshStack components at latest versions and another test of "stable provider version works against latest meshStack"

Does that make sense?

Comment thread tools/scorecard/scorecard.mjs

// ─── THE PROVIDER FLOOR KNOB ────────────────────────────────────────────────
//
// One central place defining the minimum provider version the *entire hub* is

@JohannesRudolph JohannesRudolph Aug 18, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

f: this is unnecessary wordy. just focus on the fact that we want to make sure the hub is verified only against recent versions of the provider

also old providers may become incompatible with never versions of meshStack (e.g. when GA apis retire, or breaking changes in preview APIs)

@grubmeshi

grubmeshi commented Aug 19, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

@JohannesRudolph

  • run foundation mode tests for all hub modules (difficult to achieve, though we have decent coverage across demo + try meshstacks)
  • run smoke-test workflow as a matrix once with and once without the dev override. This means we have a true CI tests for all meshStack components at latest versions and another test of "stable provider version works against latest meshStack"

I'd prefer option 2 for now, as for the foundation tests, I'd actually aim to "dry run" or "test" upgrade paths once hub artifacts/modules/ref archs get properly versions (aka that repo here will be tagged with github releases are written down).

Option 2 is then a good addition to our e2e test harness and can be achieved quite easily I guess?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@JohannesRudolph @grubmeshi