ADR-010: Measure Helm template coverage with chart-coverage by NiallTwomey2 · Pull Request #31 · redhat-developer/rhdh-adr

NiallTwomey2 · 2026-10-29T16:14:30Z

Record the RHIDP-13290 outcome: chart-coverage + Codecov for Helm template branch coverage
Treat helmcov as a first-class alternative (requires helm-unittest *_test.yaml; rhdh-chart has none)
Also reject helm-unittest --coverage, Terratest/ct-as-coverage, static values-key coverage

RHIDP-18485
RHIDP-13290
RHIDP-11864

rhdh-qodo-merge · 2026-10-29T16:14:45Z

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0)

Great, no issues found!

Qodo reviewed your code and found no material issues that require review

rhdh-qodo-merge · 2026-10-29T16:15:00Z

PR Summary by Qodo

ADR: Measure Helm template branches with chart-coverage

📝 Documentation 🕐 10-20 Minutes

AI Description

• Record the decision to measure Helm template branch coverage using existing CI values and
 chart-coverage.
• Propose informational Codecov reporting without replacing chart-testing or introducing a merge
 gate.
• Explain why helm-unittest, helmcov, and other coverage approaches do not meet current constraints.

Diagram

graph TD
  Values["CI values"] --> Coverage["chart-coverage"] --> Report["Cobertura XML"] --> Codecov["Codecov checks"]
  Templates["Helm templates"] --> Coverage

High-Level Assessment

The ADR's proposed approach best fits its current constraints: chart-coverage can use existing CI values without introducing test suites or replacing chart-testing. The document considers helm-unittest and helmcov, but both require suites the chart repository does not currently have; they remain options if that changes.

Files changed (1) +85 / -0

Documentation (1) +85 / -0

015-helm-chart-template-coverage.mdDocument the Helm template coverage decision +85/-0

Document the Helm template coverage decision

• Adds an ADR selecting chart-coverage and informational Codecov reporting for branches exercised by existing chart CI values. It records the proposed CI integration, expected limitations, and reasons for rejecting suite-dependent, cluster-based, static-key, and custom-instrumentation alternatives. No coverage workflow or tool is added in this PR.

decisions/015-helm-chart-template-coverage.md

rm3l · 2026-10-30T09:23:48Z

/cc

rm3l · 2026-10-30T09:28:50Z

+**Who is impacted**:
+- Chart authors, who need to see uncovered template branches on PRs
+- QE / Install, who own Helm quality signals
+- Anyone comparing chart coverage to operator Codecov coverage


Suggested change

- Anyone comparing chart coverage to operator Codecov coverage

This is not a goal IMO. It doesn't make sense to compare coverage between those different install methods. The point is to ensure coverage is maintained at a reasonable level and does not degrade as we change code in the chart repo itself.

rm3l · 2026-10-30T09:29:26Z

+
+**Constraints**:
+- Do not rewrite charts or replace chart-testing
+- OSS, pinable, Apache-2.0-compatible


Suggested change

- OSS, pinable, Apache-2.0-compatible

- OSS, pinable

No constraint on the specific license, as long as it OSS and can be used.

rm3l · 2026-10-30T09:30:17Z

+- Do not rewrite charts or replace chart-testing
+- OSS, pinable, Apache-2.0-compatible
+- Codecov-compatible report (Cobertura XML). Go `cover.out` is not required
+- First numbers will be low (local baseline: rhdh ~28%, must-gather ~61%, orchestrator-infra ~84%). Coverage must not become a merge gate on day one


Suggested change

- First numbers will be low (local baseline: rhdh ~28%, must-gather ~61%, orchestrator-infra ~84%). Coverage must not become a merge gate on day one

- Coverage must not become a merge gate on day one

Not necessary to mention this baseline IMO.

rm3l · 2026-10-30T09:31:08Z

+- Codecov-compatible report (Cobertura XML). Go `cover.out` is not required
+- First numbers will be low (local baseline: rhdh ~28%, must-gather ~61%, orchestrator-infra ~84%). Coverage must not become a merge gate on day one
+
+Spike notes: [Google doc](https://docs.google.com/document/d/1iMGzEt_tpxikwQ01hFQphLoEBCWeHrXTlTR1g00NGhg/edit). Implementation: [RHIDP-18349](https://redhat.atlassian.net/browse/RHIDP-18349), [rhdh-chart#698](https://github.com/redhat-developer/rhdh-chart/pull/698).


Could move under a References section at the bottom of the ADR.

rm3l · 2026-10-30T09:32:55Z

+Use **[chart-coverage](https://github.com/jCOTINEAU/chart-coverage)** (`helm-coverage.sh`) to measure Helm **template branch** coverage from existing `ci/` values, and upload Cobertura XML to **Codecov** as an informational check.
+
+**Implementation approach**:
+- Vendor `helm-coverage.sh` in `rhdh-chart` (Apache-2.0), pinned to a SHA. Do not curl `main` in CI


As I commented out in redhat-developer/rhdh-chart#698 (comment), I would suggest not vendoring it. We can curl it directly as needed using a specific commit SHA.
The "Do not curl main" concern is valid (pin to a SHA, not a branch), but that doesn't require vendoring; curl with a pinned SHA achieves the same.

Ah ok, so instead of actively vendoring a copy of the coverage tool, fetching it from the existing repo that is included in CI which is this third party repo the helm chart uses for coverage - repo, instead of vendoring a direct copy of it everytime

rm3l · 2026-10-30T09:34:56Z

+
+**Implementation approach**:
+- Vendor `helm-coverage.sh` in `rhdh-chart` (Apache-2.0), pinned to a SHA. Do not curl `main` in CI
+- New cluster-free GitHub Actions workflow on PRs to `main` / `release-*` and on push to `main`


My comment in redhat-developer/rhdh-chart#698 (comment) suggests hooking coverage into the existing test.yaml (PRs, changed charts only) and nightly.yaml (all charts), not a standalone workflow.

rm3l · 2026-10-30T09:39:30Z

+- ✅ Uncovered template branches become visible on PRs once the `main` baseline exists
+
+### Negative
+- ❌ First rhdh figure is ~28%. That is expected (only `ci/` branches), but it looks “low” next to Go coverage


Suggested change

- ❌ First rhdh figure is ~28%. That is expected (only `ci/` branches), but it looks “low” next to Go coverage

Not necessarily negative. The goal is just to start tracking coverage as a starting point, regardless of what the numbers are.

rm3l · 2026-10-30T09:39:51Z

+
+### Negative
+- ❌ First rhdh figure is ~28%. That is expected (only `ci/` branches), but it looks “low” next to Go coverage
+- ❌ Vendoring ~1.5k lines of upstream bash; we exclude it from Differential ShellCheck and Sonar rather than forking the tool


Suggested change

- ❌ Vendoring ~1.5k lines of upstream bash; we exclude it from Differential ShellCheck and Sonar rather than forking the tool

Vendoring not needed.

NiallTwomey2 · 2026-11-03T13:07:05Z

 **Who is impacted**:
 - Chart authors, who need to see uncovered template branches on PRs
 - QE / Install, who own Helm quality signals
- Anyone comparing chart coverage to operator Codecov coverage


Dropping this as it is not relevant, comparison of chart coverage to operator is not applicable to tracking chart coverage over a period of time - out of scope

NiallTwomey2 · 2026-11-03T13:10:40Z


 **Constraints**:
 - Do not rewrite charts or replace chart-testing
- OSS, pinable, Apache-2.0-compatible


No license constraint, instead it is Open-Source

NiallTwomey2 · 2026-11-03T13:12:48Z

- OSS, pinable, Apache-2.0-compatible
+- OSS, pinable
 - Codecov-compatible report (Cobertura XML). Go `cover.out` is not required
- First numbers will be low (local baseline: rhdh ~28%, must-gather ~61%, orchestrator-infra ~84%). Coverage must not become a merge gate on day one


Removing the base-line numbers 28/61/84, from constraints

NiallTwomey2 · 2026-11-03T13:18:10Z

 Use **[chart-coverage](https://github.com/jCOTINEAU/chart-coverage)** (`helm-coverage.sh`) to measure Helm **template branch** coverage from existing `ci/` values, and upload Cobertura XML to **Codecov** as an informational check.

 **Implementation approach**:
- Vendor `helm-coverage.sh` in `rhdh-chart` (Apache-2.0), pinned to a SHA. Do not curl `main` in CI


No more vendoring a copy of helm-coverage.sh in rhdh-chart, with each CI run, curl pinned SHA instead of vendor.
Updated: fetch helm-coverage.sh from ./chart-coverage from a pinned SHA.

NiallTwomey2 · 2026-11-03T13:32:00Z

+- Hook coverage into existing workflows; do not add a standalone coverage workflow
+- **PRs (`test.yaml`)**: coverage only for charts already in that workflow's matrix (changed charts; all non-excluded charts if CI files changed)
+- **Nightly (`nightly.yaml`)**: coverage for every supported chart the nightly matrix already lists (`ct.yaml` exclusions unchanged)


Do not vendor, use curl + SHA.
Replaced "new standalone workflow" with hook into test.yaml and nightly.yaml.

test.yaml - only charts what the PR already tests.
nightly.yaml - every supported chart the nightly job (which is tested once a day) already lists (ct.yaml)

NiallTwomey2 · 2026-11-03T13:33:11Z

- ❌ First rhdh figure is ~28%. That is expected (only `ci/` branches), but it looks “low” next to Go coverage
- ❌ Vendoring ~1.5k lines of upstream bash; we exclude it from Differential ShellCheck and Sonar rather than forking the tool


Deleting this as it is not relevant, we are measuring against the baseline coverage which is 28%

NiallTwomey2 · 2026-11-03T13:36:00Z

- ❌ First rhdh figure is ~28%. That is expected (only `ci/` branches), but it looks “low” next to Go coverage
- ❌ Vendoring ~1.5k lines of upstream bash; we exclude it from Differential ShellCheck and Sonar rather than forking the tool


No need to vendor a copy, replacing the negatives in regard to vendoring, with SHA-bump trade off

Co-authored-by: Armel Soro <asoro@redhat.com>

rm3l · 2026-11-04T17:26:29Z

@NiallTwomey2 From https://github.com/redhat-developer/rhdh-adr/actions/runs/41816281618?pr=31, please rename the decision file accordingly:

⚠️ Recommended to name the next ADR file as: decisions/010-helm-chart-template-coverage to follow the order of files in decisions/ folder of rhdh-adr.

/retitle ADR-010: Measure Helm template coverage with chart-coverage

… ADR's

NiallTwomey2 · 2026-11-05T08:47:07Z

Ack, I understand, renamed the ADR file to 010-helm-chart-template-coverage to reflect the sequences of live ADR files on main - as per the suggestion

ADR: Measure Helm template coverage with chart-coverage

297963c

openshift-ci Bot requested a review from rm3l October 30, 2026 09:23

rm3l requested changes Oct 30, 2026

View reviewed changes

rm3l changed the title ~~ADR: Measure Helm template coverage with chart-coverage~~ ADR-015: Measure Helm template coverage with chart-coverage Oct 30, 2026

ADR: pin coverage tool by SHA and hook into existing CI

ed8e016

NiallTwomey2 commented Nov 3, 2026

View reviewed changes

Merge branch 'main' into adr/015-helm-chart-template-coverage

c2b3a32

rm3l reviewed Nov 4, 2026

View reviewed changes

NiallTwomey2 and others added 9 commits November 4, 2026 10:39

Update decisions/015-helm-chart-template-coverage.md

63f4e7c

Co-authored-by: Armel Soro <asoro@redhat.com>

Update decisions/015-helm-chart-template-coverage.md

1361922

Co-authored-by: Armel Soro <asoro@redhat.com>

Update decisions/015-helm-chart-template-coverage.md

ab3ac28

Co-authored-by: Armel Soro <asoro@redhat.com>

Update decisions/015-helm-chart-template-coverage.md

8867ca6

Co-authored-by: Armel Soro <asoro@redhat.com>

Update decisions/015-helm-chart-template-coverage.md

645dedb

Co-authored-by: Armel Soro <asoro@redhat.com>

Update decisions/015-helm-chart-template-coverage.md

ced3ffe

Co-authored-by: Armel Soro <asoro@redhat.com>

Update decisions/015-helm-chart-template-coverage.md

583c505

Co-authored-by: Armel Soro <asoro@redhat.com>

Update decisions/015-helm-chart-template-coverage.md

8b9687b

Co-authored-by: Armel Soro <asoro@redhat.com>

Update decisions/015-helm-chart-template-coverage.md

c9ad79f

Co-authored-by: Armel Soro <asoro@redhat.com>

NiallTwomey2 requested a review from rm3l November 4, 2026 11:58

rm3l approved these changes Nov 4, 2026

View reviewed changes

openshift-ci Bot changed the title ~~ADR-015: Measure Helm template coverage with chart-coverage~~ ADR-010: Measure Helm template coverage with chart-coverage Nov 4, 2026

Renaming Helm template coverage ADR file to reflect the order of live…

f31d2b2

… ADR's

NiallTwomey2 requested a review from rm3l November 5, 2026 08:47

rm3l merged commit 8c2dd56 into main Nov 5, 2026
1 check passed

rm3l deleted the adr/015-helm-chart-template-coverage branch November 5, 2026 10:46

	- First numbers will be low (local baseline: rhdh ~28%, must-gather ~61%, orchestrator-infra ~84%). Coverage must not become a merge gate on day one
	- Coverage must not become a merge gate on day one

		- ❌ First rhdh figure is ~28%. That is expected (only `ci/` branches), but it looks “low” next to Go coverage
		- ❌ Vendoring ~1.5k lines of upstream bash; we exclude it from Differential ShellCheck and Sonar rather than forking the tool

Conversation

NiallTwomey2 commented Oct 29, 2026

Uh oh!

rhdh-qodo-merge Bot commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review by Qodo

Great, no issues found!

Uh oh!

rhdh-qodo-merge Bot commented Oct 29, 2026

PR Summary by Qodo

Uh oh!

rm3l commented Oct 30, 2026

Uh oh!

rm3l Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

NiallTwomey2 Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

NiallTwomey2 Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

NiallTwomey2 Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

NiallTwomey2 Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rm3l commented Nov 4, 2026

Uh oh!

NiallTwomey2 commented Nov 5, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

rhdh-qodo-merge Bot commented Oct 29, 2026 •

edited

Loading

rm3l Oct 30, 2026 •

edited

Loading

NiallTwomey2 Nov 3, 2026 •

edited

Loading

NiallTwomey2 Nov 3, 2026 •

edited

Loading

NiallTwomey2 Nov 3, 2026 •

edited

Loading

NiallTwomey2 Nov 3, 2026 •

edited

Loading