Skip to content

Add Helm template coverage CI and Codecov upload #698

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
NiallTwomey2 wants to merge 9 commits into
base: main
Choose a base branch
from
Open

Add Helm template coverage CI and Codecov upload #698

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
bf81a96
Add Helm template coverage CI and Codecov upload
NiallTwomey2 Oct 28, 2026
7f39072
Exclude vendored helm-coverage.sh from Differential Shellcheck
NiallTwomey2 Oct 28, 2026
2f6ccbf
Exclude vendored helm-coverage.sh from SonarCloud
NiallTwomey2 Oct 28, 2026
8102991
Fetch helm-coverage.sh by SHA and run coverage from test.yaml
NiallTwomey2 Nov 3, 2026
18fe929
Enforce HTTPS on coverage curl and run coverage on nightly
NiallTwomey2 Nov 3, 2026
c25cce2
Fix coverage curl --proto flag
NiallTwomey2 Nov 3, 2026
f6fe47b
ci: run helm dependency build for every coverage chart
NiallTwomey2 Nov 3, 2026
a8660fa
ci: add backstage helm repo for coverage on release-1.8/1.9
NiallTwomey2 Nov 3, 2026
628f7a7
Merge branch 'main' into RHIDP-18349-chart-coverage
rm3l Nov 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 71 additions & 0 deletions .github/workflows/nightly.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,3 +138,74 @@ jobs:
extra_helm_args: ${{ matrix.chart == 'rhdh' && format('--set image.repository={0} --set image.tag={1} --set image.pullPolicy=Always', steps.image.outputs.repo, steps.image.outputs.tag) || matrix.chart == 'backstage' && format('--set upstream.backstage.image.repository={0} --set upstream.backstage.image.tag={1} --set upstream.backstage.image.pullPolicy=Always', steps.image.outputs.repo, steps.image.outputs.tag) || '' }}
# helm_extra_args: general helm flags (e.g. --values), passed to ct install via --helm-extra-args
helm_extra_args: ${{ matrix.chart == 'rhdh' && '--values /tmp/rhdh-nightly-values.yaml' || matrix.chart == 'backstage' && '--values /tmp/backstage-nightly-values.yaml' || '' }}
coverage:

@rm3l rm3l Nov 9, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
coverage:
coverage:

needs: discover-charts
name: Coverage (${{ matrix.branch }} / ${{ matrix.chart }})
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
include: ${{ fromJson(needs.discover-charts.outputs.matrix) }}
Comment thread
rm3l marked this conversation as resolved.
Resolved
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
ref: ${{ matrix.branch }}
fetch-depth: 0

@rm3l rm3l Nov 9, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
fetch-depth: 0

Full history not needed.


- name: Check chart exists on branch
id: check
env:
CHART: charts/${{ matrix.chart }}
run: |
if [[ -d "$CHART" ]]; then
echo "exists=true" >> "$GITHUB_OUTPUT"
else
echo "::notice::Chart $CHART does not exist on branch ${{ matrix.branch }}, skipping"
echo "exists=false" >> "$GITHUB_OUTPUT"
fi

- name: Set up Helm
if: steps.check.outputs.exists == 'true'
uses: azure/setup-helm@9bc31f4ebc9c6b171d7bfbaa5d006ae7abdb4310 # v5.0.1
with:
version: v4.3.1

- name: Run coverage
if: steps.check.outputs.exists == 'true'
env:
CHART_COVERAGE_SHA: 18c41243aca54231d36905185f2f65c231b06a84
CHART: ${{ matrix.chart }}
run: |
set -euo pipefail
curl -fsSL --proto '=https' --tlsv1.2 -o /tmp/helm-coverage.sh \
"https://raw.githubusercontent.com/jCOTINEAU/chart-coverage/${CHART_COVERAGE_SHA}/helm-coverage.sh"
chmod +x /tmp/helm-coverage.sh
mkdir -p coverage-out
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo add backstage https://backstage.github.io/charts
Comment thread
rm3l marked this conversation as resolved.
Resolved
helm repo update
chart_dir="charts/${CHART}"
helm dependency build "$chart_dir"
xml="coverage-out/coverage-${CHART}.xml"
/tmp/helm-coverage.sh --cobertura "$xml" "$chart_dir" "${chart_dir}/ci"
sed -i \
-e "s|filename=\"templates/|filename=\"${chart_dir}/templates/|g" \
-e "s|<source>templates</source>|<source>${chart_dir}</source>|" \
Comment on lines +193 to +194

@rm3l rm3l Nov 9, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
-e "s|filename=\"templates/|filename=\"${chart_dir}/templates/|g" \
-e "s|<source>templates</source>|<source>${chart_dir}</source>|" \
-e "s|<source>templates</source>|<source>${chart_dir}</source>|" \

Why is this transformation needed?
From a quick testing, filename already contains the right path starting from templates/. So it should be fine as is relative to ${chart_dir}, which should be the <source>.

"$xml"

- name: Upload XML artifacts
if: steps.check.outputs.exists == 'true'
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7
with:
name: helm-coverage-xml-${{ matrix.branch }}-${{ matrix.chart }}
Comment thread
rm3l marked this conversation as resolved.
Resolved
Comment on lines +196 to +201

@rm3l rm3l Nov 9, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- name: Upload XML artifacts
if: steps.check.outputs.exists == 'true'
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7
with:
name: helm-coverage-xml-${{ matrix.branch }}-${{ matrix.chart }}

Not needed IMO. When it passes, Codecov already provides a UI to navigate. And not critical if it fails. We can always run it locally if needed.

path: coverage-out/*.xml

- name: Upload coverage to Codecov
if: steps.check.outputs.exists == 'true'
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7
with:
token: ${{ secrets.CODECOV_TOKEN }}
files: coverage-out/coverage-${{ matrix.chart }}.xml
flags: helm-charts
fail_ci_if_error: false
Comment thread
rm3l marked this conversation as resolved.
Resolved
59 changes: 59 additions & 0 deletions .github/workflows/test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,63 @@ jobs:
monitoring_heartbeat: ${{ vars.TEST_MONITORING_HEARTBEAT_ENABLED || 'false' }}
helm_template_values_file: ${{ matrix.chart == 'rhdh' && '/tmp/rhdh-full-plugins-values.yaml' || '' }}

coverage:

@rm3l rm3l Nov 9, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This coverage job looks quite similar in both workflows. I would suggest extracting this into a composite action under .github/actions/chart-coverage/action.yml that both workflows call.

needs: discover-charts
if: needs.discover-charts.outputs.charts != '[]'
name: Coverage (${{ matrix.chart }})
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
chart: ${{ fromJson(needs.discover-charts.outputs.charts) }}
Comment thread
rm3l marked this conversation as resolved.
Resolved
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
fetch-depth: 0
Comment on lines +104 to +105

@rm3l rm3l Nov 9, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
with:
fetch-depth: 0


- name: Set up Helm
uses: azure/setup-helm@9bc31f4ebc9c6b171d7bfbaa5d006ae7abdb4310 # v5.0.1
with:
version: v4.3.1

- name: Run coverage
env:
CHART_COVERAGE_SHA: 18c41243aca54231d36905185f2f65c231b06a84
CHART: ${{ matrix.chart }}
run: |
set -euo pipefail
curl -fsSL --proto '=https' --tlsv1.2 -o /tmp/helm-coverage.sh \
"https://raw.githubusercontent.com/jCOTINEAU/chart-coverage/${CHART_COVERAGE_SHA}/helm-coverage.sh"
Comment on lines +118 to +119

@NiallTwomey2 NiallTwomey2 Nov 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fetch the coverage tool at a pinned SHA (HTTPS-only) - using the curl command

@rm3l rm3l Nov 9, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no need to add a comment like this, which is already obvious by reading the code ;)
This comment doesn't add much context to the PR and actually does bloat the review UI.

chmod +x /tmp/helm-coverage.sh
mkdir -p coverage-out
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo add backstage https://backstage.github.io/charts
Comment thread
rm3l marked this conversation as resolved.
Resolved
helm repo update
chart_dir="charts/${CHART}"
helm dependency build "$chart_dir"
Comment thread
rm3l marked this conversation as resolved.
Resolved
xml="coverage-out/coverage-${CHART}.xml"
/tmp/helm-coverage.sh --cobertura "$xml" "$chart_dir" "${chart_dir}/ci"
sed -i \
-e "s|filename=\"templates/|filename=\"${chart_dir}/templates/|g" \
-e "s|<source>templates</source>|<source>${chart_dir}</source>|" \
Comment on lines +130 to +131

@rm3l rm3l Nov 9, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
-e "s|filename=\"templates/|filename=\"${chart_dir}/templates/|g" \
-e "s|<source>templates</source>|<source>${chart_dir}</source>|" \
-e "s|<source>templates</source>|<source>${chart_dir}</source>|" \

"$xml"
- name: Upload XML artifacts
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7
with:
name: helm-coverage-xml-${{ matrix.chart }}
path: coverage-out/*.xml
Comment on lines +133 to +138

@rm3l rm3l Nov 9, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- name: Upload XML artifacts
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7
with:
name: helm-coverage-xml-${{ matrix.chart }}
path: coverage-out/*.xml


- name: Upload coverage to Codecov
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7
with:
token: ${{ secrets.CODECOV_TOKEN }}
files: coverage-out/coverage-${{ matrix.chart }}.xml
flags: helm-charts
fail_ci_if_error: false
override_commit: ${{ github.event.pull_request.head.sha }}
override_pr: ${{ github.event.number }}
# Aligning job name with the OpenShift CI config: https://github.com/openshift/release/blob/master/core-services/prow/02_config/redhat-developer/rhdh-chart/_prowconfig.yaml#L18
status:
name: Test Latest Release
Expand All @@ -104,3 +161,5 @@ jobs:
exit 1
fi
echo "All chart tests passed (or no charts to test)."

@rm3l rm3l Nov 9, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,3 +16,4 @@ output

# Temporary patch file from sync-upstream-backstage.sh conflict resolution
rhdh-vendored.patch
coverage-out/

@NiallTwomey2 NiallTwomey2 Oct 28, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ignores coverage-out/ so local XML files never get committed

@rm3l rm3l Nov 9, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same point, obvious by reading the code. So no need to add a comment for this ;)

8 changes: 8 additions & 0 deletions codecov.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
coverage:

@NiallTwomey2 NiallTwomey2 Oct 28, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Setting codecov.coverage.status.project and codecov.coverage.status.patch informational to true
to pass the GitHub checks

Chart Covered / total branches Rate
rhdh 71 / 258 27.5%
must-gather 47 / 77 61.0%
orchestrator-infra 16 / 19 84.2%
combined 134 / 354 37.9%

If informational is set to false, which is Codecov's default setting, codecov/project and codecov/patch are failing GitHub checks when project coverage is under target, or patch coverage on changed template lines are under Codecov's patch target, ~70%

@rm3l rm3l Nov 9, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment is much more useful as context for reviewers ;)

status:
project:
default:
informational: true
patch:
default:
informational: true
Loading