security(ci): evaluate the excepted RBAC controls in the posture gate - #3057
Conversation
The Kubescape gate scanned one framework (NSA-CISA, 20 controls) while the ClusterSecurityException CRs name 76. 59 excepted controls were outside the gate, including every RBAC control those CRs exist to govern, so their findings reached neither the compliance score nor Code Scanning. Add MITRE ATT&CK, which evaluates 10 of them. Measured locally on ksail 7.178.20: NSA 99.28, MITRE 96.03, combined 97 against the unchanged floor of 95. Part of #2823
Self-review caught that the first commit conflated two different counts. MITRE adds 10 controls to the SCAN (nsa 20, mitre 17, union 30) but only 7 of them are controls we write exceptions for; the untouched residue is 52, not 49. Names the seven, and states the distinction so the two numbers are not conflated again.
✅
|
ci.yaml claimed the human-readable posture table reaches the log unchanged under --format sarif. It does not: the Overall compliance-score line is absent, measured on this PR's own CI run and reproduced locally. That made the re-baselining instruction directly above it impossible to follow from the log. The threshold is unaffected and still gates — ablated at 95/99/100 under --format sarif for exit 0/1/1 against a combined score of 97. Observability gap filed as #3058.
@coderabbitai review |
|
✅ Action performedReview finished.
|
Exercised as a user, and what the evidence actually showsRan the real command, both arms. Not the change traced through code — the exact CI invocation, Does the gate still gate? This was the one that mattered: with
against a combined score of 97. It gates. (The same ablation under Does MITRE actually get evaluated on the runner, not just on my laptop? Yes, and the CI log Do the exceptions still apply? 20/20 checks green at One thing I deliberately did NOT changeThe SARIF upload still uses Scope I stopped at52 of the 76 excepted controls are in neither framework and remain invisible to the gate. That is |
📝 WalkthroughWalkthroughKubescape validation now scans the NSA-CISA and MITRE ATT&CK frameworks. The 95 threshold applies to the combined score. Both validation workflows enforce matching framework coverage. SARIF output and exception generation remain enabled. New tests cover parsing, invalid configurations, decoy commands, missing scans, duplicate scans, workflow parity, and CI wiring. Documentation records coverage and exception behavior. Possibly related issues
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/ci.yaml:
- Around line 603-640: Add an executable offline regression test near the
Kubescape workflow step that uses the pinned KSAIL_VERSION and invokes the same
`ksail workload scan` configuration with both `nsa,mitre` frameworks. Provide
fixture findings covering a MITRE-only case, then assert combined framework
coverage and the threshold exit-code matrix: 95 succeeds while 99 and 100 fail.
Replace the comment-only ablation claim with this runnable check while
preserving the production SARIF scan gate.
In `@AGENTS.md`:
- Line 103: Correct the NSA coverage statement in the scan description:
distinguish the 20 controls evaluated by NSA from the 17 controls named by
ClusterSecurityException resources, or state only the accurate exception-control
count of 17. Keep the existing total-control and framework coverage context
unchanged.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: bd19688e-654b-40da-bc2d-dce7c1856047
📒 Files selected for processing (2)
.github/workflows/ci.yamlAGENTS.md
📜 Review details
🧰 Additional context used
📓 Path-based instructions (1)
.github/workflows/*.yaml
📄 CodeRabbit inference engine (AGENTS.md)
CI for Kubernetes-related changes must perform static KSail validation and Kubescape scanning; do not rely on full-cluster system tests.
Files:
.github/workflows/ci.yaml
🧠 Learnings (2)
📚 Learning: 2026-07-28T00:33:10.828Z
Learnt from: devantler
Repo: devantler-tech/platform PR: 2825
File: .github/workflows/dr-rebuild.yaml:70-73
Timestamp: 2026-07-28T00:33:10.828Z
Learning: For GitHub Actions concurrency configuration in this repo, don’t use `queue: single` on a shared production concurrency group (e.g., `prod-deploy`) that is used for normal production deployments—`queue: single` will cancel/replace any pending run in that group. Normal production deployments must use a uniform `queue: max` for that shared group to ensure serialization without coalescing. If you need DR rebuild coalescing (replacement of pending DR rebuild runs), it must use a separate DR-only concurrency scope/group (not the shared production `prod-deploy` group).
Applied to files:
.github/workflows/ci.yaml
📚 Learning: 2026-07-29T07:29:34.804Z
Learnt from: devantler
Repo: devantler-tech/platform PR: 2848
File: scripts/validate-publication-order/main.go:315-353
Timestamp: 2026-07-29T07:29:34.804Z
Learning: When reviewing GitHub Actions workflow steps with `run:` blocks, remember that the default non-interactive Bash invocation won’t expand shell aliases unless the script explicitly enables it (e.g., `shopt -s expand_aliases`). Therefore, when assessing potential command-name shadowing (e.g., whether an alias could change what command runs), only treat alias-based behavior as possible if the step explicitly opts into alias expansion via `shopt -s expand_aliases` (or equivalent).
Applied to files:
.github/workflows/ci.yaml
🪛 LanguageTool
AGENTS.md
[uncategorized] ~101-~101: The official name of this software platform is spelled with a capital “H”.
Context: ...thoritative list is the k8s filter in .github/workflows/ci.yaml) — the validate jo...
(GITHUB)
[uncategorized] ~101-~101: The official name of this software platform is spelled with a capital “H”.
Context: ...flows/ci.yaml) — the validatejob in.github/workflows/ci.yaml` first json-parses ev...
(GITHUB)
[uncategorized] ~105-~105: The official name of this software platform is spelled with a capital “H”.
Context: ...wire (kept in sync with the comments in .github/workflows/ci.yaml): if validate outp...
(GITHUB)
[style] ~106-~106: ‘whether or not’ might be wordy. Consider a shorter alternative.
Context: ...tside both frameworks is inert here whether or not it is correct, so never read "the score...
(EN_WORDINESS_PREMIUM_WHETHER_OR_NOT)
🔇 Additional comments (2)
.github/workflows/ci.yaml (1)
570-586: LGTM!AGENTS.md (1)
101-102: LGTM!Also applies to: 104-107
Dropping a framework from the Kubescape gate REMOVES findings, so the compliance score rises and every check stays green. That is a coverage regression shaped like an improvement, and it is what left 59 excepted controls un-gated. Add a guard asserting the scan still names every required framework, with a fail-closed floor so renaming the step cannot make it pass vacuously, plus 11 RED-proved assertions and CI wiring for both halves. Also corrects AGENTS.md line 103, which stated NSA's total control count (20) where it meant the excepted-control subset (17). Part of #2823
@coderabbitai review |
|
|
CodeRabbit is rate limited at this head ( @codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4a06974e6e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Two review findings, both real. validate-main.yaml still scanned nsa only while uploading under the same Code Scanning category as the PR analysis. That category is the durable baseline PR alerts are diffed against, so every MITRE finding would have failed to persist as a main-branch alert — and a direct push to main, which bypasses the merge queue, was ungated on MITRE entirely. Both workflows now scan nsa,mitre and the guard checks both. The guard also read raw YAML, so a stale comment naming both frameworks could satisfy it while the real command used a variable. Comments are now excluded. 16 assertions, up from 11. Part of #2823
Findings fixed and threads resolved, so restarting the review loop at CodeRabbit per lane priority. @coderabbitai review |
|
|
CodeRabbit is rate limited at this head ( @codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 68010ef94d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Three review findings, two fixed here. The guard checked only that each workflow contained the required members, so nsa,mitre,pss in ci.yaml against an nsa,mitre baseline passed — and one of my own tests asserted that as correct, encoding the bug. Both workflows must now carry identical, normalised sets; ordering and repetition are ignored. The guard also ran only on the PR path, so a direct push to main could reduce that workflow's own scan with nothing checking it. validate-main.yaml now runs the guard before it scans. 20 assertions, up from 16. The raw-YAML residual (an output-only command that merely prints the invocation) is tracked separately rather than blacklisted. Part of #2823
All findings fixed and threads resolved; restarting the review loop at CodeRabbit. @coderabbitai review |
|
✅ Action performedReview finished.
|
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/guard-kubescape-gate-frameworks.sh`:
- Around line 169-177: Update scan_invocations in
scripts/guard-kubescape-gate-frameworks.sh (169-177) to accept only supported
run: lines whose first non-blank command token is the direct executable ksail
followed by workload scan, failing closed for echo or other wrapper commands.
Add the rejected echo decoy case in
scripts/tests/test-kubescape-gate-frameworks-guard.sh (109-116) and verify it is
not treated as a scan invocation.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: a31aeae3-8362-490b-9cc0-81c8e3ecec8a
📒 Files selected for processing (5)
.github/workflows/ci.yaml.github/workflows/validate-main.yamlAGENTS.mdscripts/guard-kubescape-gate-frameworks.shscripts/tests/test-kubescape-gate-frameworks-guard.sh
📜 Review details
🧰 Additional context used
📓 Path-based instructions (2)
.github/workflows/*.yaml
📄 CodeRabbit inference engine (AGENTS.md)
.github/workflows/*.yaml: CI for Kubernetes-related changes must use static manifest validation and Kubescape scanning; there is no full-cluster system test.
Do not lower the Kubescape compliance floor to accommodate findings; add minimal, justifiedClusterSecurityExceptionresources when appropriate.
Files:
.github/workflows/validate-main.yaml.github/workflows/ci.yaml
**/*
📄 CodeRabbit inference engine (AGENTS.md)
**/*: Use Conventional Commit prefixes such asfeat:,fix:, orchore:for commits and PR titles.
Keep changes small and focused, create PRs as drafts, and never push directly tomain.
Files:
AGENTS.mdscripts/tests/test-kubescape-gate-frameworks-guard.shscripts/guard-kubescape-gate-frameworks.sh
🧠 Learnings (3)
📚 Learning: 2026-07-28T00:33:10.828Z
Learnt from: devantler
Repo: devantler-tech/platform PR: 2825
File: .github/workflows/dr-rebuild.yaml:70-73
Timestamp: 2026-07-28T00:33:10.828Z
Learning: For GitHub Actions concurrency configuration in this repo, don’t use `queue: single` on a shared production concurrency group (e.g., `prod-deploy`) that is used for normal production deployments—`queue: single` will cancel/replace any pending run in that group. Normal production deployments must use a uniform `queue: max` for that shared group to ensure serialization without coalescing. If you need DR rebuild coalescing (replacement of pending DR rebuild runs), it must use a separate DR-only concurrency scope/group (not the shared production `prod-deploy` group).
Applied to files:
.github/workflows/validate-main.yaml.github/workflows/ci.yaml
📚 Learning: 2026-07-29T07:29:34.804Z
Learnt from: devantler
Repo: devantler-tech/platform PR: 2848
File: scripts/validate-publication-order/main.go:315-353
Timestamp: 2026-07-29T07:29:34.804Z
Learning: When reviewing GitHub Actions workflow steps with `run:` blocks, remember that the default non-interactive Bash invocation won’t expand shell aliases unless the script explicitly enables it (e.g., `shopt -s expand_aliases`). Therefore, when assessing potential command-name shadowing (e.g., whether an alias could change what command runs), only treat alias-based behavior as possible if the step explicitly opts into alias expansion via `shopt -s expand_aliases` (or equivalent).
Applied to files:
.github/workflows/validate-main.yaml.github/workflows/ci.yaml
📚 Learning: 2026-08-10T13:01:12.782Z
Learnt from: devantler
Repo: devantler-tech/platform PR: 3057
File: .github/workflows/ci.yaml:622-659
Timestamp: 2026-08-10T13:01:12.782Z
Learning: Repository shell tests and scripts must remain compatible with macOS Bash 3.2. Do not use Bash 4+ features such as `mapfile`; use portable constructs, such as a `while IFS= read -r` loop, instead.
Applied to files:
scripts/tests/test-kubescape-gate-frameworks-guard.shscripts/guard-kubescape-gate-frameworks.sh
🪛 ast-grep (0.45.0)
scripts/tests/test-kubescape-gate-frameworks-guard.sh
[warning] 53-53: set +e (or set +o errexit) disables the shell's errexit option, so the script keeps running after a command fails. This masks failures of security-critical operations (downloads, signature/checksum verification, permission changes, cleanup of secrets), letting the script proceed with a bad or insecure state. Leave errexit enabled (set -e / set -euo pipefail), or handle failures explicitly with if/|| and an explicit exit instead of globally turning off failure detection.
Context: set +e
Note: [CWE-754] Improper Check for Unusual or Exceptional Conditions.
(set-plus-e-error-masking-bash)
[warning] 138-138: set +e (or set +o errexit) disables the shell's errexit option, so the script keeps running after a command fails. This masks failures of security-critical operations (downloads, signature/checksum verification, permission changes, cleanup of secrets), letting the script proceed with a bad or insecure state. Leave errexit enabled (set -e / set -euo pipefail), or handle failures explicitly with if/|| and an explicit exit instead of globally turning off failure detection.
Context: set +e
Note: [CWE-754] Improper Check for Unusual or Exceptional Conditions.
(set-plus-e-error-masking-bash)
🔇 Additional comments (5)
scripts/guard-kubescape-gate-frameworks.sh (1)
1-158: LGTM!Also applies to: 180-180
scripts/tests/test-kubescape-gate-frameworks-guard.sh (1)
1-108: LGTM!Also applies to: 118-173
.github/workflows/ci.yaml (1)
61-68: LGTM!Also applies to: 157-159, 468-475, 589-659
.github/workflows/validate-main.yaml (1)
175-200: LGTM!AGENTS.md (1)
101-107: LGTM!
CI is fully green at this head and all twelve review threads are resolved — including the three P2s from the last Codex round, each fixed with a paired RED/GREEN arm rather than reasoning. A new head opens a new round, so restarting the loop at CodeRabbit per lane priority. @coderabbitai review |
|
|
CodeRabbit is rate limited at this head ( Your three P2s from the previous head are fixed, each with a paired arm rather than an argument: the punctuated-suffix sets now reject (pre-fix they were accepted), two scan invocations now reject (pre-fix rc 0), and the wiring assertions read parsed @codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 64411b578a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The greedy --framework read judged the line on the throwaway scan. Codex, #3057.
CI is green at this head and all thirteen threads are resolved. Codex's fourth P2 — a greedy @coderabbitai review |
🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
echo '=== Changed files ==='
git diff --name-status "$(git merge-base HEAD origin/main)" HEAD 2>/dev/null || \
git diff --name-status HEAD~1 HEAD
echo '=== Diff stat ==='
git diff --stat "$(git merge-base HEAD origin/main)" HEAD 2>/dev/null || \
git diff --stat HEAD~1 HEAD
echo '=== Bash script outlines ==='
ast-grep outline scripts/guard-kubescape-gate-frameworks.sh --items all 2>/dev/null || true
ast-grep outline scripts/tests/test-kubescape-gate-frameworks-guard.sh --items all 2>/dev/null || true
echo '=== Guard script ==='
cat -n scripts/guard-kubescape-gate-frameworks.sh
echo '=== Guard tests ==='
cat -n scripts/tests/test-kubescape-gate-frameworks-guard.sh
echo '=== Kubescape references in workflows ==='
rg -n -C 12 --glob '*.yaml' --glob '*.yml' '(kubescape|guard-kubescape-gate-frameworks|--framework|compliance-threshold)' .github/workflowsLength of output: 73754 🏁 Script executed: #!/bin/bash
set -euo pipefail
base="$(git merge-base HEAD origin/main 2>/dev/null || git rev-parse HEAD~1)"
echo '=== Changed file modes ==='
git diff --summary "$base" HEAD
git ls-tree HEAD scripts/guard-kubescape-gate-frameworks.sh \
scripts/tests/test-kubescape-gate-frameworks-guard.sh
echo '=== Changed hunks: guard and tests ==='
git diff --unified=35 "$base" HEAD -- \
scripts/guard-kubescape-gate-frameworks.sh \
scripts/tests/test-kubescape-gate-frameworks-guard.sh \
AGENTS.md
echo '=== Repeated and malformed --framework fixture coverage ==='
rg -n -C 2 --glob '*.sh' \
'(--framework[^[:space:]].*--framework|nsa,,mitre|nsa,mitre,|,--framework|framework_tokens|framework_argument)' \
scripts
echo '=== Workflow parse and invocation context ==='
sed -n '45,75p;450,480p;585,665p' .github/workflows/ci.yaml
sed -n '165,205p' .github/workflows/validate-main.yamlLength of output: 66835
✅ Action performedReview finished.
|
Exercised at this head, and what the evidence showsRan the real guard against the real workflows, not the change traced through code. Every fix has a paired arm measured on this branch, because a guard that only ever passes proves nothing:
Directly measured the greedy read rather than inferring it: on 30 assertions (from 24), Scoped honestlyThis closes the command-shape and YAML-context classes. A comment inside a One defect found while building the wiring check, worth recording because it fails in the dangerous direction: |
CodeRabbit was asked first at this head (marker @codex review |
|
Codex Review: Didn't find any major issues. Swish! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Green review secured at Codex reviewed this head at 17:30:08Z ( CodeRabbit was asked first, per lane priority, and completed at this head without a gate-satisfying artifact — its verdict named no sha — so this advanced one lane rather than spending a second provider after a success. Merge is blocked by platform#3065, re-verified live this run. This PR merges as soon as the fence is cleared; nothing else is outstanding on it. |
1 participant
Why
The security gate that protects
mainwas only looking at a fraction of what wewrite security exceptions for.
Our justified exceptions cover 76 distinct security controls. The gate scanned a
single framework that checks 17 of them — so findings on the other 59 reached
neither the pass/fail score nor the security alerts. That included every access-control
rule those exception files exist to govern, which is why they have repeatedly
looked ineffective: nothing was measuring them.
What
The gate now checks two frameworks instead of one, bringing 7 of the missing
controls under it — three of which have real findings today. The pass/fail floor
is unchanged, and the combined score sits above it.
Expect more security alerts on this repo. These are pre-existing findings becoming
visible, not new problems introduced here.
One thing worth your judgement: the score's margin over the floor is thinner than
before — 97 against a floor of 95, where the single framework scored 99. That is
the honest cost of measuring more. My own CI run is the real test of whether the
margin holds on the CI runner, which has historically scored differently from a
laptop.
52 excepted controls are in neither framework and stay invisible. That residue is
recorded on the issue rather than fixed here.
Part of #2823