Govern the solution server hint prompt as a versioned Jinja2 template - #930
Open
fabianvf wants to merge 3 commits into
Open
Govern the solution server hint prompt as a versioned Jinja2 template#930fabianvf wants to merge 3 commits into
fabianvf wants to merge 3 commits into
Conversation
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (6)
Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review. 📝 WalkthroughWalkthroughThis PR adds prompt governance documentation, a manifest-driven Jinja2 renderer, a ChangesPrompt governance and rendering
Tooling updates
Estimated code review effort: 4 (Complex) | ~45 minutes Merge Risk: ⚪ Minimal · up to The change is merge-ready after normal checks and review; no actionable merge-blocking risk remains. Sequence Diagram(s)sequenceDiagram
participant Server as server.generate_hint_v3
participant PromptAPI as render_prompt
participant Jinja as Jinja2 Environment
participant Template as generate_hint_v3.md.jinja
Server->>PromptAPI: Pass incidents and ast_diff_str
PromptAPI->>Jinja: Resolve generate_hint_v3
Jinja->>Template: Render template with context
Template-->>Jinja: Return prompt text
Jinja-->>PromptAPI: Return rendered prompt
PromptAPI-->>Server: Return final prompt string
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Full details: Docstring CoverageExplanation Docstring coverage is 20.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 15 functions across 6 files. (3 skipped: 3 unsupported.) ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
fabianvf
force-pushed
the
prompt-template-governance
branch
3 times, most recently
from
June 24, 2026 20:55
68e68e4 to
e007b11
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (2)
kai_mcp_solution_server/tests/prompts/test_semantic_regression.py (1)
69-73: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAssert all incident fields that the prompt contract depends on.
The regression currently checks only a subset of incident fields. Add assertions for
code_snip,line_number, andvariablesto catch accidental template regressions in those sections.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@kai_mcp_solution_server/tests/prompts/test_semantic_regression.py` around lines 69 - 73, The semantic regression test is only validating a subset of the incident data rendered by the prompt. Update the assertions in test_semantic_regression to also check code_snip, line_number, and variables for each incident, alongside the existing uri, message, and violation_name checks, so the prompt contract covers all fields the template depends on..github/workflows/prompt-validation.yml (1)
25-35: 🔒 Security & Privacy | 🔵 Trivial | 💤 Low valueOptional: align with the repo's action-pinning / credential-hygiene policy.
zizmorflags theseuses:references as unpinned (not pinned to a commit SHA) and notesactions/checkoutdoesn't setpersist-credentials: false. If the repo enforces a blanket pinning policy, pin to immutable SHAs and disable credential persistence on checkout; otherwise this is consistent with existing workflows and can be deferred.🔒 Suggested hardening
- uses: actions/checkout@<commit-sha> # v4 with: persist-credentials: false🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/prompt-validation.yml around lines 25 - 35, The workflow step in the prompt-validation job uses unpinned GitHub Actions and checkout still persists credentials. If this repo follows the pinning/credential-hygiene policy, update the actions referenced by actions/checkout, actions/setup-python, and astral-sh/setup-uv in this job to immutable commit SHAs, and set persist-credentials to false on the actions/checkout step; otherwise leave it as-is if the repo intentionally allows this pattern.Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@kai_mcp_solution_server/PROMPT_GOVERNANCE.md`:
- Around line 75-76: The required status-check name in PROMPT_GOVERNANCE should
match the exact workflow/job label reported by the CI system. Update the status
check entry from Validate prompt templates to the actual check name used by the
prompt validation workflow, and keep the Code Owners requirement unchanged so
the documented branch protection gate aligns with the status shown by the Prompt
Template Validation (ISO 42001 A.5.2) job.
In `@kai_mcp_solution_server/scripts/prompts_version.py`:
- Around line 46-111: `check()` currently validates template integrity but
misses the manifest↔package version contract, so add a version lockstep
validation between `manifest.yaml` and the package version in `pyproject.toml`.
Update `check()` in prompts_version.py to load/compare both versions, report a
clear problem when they differ, and fail the check alongside the existing
`problems` list; use the existing `load_manifest()` flow and extend it with a
helper or inline read of the package version so the CI gate catches drift.
---
Nitpick comments:
In @.github/workflows/prompt-validation.yml:
- Around line 25-35: The workflow step in the prompt-validation job uses
unpinned GitHub Actions and checkout still persists credentials. If this repo
follows the pinning/credential-hygiene policy, update the actions referenced by
actions/checkout, actions/setup-python, and astral-sh/setup-uv in this job to
immutable commit SHAs, and set persist-credentials to false on the
actions/checkout step; otherwise leave it as-is if the repo intentionally allows
this pattern.
In `@kai_mcp_solution_server/tests/prompts/test_semantic_regression.py`:
- Around line 69-73: The semantic regression test is only validating a subset of
the incident data rendered by the prompt. Update the assertions in
test_semantic_regression to also check code_snip, line_number, and variables for
each incident, alongside the existing uri, message, and violation_name checks,
so the prompt contract covers all fields the template depends on.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 58245fea-789e-4024-9729-d5c539b30002
⛔ Files ignored due to path filters (1)
kai_mcp_solution_server/uv.lockis excluded by!**/*.lock
📒 Files selected for processing (16)
.github/CODEOWNERS.github/workflows/prompt-validation.yml.trunk/trunk.yamlkai_mcp_solution_server/PROMPT_GOVERNANCE.mdkai_mcp_solution_server/docs/ISO-42001-A5.2-evidence.mdkai_mcp_solution_server/pyproject.tomlkai_mcp_solution_server/requirements.txtkai_mcp_solution_server/scripts/prompts_version.pykai_mcp_solution_server/src/kai_mcp_solution_server/prompts/__init__.pykai_mcp_solution_server/src/kai_mcp_solution_server/prompts/manifest.yamlkai_mcp_solution_server/src/kai_mcp_solution_server/prompts/templates/generate_hint_v3.md.jinjakai_mcp_solution_server/src/kai_mcp_solution_server/server.pykai_mcp_solution_server/tests/prompts/__init__.pykai_mcp_solution_server/tests/prompts/oracle.pykai_mcp_solution_server/tests/prompts/test_parity.pykai_mcp_solution_server/tests/prompts/test_semantic_regression.py
Comment on lines
+75
to
+76
| 1. **Require the status check** `Validate prompt templates`. | ||
| 2. **Require review from Code Owners** so `prompts/**` edits need `@konveyor/kai-prompt-reviewers`. |
Contributor
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
Align required status-check name with the actual workflow/job check.
Line 75 requires Validate prompt templates, but the workflow context is named Prompt Template Validation (ISO 42001 A.5.2). If these don’t match the actual reported check name, branch protection cannot enforce this gate as documented.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@kai_mcp_solution_server/PROMPT_GOVERNANCE.md` around lines 75 - 76, The
required status-check name in PROMPT_GOVERNANCE should match the exact
workflow/job label reported by the CI system. Update the status check entry from
Validate prompt templates to the actual check name used by the prompt validation
workflow, and keep the Code Owners requirement unchanged so the documented
branch protection gate aligns with the status shown by the Prompt Template
Validation (ISO 42001 A.5.2) job.
Extract the live generate_hint_v3 prompt out of server.py into a versioned Jinja2 asset under src/kai_mcp_solution_server/prompts/, rendered via render_prompt(). The incident loop now lives in the template; the AST-diff computation stays in Python and is passed in as a variable. This brings the solution server's hint prompt under the same governance the editor-extensions repo adopted for its prompts (ISO 42001 A.5.2) — the surface that work explicitly left to be governed here. - prompts/: Jinja2 template + manifest (semver + checksum), render registry; ships as package data (verified present in the built wheel) - byte-exact parity tests prove the template reproduces the previous prompt string exactly across the incident / AST-diff matrix, plus a deterministic mock-model semantic-regression check - scripts/prompts_version.py + prompt-validation.yml validate Jinja syntax, declared variables, and manifest checksums on any prompts/** change - PROMPT_GOVERNANCE.md, a new CODEOWNERS, and an audit-evidence doc - remove the never-called generate_hint_v1/v2 (dead code) and the now-unused get_diff import Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Fabian von Feilitzsch <fabian@fabianism.us>
Trunk Check fails on any PR touching Markdown or TOML because two tools can't
run in CI:
- markdownlint@0.48.0 crashes at startup under the bundled node 18.12.1
("SyntaxError: Invalid regular expression flags" — string-width uses the
unicode-sets `v` regex flag, which needs node 20+).
- taplo@0.10.0's Linux binary fails to install ("Binary not found") with the
old plugins definitions.
Bump the node runtime to 22.16.0, the trunk-io/plugins ref to v1.10.2 (refreshes
tool download definitions; pinned tool versions are unchanged), and markdownlint
to 0.49.0. `trunk check` reports no new issues across the changed files.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Fabian von Feilitzsch <fabian@fabianism.us>
From CodeRabbit's review: - prompts_version.py: the check now fails if the prompt-set version in manifest.yaml drifts from the package version in pyproject.toml, enforcing the lockstep the governance doc documents. - test_semantic_regression.py: also assert code_snip, line_number, and variables render, so the prompt contract covers every incident field the template depends on, not just uri/message/violation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Fabian von Feilitzsch <fabian@fabianism.us>
fabianvf
force-pushed
the
prompt-template-governance
branch
from
August 25, 2026 18:16
009c511 to
6efdcc7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Moves the solution server's live hint prompt (
generate_hint_v3) out of an inline f-string inserver.pyand into a versioned Jinja2 template underkai_mcp_solution_server/src/kai_mcp_solution_server/prompts/, rendered through a smallrender_prompt(...)registry. A checksummedmanifest.yamltracks it.Why
This is the Kai side of the prompt-template governance we just landed in editor-extensions (ISO 42001 A.5.2). That work explicitly scoped out the solution-server prompts, noting they're owned here — this closes that loop. The hint prompt is effectively the instruction set for hint generation, so it gets version control, peer review, and CI validation independent of the application code. The incident loop now lives in the template; the AST-diff computation stays in Python and is passed in as a variable.
Behavior
No behavior change for the live prompt. A parity suite renders the template and asserts it matches a verbatim copy of the old f-string byte-for-byte across the incident / AST-diff matrix (0/1/N incidents, empty/populated diff, multi-line and special-char fields). Jinja renders
None/dicts identically to the old f-strings, so there's no silent drift.generate_hint_v1andgenerate_hint_v2were never called (only v3 is wired up, atserver.py:1098) — they're removed as dead code, along with the now-unusedget_diffimport.What's here
prompts/— the Jinja2 template + manifest and the render registry; shipped as package data (verified present in the built wheel).scripts/prompts_version.py+.github/workflows/prompt-validation.yml— validate Jinja syntax, declared-vs-referenced variables, and manifest checksums on anyprompts/**change, plus the parity and a deterministic (mock-model) semantic-regression test.PROMPT_GOVERNANCE.md, a new.github/CODEOWNERS(scoped to the prompts dir only), and an audit-evidence doc.Verification
uv build(template + manifest in the wheel),pytest tests/prompts(7 pass),python scripts/prompts_version.py check,mypy src, and ruff/Trunk all green.requirements.txtregenerated for the newjinja2dependency.Needs a repo admin (can't be done in code)
@konveyor/kai-prompt-reviewersteam.Validate prompt templatescheck + Code-Owner review on the prompts directory (documented inPROMPT_GOVERNANCE.md).🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Bug Fixes
Documentation