Repository navigation
test: cover policy hardening regressions - #4
Merged
Merged
Conversation
Reviewer's GuideAdds regression-focused tests around policy engine trust handling, decorators, loader validation, regex safety, audit integrity, and trust metadata to harden fail-closed and compatibility behaviors. File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (6)
📝 WalkthroughWalkthroughThe test suite adds regression coverage for fail-closed decorator behavior, trust warnings and denials, loader and model validation, unsafe regex rejection, schema hash validation, and legacy audit integrity verification. ChangesRegression Test Coverage
Estimated code review effort: 3 (Moderate) | ~20 minutes ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've found 1 issue, and left some high level feedback:
- In
test_orphaned_trust_config_warns_only_once, the assertion relies on the exact warning message substring; consider keying off a more stable attribute (e.g., a specific warning class, log code, or constant message) to make the test less brittle to wording changes. - The async decorator test overrides
inspect.signatureglobally viamonkeypatch.setattr("policyforge.decorators.inspect.signature", ...); consider scoping this more narrowly (or restoring the original in afinally) to reduce the risk of surprising interactions if additional tests in this module start relying oninspect.signature.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- In `test_orphaned_trust_config_warns_only_once`, the assertion relies on the exact warning message substring; consider keying off a more stable attribute (e.g., a specific warning class, log code, or constant message) to make the test less brittle to wording changes.
- The async decorator test overrides `inspect.signature` globally via `monkeypatch.setattr("policyforge.decorators.inspect.signature", ...)`; consider scoping this more narrowly (or restoring the original in a `finally`) to reduce the risk of surprising interactions if additional tests in this module start relying on `inspect.signature`.
## Individual Comments
### Comment 1
<location path="tests/test_engine_trust_integration.py" line_range="227-230" />
<code_context>
assert decision.verdict == Verdict.DENY
assert decision.matched_rule == "tool_trust_unwired"
+ def test_orphaned_trust_config_warns_only_once(self, tmp_path, caplog):
+ import logging
+
+ policy_yaml = tmp_path / "p.yaml"
+ policy_yaml.write_text(
+ """
+tool_trust:
+ mode: enforce
+name: demo
+default_verdict: ALLOW
+""",
+ encoding="utf-8",
+ )
+
+ with caplog.at_level(logging.WARNING, logger="policyforge.engine"):
+ engine = PolicyEngine()
+ engine.load(policy_yaml)
+ engine.load(policy_yaml)
+
+ warnings = [
+ record for record in caplog.records if "no TrustManager was passed" in record.message
+ ]
+ assert len(warnings) == 1
+
def test_no_warning_when_trust_manager_present(self, tmp_path, ledger_path, caplog):
</code_context>
<issue_to_address>
**suggestion (testing):** Strengthen the "warns only once" behavior by exercising more than two load calls.
This test only covers two loads before asserting a single warning. To better lock in the "warn once" behavior, consider either looping over multiple `engine.load(policy_yaml)` calls within the same `caplog` context, or performing additional loads after the current assertion and checking that `len(warnings)` does not increase. This would more robustly guard against regressions where future changes re‑emit the warning on subsequent loads.
```suggestion
with caplog.at_level(logging.WARNING, logger="policyforge.engine"):
engine = PolicyEngine()
# Call load multiple times to ensure the warning is only ever emitted once
for _ in range(5):
engine.load(policy_yaml)
```
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
Comment on lines
+227
to
+230
| with caplog.at_level(logging.WARNING, logger="policyforge.engine"): | ||
| engine = PolicyEngine() | ||
| engine.load(policy_yaml) | ||
| engine.load(policy_yaml) |
There was a problem hiding this comment.
suggestion (testing): Strengthen the "warns only once" behavior by exercising more than two load calls.
This test only covers two loads before asserting a single warning. To better lock in the "warn once" behavior, consider either looping over multiple engine.load(policy_yaml) calls within the same caplog context, or performing additional loads after the current assertion and checking that len(warnings) does not increase. This would more robustly guard against regressions where future changes re‑emit the warning on subsequent loads.
Suggested change
| with caplog.at_level(logging.WARNING, logger="policyforge.engine"): | |
| engine = PolicyEngine() | |
| engine.load(policy_yaml) | |
| engine.load(policy_yaml) | |
| with caplog.at_level(logging.WARNING, logger="policyforge.engine"): | |
| engine = PolicyEngine() | |
| # Call load multiple times to ensure the warning is only ever emitted once | |
| for _ in range(5): | |
| engine.load(policy_yaml) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
How
Tests
pytest -q— 256 passedpytest -q --cov=policyforge --cov-branch— 95% branch-aware coverageruff check policyforge testsblack --check policyforge testsmypy policyforgegit diff --checkSummary by Sourcery
Expand test coverage for trust enforcement, policy loading, regex safety, and audit/trust compatibility to guard against regressions in policy hardening.
Tests:
Summary by CodeRabbit