Repository navigation
Let a finding carry A-3's optional subject (first step of #238) - #390
shubhamkataria2005 wants to merge 4 commits into
Conversation
A-3 is ratified four of four, so `subject` may now exist. This adds it to make_finding() and does nothing else: no check populates it here. A-3's own order is the field first and a consumer next, and #238's chaining is that consumer. As promised on #300, this lands on its own, ahead of the chaining work, so a change to analysis/findings.py is reviewed on its own terms rather than arriving inside a feature PR. WHAT A-3 ASKS FOR, AND HOW IT IS HELD optional make_finding(..., subject=None) is the default absent is absent no key at all, never "subject": None -- every existing finding keeps exactly its seven F-1 keys every key optional a check fills what it actually knows no new vocabulary a CLOSED key set: src, dst, srcPort, dstPort, protocol never replaces prose evidence.detail is untouched The closed key set is the decision worth reviewing. A free-form dict would let "dest" and "dst" both appear, and two findings about the same address would then fail to join, silently -- the failure A-3 was raised to remove from regex parsing, returning through a typo. Values are validated the way the other fields are, and fail loudly: an address or CIDR for src/dst, an integer 0-65535 for ports, a non-empty name for protocol (lower-cased, so "TCP" and "tcp" join). An empty subject is refused; A-3 says a check with nothing to put in it leaves it out. Mutation-tested: absent becoming subject=None, the key set opened, and protocol not normalised each fail the new tests. A guard test fails if any check starts populating subject before its consumer exists. 1865 passed, 13 skipped with Batfish running. ruff clean on touched files.
|
@ARSH871-bot — this is the |
ARSH871-bot
left a comment
There was a problem hiding this comment.
Governance first. A-3 is ratified on main, four of four, each row ticked by its own signer: you (#269, 125936d), Samika (#287, c327fa0), Ankeet (#299, addddd6), me (#300, 542d864). So subject existing is not a repeat of A-2, and your commit message is right about that.
What A-3 did not ratify is the key set. SUBJECT_KEYS (analysis/findings.py:133) is field vocabulary: five names, each with a value format. A-3's text (docs/finding-format.md:316-327) shows one example with dst, dstPort, protocol; src and srcPort appear nowhere in it, and your description says A-3 "never enumerates keys". CLAUDE.md §7a puts the field vocabulary in findings.py under all four, and findings.py:4-5 says the doc is the agreement and the module the enforcement. Here the module would enforce something the doc doesn't say. Samika signed #269 on "doesn't add ... new vocabulary", and on #300 you described this PR as only the optional keyword.
You asked me for a yes or no on the set. I think closed is the right design, for exactly your reason. But it isn't mine alone to agree. Please:
- Write the five keys and their value forms into A-3's section of
docs/finding-format.mdin this PR. - Ask Ankeet and Samika for a yes here, and record the four answers in the doc.
Code:
protocolis still open vocabulary (findings.py:164-167).subject={"dst":"10.20.0.5","protocol":"!tcp","dstPort":443}is accepted.!tcpispolicy_compliance's own syntax (analysis/checks/policy_compliance.py:219), and that is A-3's named first producer."6","any"and"banana"pass too. Your argument for closing the keys applies one level down:tcpand6never join, and!tcpmeans the opposite. Close the value set.- Addresses aren't canonical (
findings.py:154-159)."10.20.0.5"and"10.20.0.5/32"don't join; neither do"2001:DB8::1"and"2001:db8::1"."10.20.0.1/16"is accepted because ofstrict=False. You lower-case protocol so TCP and tcp join; addresses need the same: store a canonical form, or refuse non-canonical input. - 6 of 13 mutants survive. Nothing tests
srcorsrcPort: deleting"src"from the address branch, or"srcPort"from the port branch, stays green. Accepting 65536 or -1 stays green, because the only port case is 70000. Deleting the dict check stays green, because a string fails on the unknown-key check anyway; a list like["dst"]would separate them. Please addsrc/srcPortcases, 0 and 65535 accepted, -1 and 65536 refused, and anipaddressobject stored as a string. - The tripwire (
tests/test_finding_subject.py:75-82).Path("analysis")is relative to the cwd. With a realsubject=planted in a check, it fails from the repo root and passes from any other directory, becauserglobon a missing path yields nothing. It also missesfinding["subject"] = {...}(planted inaccess_control.py: still green) and anything inweb/orai/. Please anchor it onPath(__file__).resolve().parents[1]and assert the file list is non-empty. Better still, walk the AST for asubjectkeyword or a["subject"]assignment. - Description. The
test_analysis_cache.py:362assertion runs on hand-built dicts (:70-109) and never callsmake_finding. Withmake_findingmutated to always add"subject": Noneit stays 18/18 green. The property is held by yourtest_a_finding_without_subject_is_unchangedandtest_cve_mapping.py:106(4 failures under that mutant). Please correct the line so nobody leans on the wrong test later.
For #238, not this PR, but please put these on its checklist before the first producer lands:
ai/explain.py:671puts the whole finding,subjectincluded, into the prompt._names_an_invented_fact(:754-775) checks only summary, device, detail and source. A model repeating asubjectaddress gets rejected as inventing it, and the explanation falls back. That's the safe direction, but it's Ankeet's layer.- Saved scans keep only
_F1_FIELDS(web/main.py:1771,:1792), andtests/test_history_api.py:149pins that, sosubjectwill be silently dropped from history. make_findingacceptssubjectonnone/errorfindings. A chain must never run through a "could not check" card.
Nits: findings.py:172 still says "eight fields". findings.py:157-158 should raise ... from None.
What I ran:
- Your branch: 1864 passed, 1 failed, 13 skipped. The failure is a live-Batfish test that passes alone.
- Merged onto main dc93ccb: 1901 passed, 1 failed (same flake), 13 skipped; +17 is your 15 plus 2 hygiene cases for the new file.
- Without
subject, findings are byte-identical to main across 164 inputs, explicitNoneincluded. - Merges cleanly with #387 on top of main.
Requesting changes for 1–6. The "absent is absent" property is solid.
…key set to A-3 From @ARSH871-bot's review of the first version, each item reproduced first. GOVERNANCE. A-3 ratified an optional `subject`; it never listed the keys, and a closed key set is field vocabulary under all four (CLAUDE.md 7a). The five keys and their value forms are now written into A-3's section of docs/finding-format.md with their own sign-off table -- one of four, my row only. This PR should not merge until that table is full. No check populates `subject`, so changing the set before then costs nothing. CLOSED AT BOTH LEVELS. Closing the keys alone was half the argument. * protocol VALUES are now closed: tcp, udp, icmp, or 6, 17, 1, stored as the lower-case name. "!tcp" -- policy_compliance's own syntax for NOT tcp -- was accepted, and means the opposite. So were "6", "any", "banana". * addresses are canonical: stored as the network in Python's canonical form (10.20.0.5 -> 10.20.0.5/32, 2001:DB8::1 -> 2001:db8::1/128), so two spellings of one host join. Host bits below the prefix (10.20.0.1/16) are refused rather than silently masked. ipaddress objects store as strings. TESTS. Six of thirteen mutants survived the first version. Now killed: src and srcPort each dropped from their branch, 65536 and -1 accepted, the dict check deleted (a list like ["dst"] now separates it from the key check). Also killed: addresses stored as given, host bits masked, the protocol value set reopened. THE TRIPWIRE was relative to the cwd -- from any directory but the repo root it scanned nothing and passed. It now anchors on __file__, asserts it scanned more than twenty files, walks the AST of analysis/, web/ and ai/ for a `subject=` keyword or a `["subject"] =` assignment, and a second test plants both shapes to prove the detector is not blind. Nits: "eight fields" -> "nine"; the address error raises `from None`. 1929 passed, 13 skipped with Batfish running. ruff clean.
|
@ARSH871-bot — all six items are addressed in
Nits: "eight fields" → "nine", and
@patelankeet2 @SamikaPerera — this needs a yes or no from each of you on the key set. It's the table under "The key set — NOT covered by the signatures below" in A-3's section. Five keys ( |
Agreed to the key set after checking it against the actual code rather than the table alone: SUBJECT_KEYS, SUBJECT_PROTOCOLS and _canonical_address() in analysis/findings.py match the table exactly. Ran tests/test_finding_subject.py (42 passed) and independently mutation-tested three of the claimed guards rather than trusting the PR's own table: - removed strict=True from the address canonicalisation -> the host-bits-set test failed for the right reason - removed address canonicalisation entirely -> 6 tests failed for the right reason, including the two-spellings-join test - removed the bool guard on srcPort/dstPort -> failed for the right reason (bool is a subclass of int in Python, so True would otherwise be silently accepted as port 1) One non-blocking note for the PR comment: the equivalent bool guard on protocol is defensive redundancy, not load-bearing - no string form of True/False collides with SUBJECT_PROTOCOLS, so removing it does not fail any test. Full suite on f88e7ee: 1929 passed, 13 skipped, matching the PR's own number.
|
Yes from me on the key set, ticked in What I checked before signingRead Ran Mutation-tested three of the guards myself rather than only trusting the PR's own table: Full suite on One non-blocking noteThe equivalent bool guard on Agreed the key set is the right design, for the reason already written into the doc: closing one level (keys) without closing the next (protocol values, address spelling) would have left the exact join-failure A-3 exists to remove. |
patelankeet2
left a comment
There was a problem hiding this comment.
APPROVE on the code, as a fresh look since Arsh's CHANGES_REQUESTED is void under M-2 (two commits landed after it, including my own).
Went through all seven items from Shubham's "all six items are addressed in f88e7ee" table plus the nits, not just the key-set column I signed off on three days ago.
Verified today, independently
- Tripwire (item 6). Confirmed
REPO = Path(__file__).resolve().parents[1]is genuinely cwd-independent, not just correct-looking - rantest_no_product_code_populates_subject_yetfrom a different working directory (docs/) and it still scanned the right files. Then mutation-tested it against real product code, not just the synthetic planted file the second test uses: added a realf["subject"] = {...}assignment toanalysis/snapshot.py, confirmed the tripwire caught it by name and line number, reverted. - Protocol set reopened (item 5's third extra mutant). Hadn't checked this specific one three days ago - replaced the whole
SUBJECT_PROTOCOLSlookup withstr(value)and a never-true guard. 10 tests failed correctly, confirming the closed set is genuinely load-bearing everywhere, not just against the bool case I'd already tested. - Item 7 (wrong test credited) and the nits - confirmed present and correct: the PR body carries the "Corrected after review" note crediting the right test,
findings.py:199says "nine fields", and_canonical_address()'sraisecarriesfrom None. - Items 1-4 and the rest of item 5 (address canonicalisation, the port bool guard) were already independently verified and mutation-tested in my sign-off comment on the 18th - re-ran
tests/test_finding_subject.pyfresh today, still 42 passed.
Full suite on d01ca23: 1929 passed, 13 skipped - matches the number posted.
Not blocking this review, but blocking merge
The key-set sign-off table is 2 of 4 (Shubham, and my own tick from the 18th). Per the note in docs/finding-format.md, this PR should not merge until Arsh and Samika have answered - that gate is real and separate from whether the code itself is correct, which is what this review is approving.
First step of #238, not closing it. On #300 I said this change to
analysis/findings.pywould land on its own, ahead of the chaining work, so it could be reviewed on its own terms rather than arriving inside a feature PR. This is that.@ARSH871-bot —
analysis/findings.pyis yours, so this is yours to say no to.What it does
make_finding()accepts A-3's optionalsubjectand validates it. Nothing populates it. A-3's order is the field first and a consumer next; the chaining in #238 is the consumer.How each of A-3's conditions is held
subject=Nonedefault"subject": Nonesrc,dst,srcPort,dstPort,protocolevidence.detailThe decision I would most like checked
The closed key set. A-3 shows an example but never enumerates keys. A free-form dict would let
"dest"and"dst"both appear, and two findings about the same address would fail to join — silently. That is the exact failure A-3 was raised to remove from regex parsing, coming back through a typo. So unknown keys raise, and adding one should come with a consumer that needs it.Values fail loudly, like every other field: address/CIDR for
src/dst, integer 0–65535 for ports (aboolis refused), non-empty name forprotocol, lower-cased soTCPandtcpjoin. An emptysubjectis refused.Tests
tests/test_finding_subject.py, including a guard that fails if any check starts populatingsubjectbefore its consumer exists, so the field is never half-used.1865 passed, 13 skippedwith Batfish running.