Denylist scan: read binary and oddly encoded content as git sees it by mishrasanjeev · Pull Request #1873 · mishrasanjeev/agentic-org

mishrasanjeev · 2026-10-30T07:31:52Z

Why

The vendor denylist scan in #1870 learned to skip binary files in the line diff and check their readable text instead. An independent review then found ways a denylisted term could still get through:

Line splitting: lines were split with splitlines(), which also breaks at \r, \v, \f, \x1c–\x1e, U+0085, U+2028 and U+2029. This was the real cause of the "unparseable diff hunk header" crash on the video.
Files that weren't read at all:
- a directory that replaced a binary file of the same name;
- a binary at the root whose name starts with :;
- files reclassified by a .gitattributes rule or a NUL byte.
Binary content: it only got a weak check.

A second review of the first fix found seven more gaps: empty binary-marked files, UTF-16 split inside characters, text with one stray byte, short terms in metadata, commit-message encodings, untested fail-closed paths, and UTF-16 text with no NUL bytes.

A third review found two more:

A short term as a value in markup (XMP, a PDF dictionary, JSON) was missed, though the scanner in Docs: a plain-words start page and a real OACP order on video #1870 found it.
Arrays of numbers below 65,536 were read as UTF-32 text, where letters among them could match a term by chance.

A fourth review found that the fix for number arrays also dropped text in other scripts without a byte-order mark (Russian, Greek, Japanese), and that CONTRIBUTING.md had lost its CRLF line ends. A fifth review found two problems with that round's script rule:

it still dropped katakana-heavy Japanese text and Chinese text with brackets on long lines;
with its whitespace rule, it let in arrays of numbers in one narrow range and arrays of token ids.

A sixth review found that the next rule (separators and folding letters) dropped three kinds of text:

Japanese with fullwidth digits;
Chinese with half-width punctuation;
text after a long English preamble.

So the accept-or-reject test is gone: such a guess is now always decoded and read safely, as below.

What changes

scripts/check_denylist.py:

Lines: split on \n only, as git does. This applies to the diff, commit messages, --text-file inputs and the audit. Inside a hunk, diff --git and @@ can no longer pose as headers. The patch is parsed as bytes.
Which files are read:
- One git diff --raw --numstat -z --no-renames --no-abbrev gives each path's status, blob ids and binary flag. It runs with --attr-source=<base>, --no-ext-diff, --no-textconv and --no-relative.
- New blobs are read by object id through a single git cat-file --batch. No path is ever parsed as a pathspec, and no command line grows with the number of files.
- A "Binary files" line in the patch must name a blob that is checked whole, or a deletion. Otherwise the scan fails with exit 2.
- Empty files and mode-only changes need no such line, and are fine.
How a blob is read:
- Wide text first: UTF-16 and UTF-32 is decoded and checked line by line. It is found by its byte-order mark, or by where the NULs fall.
- Without a byte-order mark: text that is at least half ASCII is checked as text. Any other text is checked in stretches of ASCII text. A term counts only when written in ASCII letters, as a name in Russian, Japanese or Chinese text usually is.
- What a stretch crosses: spaces, punctuation and combining marks, invisible formatting characters (soft hyphen, zero-width space), and the dash symbols that Japanese encodings decode to. Every other character ends it.
- Number arrays: they decode to characters all over Unicode, with few ASCII letters, so scattered letters can no longer join into a term.
- Measured: with 2,000 invented four-letter terms, arrays of 65,536 numbers gave up to 53 chance matches read as plain text, and none read by stretches. The arrays were uniform, in one range of accented or fullwidth letters, or token ids.
- Mostly text: a blob that is text once NULs are removed, with up to 2% odd bytes, is checked line by line at every term length.
- Anything else: its readable runs are checked. The runs cover UTF-8 and UTF-16/32 in either byte order. A single NUL or control byte between two words counts as a space.
- Terms under 5 characters inside runs: they count only as a whole word with other words beside it:
  - one other word of 3 or more letters when whitespace is next to it ("Encoded with Xyzw Studio");
  - two when only punctuation bounds it, as for a value in XMP, a PDF dictionary or JSON.
- Chance matches: 1 GB of random bytes gave none. The rule admits about one qualifying four-letter word per GB, which means an estimated one chance match in 700 GB for the committed list.
Commit messages: read through git log -z --encoding=UTF-8, as stored, and decoded with the codec named in the commit's encoding header. The records must match rev-list.
Audit: every tracked file goes through the same blob check, including files with NULs and wide text.
Output and exit codes unchanged: only locations are printed, never the matched words. Exit 2 on any git failure, 1 on a match, 0 otherwise.

CONTRIBUTING.md says how binary files are read, what isn't matched inside them, and that the scan needs git 2.40 or later.

Checks

tests/unit/test_check_denylist.py: 181 passed.
- Text that must be found: Russian, Japanese and Chinese without a byte-order mark, including:
  - a katakana-heavy guide;
  - a table with fullwidth digits;
  - notes with half-width punctuation;
  - text after a long English preamble;
  - a five-letter term followed by kana, and a two-word term joined by "・";
  - terms with a soft hyphen, a zero-width space or a decoded Japanese dash inside.
- Arrays that must give no match: narrow ranges, fullwidth forms and token ids. Each test shows that the plain reading matches invented terms while the scan finds nothing.
- Against earlier rounds: each new case fails on the round that missed it.
Speed: checking the 9.4 MB video takes 2.5 s, against 2.3 s on the second-round scanner.
CONTRIBUTING.md keeps its CRLF line ends: against main its diff is the 14 added lines.
- Against the scanner on main (1bca6df), 60 of the new cases fail, including the regression for the @@ crash.
- Every bypass from both reviews has a test.
check_denylist.py scan --base origin/main --head HEAD: exit 0. audit: exit 0, with house-term warnings only, the same as before.
False positives: with the committed list, there were no findings in 1 GB of random bytes or in the 48 binary blobs in the repository's history (29.5 MB).
Speed: a scan over the range with the 9.4 MB video takes 3.5 s, against about 2.4–3.2 s before. The third round adds about 6% to the check of that video.
ruff check: clean.

Known limits, documented in the code:

A UTF-16 file with no byte-order mark and no ASCII characters at all can't be identified.
Inside UTF-16 or UTF-32 text that has no byte-order mark and is not mostly ASCII, a name written in accented or fullwidth letters isn't found.
Inside binary runs, a term under 5 characters isn't matched in these cases:
- A value on its own: no other word beside it, such as "Xyzw" or (Xyzw).
- One word beside it, no whitespace: only punctuation bounds it, and the run has just one other word.
- The scanner in Docs: a plain-words start page and a real OACP order on video #1870 matched these, along with far more chance matches in compressed media (about one per 100 MB of random bytes).

The scan could be bypassed, or crashed, by content that is not plain UTF-8 text: - The diff was split with str.splitlines(), which also breaks at \r, \f, \v, \x1c-\x1e, U+0085, U+2028 and U+2029. Text after one of those lost its "+" and went unchecked, a "diff --git" after one reset the parser, and "@@" after one was refused as a hunk header (the crash a video triggered). The diff, commit messages, extra text files and the audit now split on "\n" only, as git does. - Binary files were left out of the diff with exclude pathspecs and found again with ls-tree by path. A directory replacing a binary file of the same name hid every file under it, a name starting with ":" was read as pathspec magic and skipped, and each path went on the command line twice, so a change with many media files could not be scanned. Paths, status and new blob ids now come from one "git diff --raw --numstat -z", blobs are read by id through a single "git cat-file --batch", only deleted files are skipped, and a blob that cannot be read fails the check. - Binary-ness came from the working tree's attributes, so a change could add a .gitattributes rule moving its own text files out of the line diff. Both diffs now run with --attr-source set to the base commit (git 2.40+), and the number of "Binary files" lines must match the numstat count. - A file git calls binary was only searched for ASCII runs of six or more printable characters, which dropped tab- or NBSP-separated and accented terms, UTF-16 text and short metadata values. Such a file is now checked line by line when it is text with NUL bytes, UTF-16 or UTF-32, and otherwise its UTF-8 (with multibyte characters), UTF-16 and UTF-32 runs of five or more characters are checked. In those runs only terms of five or more characters count: in 3 GB of random bytes the committed list's four-character terms matched by chance about once per 100 MB, and longer terms never did. - The audit skipped every file holding a NUL byte; it now reads each file the same way. - Commit messages are read from the commit objects, so a \x1e in a message no longer splits the log record and hides the rest; a line that is not UTF-8 is also read without NULs and as Windows-1252. Findings are still reported by location without the matched words, with the same exit codes, and house-term warnings are unchanged.

--numstat counts a binary file whose mode alone changed, but the patch prints no "Binary files ... differ" line for it, so the consistency check between the two failed such a change with exit 2. Only binary files whose blob changed are now counted against the patch, and a mode change reads nothing, since the content is what the base already had. The test for a name starting with ":" now uses that name alone: a second name with exclude magic made the old ls-tree call fail outright, which hid whether the bypass itself was caught.

A review of the binary-scan rework found these gaps: - An empty file that the base .gitattributes marks binary (this repository's own "*.pdf binary"), added or deleted, failed the scan with exit 2: --numstat counts it as binary, but the patch has no "Binary files" line for it. The patch is now read with --full-index, and the new object of each "Binary files" line must be a blob that is read whole, or a deletion. That is the direction that matters; an empty file or a new mode needs no such line. - UTF-16 and UTF-32 text was read by dropping its NULs whenever that gave UTF-8, so it was split at every 0x0A byte, including the one inside U+200A (0A 20), and a term with a hair space between its words passed. Such text is now decoded first and split on "\n" characters. A UTF-16 file with a byte-order mark and no NUL byte (Latin Extended letters and a thin space) is diffed by git as text, so every changed text blob is now read as well and decoded if it is UTF-16 or UTF-32, in the scan and in the audit. - A text file with a NUL and one invalid byte fell to the readable-run scan, where a NUL or control byte between words, and every term under five characters, went unmatched. Text is now judged by its share of odd bytes (control bytes and bytes that are not UTF-8: at most 2%, or 8), not by strict UTF-8, and a line that is not UTF-8 gets its Windows-1252 reading also when it holds NULs. In binary data a single NUL or control byte between two words is read as a space, and a term under five characters counts as a word of prose: whole, next to whitespace and beside another word. 1 GB of random bytes and the 29.5 MB of binary blobs in the history gave no chance match with the committed list; the expected rate for its four-character terms is about one in 750 GB. - Commit messages were read from the commit objects only, so a message stored in Shift_JIS or ISO-2022-JP under an encoding header passed. They are now also read through "git log -z --encoding=UTF-8", whose records must match rev-list, and decoded with the declared codec in Python, for an encoding the git build cannot convert. - Tests now pin the fail-closed paths: a missing object or one of another kind, a binary file the patch leaves out that is not read whole, and a commit that git log does not show.

chatgpt-codex-connector · 2026-10-30T07:32:05Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-30T07:34:15.952070Z	`976160f`	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 976160fe5f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-10-30T07:34:13Z

 are not a secret: with the committed salt anyone can test a guessed name. Failures give the location, not the matched words. The check fails
 closed when git or the hash file misbehaves.

+Every file a change adds or changes is also read whole. UTF-16 or UTF-32 text


Remove tool attribution from the commit identity

Commit def543046dfbbbc088fe6ebf111bb9668a2cec73 records both its author and committer as Codex <codex@openai.com>. The repository explicitly prohibits tool attribution in author and committer fields, so this commit violates the repository's publication rules regardless of the scanner's correctness. Use an approved non-tool commit identity before merging.

AGENTS.md reference: AGENTS.md:L15-L21

Useful? React with 👍 / 👎.

…r arrays posing as UTF-32 A term under five characters inside a binary file's readable text counted only next to whitespace, so a value in XMP, a PDF dictionary or JSON metadata was missed even with other words around it. It now also counts when punctuation alone bounds it and the run holds two other words of three or more letters. 1 GB of random bytes gave no chance match of the committed list, and the rule admits one more four-letter word per GB than before (about one chance match in 700 GB). Arrays of 16- or 32-bit numbers below 65,536 have the NUL bytes of UTF-16 or UTF-32 text, and were decoded and checked as text, where accented letters among them could spell a term by chance. A guess without a byte-order mark now counts only when at least half the decoded characters are ASCII text.

…; keep CONTRIBUTING's line ends The previous change accepted a UTF-16 or UTF-32 guess without a byte-order mark only when half its characters were ASCII, which dropped Russian, Greek or Japanese text: a short term in it was no longer found. A guess now counts when it reads as text in some script: half its characters ASCII text, or one in a hundred whitespace, or all but one in twenty of the others from at most four scripts. Arrays of 16- or 32-bit numbers still fail all three: in random arrays from ranges between 0 and 65,535, under 0.2% of the characters were whitespace and the four commonest scripts held at most 90% of the others. The number-array test now plants the term's letters side by side, so the scanner before the change finds them, and new tests cover Russian, Japanese and Chinese text without a byte-order mark. CONTRIBUTING.md is back to its CRLF line ends, so its diff is only the edited paragraph.

…ators and folding letters The script count of the previous change split Japanese and Chinese punctuation and the long-vowel mark into scripts of their own, so katakana-heavy text and text with brackets on long lines were still dropped. Its whitespace and script rules also let in arrays of numbers in one narrow range, and arrays of token ids whose line ends decode to a lone CR, where accented letters folded to ASCII gave chance matches. The measurements the previous commit gave for arrays held only for wide ranges. A guess without a byte-order mark now counts when half its characters are ASCII text, or else when one in a hundred is a separator (a space, a line end, or the punctuation of CJK, Hindi and Arabic text; a lone CR is not one) and under one in ten of its other characters fold to an ASCII letter or digit. Prose in eight scripts had 5% to 23% separators and no folding letters; of uniform arrays from 16 ranges only values inside the CJK punctuation block passed, and those fold to no letter; arrays of BPE token ids have 18% to 24% folding letters.

…hes of ASCII text Each earlier rule for deciding whether a UTF-16 or UTF-32 guess without a byte-order mark was text either dropped real text (Japanese with fullwidth digits, Chinese with half-width punctuation, text after a long English preamble) or let number arrays in. The chance matches in those arrays come from letters scattered among unrelated characters being joined into a term. A guess is now always decoded. When it is at least half ASCII text it is checked as text, as before. Otherwise each line is checked in stretches of ASCII text, cut at every other character but a space, punctuation or a combining mark: a term counts only when written in ASCII letters, as a name in Russian, Japanese or Chinese text is. With 2,000 invented four-letter terms, arrays of 65,536 numbers (uniform, in one range of accented or fullwidth letters, or token ids) gave up to 53 chance matches read as text and none read by stretches. In such text, a name written in accented or fullwidth letters is not found. The array tests now draw their values from one generator each, and show that the plain reading matches invented terms where the scan does not.

…cters and dash symbols A name in wide text without a byte-order mark was missed when a soft hyphen, a zero-width space, or a dash that a Japanese encoding decodes to a symbol (U+2212, U+FF5E, U+FF0B, or the long-vowel mark typed as a dash) sat between its words. Those characters no longer end a stretch of ASCII text. With the same 2,000 invented terms, the number arrays still give no chance match.

…t each separator between a term's words The halfwidth katakana long-vowel mark (U+FF70) now joins a stretch of ASCII text, like its fullwidth twin. A test puts each invisible character and dash symbol between the two words of a term, so removing any one of them fails it. The comment now says where each symbol comes from: the fullwidth tilde is what cp932 decodes the wave dash to, and U+FF0B is a plus sign.

mishrasanjeev added 3 commits October 30, 2026 11:04

chatgpt-codex-connector Bot reviewed Oct 30, 2026

View reviewed changes

mishrasanjeev added 6 commits October 30, 2026 15:06

mishrasanjeev merged commit 9289745 into main Oct 31, 2026
28 checks passed

Navigation Menu

Repository navigation

Denylist scan: read binary and oddly encoded content as git sees it - #1873

Denylist scan: read binary and oddly encoded content as git sees it#1873
mishrasanjeev merged 9 commits into
mainfrom
fix/denylist-binary-scan

mishrasanjeev commented Oct 30, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot commented Oct 30, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Oct 30, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

mishrasanjeev commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What changes

Checks

Uh oh!

chatgpt-codex-connector Bot commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

mishrasanjeev commented Oct 30, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Oct 30, 2026 •

edited

Loading