Skip to content

prism.py: _strip_single_line_comments's str.splitlines() inserts phantom newlines on Form Feed/Vertical Tab (20 line_exclusive languages) #1954

Description

@squid-protocol

Summary

gitgalaxy/core/prism.py's _strip_single_line_comments (the comment stripper for the entire
"line_exclusive" lexical family -- 20 languages: agc_assembly, assembly, batch, csv, dockerfile,
embedded_python, jcl, m4, makefile, markdown, matlab, nix, pbtxt, perl, python, ruby, shell, tcl,
yaml, zig) splits its input with for line in text.splitlines(): (prism.py:1081) and rejoins
with "\n".join(code) / "\n".join(comments) (prism.py:1125).

Python's str.splitlines() splits on far more than \n/\r\n -- it also breaks on \v
(vertical tab), \f (form feed), \x1c-\x1e (file/group/record separator), \x85 (NEL),
/ (Unicode line/paragraph separator). Any of these characters appearing anywhere
in a line_exclusive language's source -- even inside what a human would read as one physical
line, e.g. inside a comment -- gets silently converted into a literal \n when the stream is
rejoined. Every downstream line-number computation that counts \n characters in the resulting
code/comment stream (e.g. detector.py's _slice_by_labels "FAST O(N) LINE TRACKER",
code.count("\n", last_counted_idx, start_idx)) then reports a line number that's too high by
one for every such character encountered so far in the file -- a monotonically growing drift that
never resets, since each phantom newline permanently shifts everything downstream of it.

Confirmed evidence

language-crucible/data/assembly/cosmopolitan/ape.S (real Cosmopolitan libc source, which uses
\f form-feed characters as a deliberate page-break idiom inside its comments/ASCII-art banners)
reproduces this exactly. Real Form Feed positions (grep -n $'\f' ape.S): lines 101, 232, 590,
875, 992, 1223, 1281, 1363, 1677 -- 9 total.

Real function_data.start_line drift observed against ctags' (correct) line numbers, growing in
lockstep with the cumulative Form Feed count up to that point:

function real/ctags line GitGalaxy's reported start_line drift FFs seen so far
ape_mz 118 119 +1 1 (line 101)
stub 216 217 +1 1
pc 251 253 +2 2 (line 232)
dsknfo 346 348 +2 2
ape_loader 709 712 +3 3 (line 590)
ape_idata_idt 1152 1157 +5 5 (lines 875, 992)
realmodeloader 1283 1290 +7 7 (lines 1223, 1281)
longmodeloader 1365 1373 +8 8 (line 1363)
kernel 1743 1752 +9 9 (line 1677)

Every drift value exactly equals the number of Form Feeds encountered before that point in the
file -- confirms the mechanism precisely, not just a correlated coincidence.

Directly confirmed the mechanism itself, not just the symptom: Prism.split_streams(text, "assembly") called on the raw file text preserves the real newline count perfectly in isolation
(text.count("\n") == result["code_stream"].count("\n"), 1888 == 1888) when a minimal repro with
no Form Feed is used -- the bug only manifests with a real line_exclusive-routed file containing
one of splitlines()'s extra boundary characters, which is exactly what happens for the real
ape.S through the full pipeline.

Suggested fix direction (not yet implemented)

Replace text.splitlines() with text.split("\n") in _strip_single_line_comments (prism.py
line 1081) so only real \n characters are treated as line boundaries -- Form Feed, vertical tab,
and the other splitlines()-special characters would then correctly stay inline as ordinary
non-newline content within whatever line they appear on, matching how a human (and every other
tool -- ctags, tree-sitter) reads the file. Worth double-checking \r\n-terminated files still
round-trip correctly afterward (a plain split("\n") leaves a trailing \r on each line for
CRLF input, which may need normalizing elsewhere in the pipeline -- or may already be normalized
upstream; not yet verified either way).

Scope

Confirmed live for assembly (this investigation). Not yet confirmed for the other 19
line_exclusive languages, but the bug is in code shared unconditionally by all of them (the
text.splitlines() call happens before any language-specific branching in the function) --
whether it manifests in practice for a given language depends only on whether real corpus source
for that language happens to contain a Form Feed/vertical tab/other splitlines()-special
character, which is corpus-dependent and not yet surveyed.

Found via the tri-comparison-ledger-sweep skill's investigation flow while validating the
assembly language's tri-comparison ledger (Claude direct analysis + a Gemini/agy read-only
dispatch that first identified the Form-Feed/splitlines() mechanism, independently verified by
re-reading prism.py's real source and re-grepping the real Form Feed positions before filing).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions