Summary
gitgalaxy/core/prism.py's _strip_single_line_comments (the comment stripper for the entire
"line_exclusive" lexical family -- 20 languages: agc_assembly, assembly, batch, csv, dockerfile,
embedded_python, jcl, m4, makefile, markdown, matlab, nix, pbtxt, perl, python, ruby, shell, tcl,
yaml, zig) splits its input with for line in text.splitlines(): (prism.py:1081) and rejoins
with "\n".join(code) / "\n".join(comments) (prism.py:1125).
Python's str.splitlines() splits on far more than \n/\r\n -- it also breaks on \v
(vertical tab), \f (form feed), \x1c-\x1e (file/group/record separator), \x85 (NEL),
/
(Unicode line/paragraph separator). Any of these characters appearing anywhere
in a line_exclusive language's source -- even inside what a human would read as one physical
line, e.g. inside a comment -- gets silently converted into a literal \n when the stream is
rejoined. Every downstream line-number computation that counts \n characters in the resulting
code/comment stream (e.g. detector.py's _slice_by_labels "FAST O(N) LINE TRACKER",
code.count("\n", last_counted_idx, start_idx)) then reports a line number that's too high by
one for every such character encountered so far in the file -- a monotonically growing drift that
never resets, since each phantom newline permanently shifts everything downstream of it.
Confirmed evidence
language-crucible/data/assembly/cosmopolitan/ape.S (real Cosmopolitan libc source, which uses
\f form-feed characters as a deliberate page-break idiom inside its comments/ASCII-art banners)
reproduces this exactly. Real Form Feed positions (grep -n $'\f' ape.S): lines 101, 232, 590,
875, 992, 1223, 1281, 1363, 1677 -- 9 total.
Real function_data.start_line drift observed against ctags' (correct) line numbers, growing in
lockstep with the cumulative Form Feed count up to that point:
| function |
real/ctags line |
GitGalaxy's reported start_line |
drift |
FFs seen so far |
| ape_mz |
118 |
119 |
+1 |
1 (line 101) |
| stub |
216 |
217 |
+1 |
1 |
| pc |
251 |
253 |
+2 |
2 (line 232) |
| dsknfo |
346 |
348 |
+2 |
2 |
| ape_loader |
709 |
712 |
+3 |
3 (line 590) |
| ape_idata_idt |
1152 |
1157 |
+5 |
5 (lines 875, 992) |
| realmodeloader |
1283 |
1290 |
+7 |
7 (lines 1223, 1281) |
| longmodeloader |
1365 |
1373 |
+8 |
8 (line 1363) |
| kernel |
1743 |
1752 |
+9 |
9 (line 1677) |
Every drift value exactly equals the number of Form Feeds encountered before that point in the
file -- confirms the mechanism precisely, not just a correlated coincidence.
Directly confirmed the mechanism itself, not just the symptom: Prism.split_streams(text, "assembly") called on the raw file text preserves the real newline count perfectly in isolation
(text.count("\n") == result["code_stream"].count("\n"), 1888 == 1888) when a minimal repro with
no Form Feed is used -- the bug only manifests with a real line_exclusive-routed file containing
one of splitlines()'s extra boundary characters, which is exactly what happens for the real
ape.S through the full pipeline.
Suggested fix direction (not yet implemented)
Replace text.splitlines() with text.split("\n") in _strip_single_line_comments (prism.py
line 1081) so only real \n characters are treated as line boundaries -- Form Feed, vertical tab,
and the other splitlines()-special characters would then correctly stay inline as ordinary
non-newline content within whatever line they appear on, matching how a human (and every other
tool -- ctags, tree-sitter) reads the file. Worth double-checking \r\n-terminated files still
round-trip correctly afterward (a plain split("\n") leaves a trailing \r on each line for
CRLF input, which may need normalizing elsewhere in the pipeline -- or may already be normalized
upstream; not yet verified either way).
Scope
Confirmed live for assembly (this investigation). Not yet confirmed for the other 19
line_exclusive languages, but the bug is in code shared unconditionally by all of them (the
text.splitlines() call happens before any language-specific branching in the function) --
whether it manifests in practice for a given language depends only on whether real corpus source
for that language happens to contain a Form Feed/vertical tab/other splitlines()-special
character, which is corpus-dependent and not yet surveyed.
Found via the tri-comparison-ledger-sweep skill's investigation flow while validating the
assembly language's tri-comparison ledger (Claude direct analysis + a Gemini/agy read-only
dispatch that first identified the Form-Feed/splitlines() mechanism, independently verified by
re-reading prism.py's real source and re-grepping the real Form Feed positions before filing).
Summary
gitgalaxy/core/prism.py's_strip_single_line_comments(the comment stripper for the entire"line_exclusive"lexical family -- 20 languages: agc_assembly, assembly, batch, csv, dockerfile,embedded_python, jcl, m4, makefile, markdown, matlab, nix, pbtxt, perl, python, ruby, shell, tcl,
yaml, zig) splits its input with
for line in text.splitlines():(prism.py:1081) and rejoinswith
"\n".join(code)/"\n".join(comments)(prism.py:1125).Python's
str.splitlines()splits on far more than\n/\r\n-- it also breaks on\v(vertical tab),
\f(form feed),\x1c-\x1e(file/group/record separator),\x85(NEL),/(Unicode line/paragraph separator). Any of these characters appearing anywherein a
line_exclusivelanguage's source -- even inside what a human would read as one physicalline, e.g. inside a comment -- gets silently converted into a literal
\nwhen the stream isrejoined. Every downstream line-number computation that counts
\ncharacters in the resultingcode/commentstream (e.g.detector.py's_slice_by_labels"FAST O(N) LINE TRACKER",code.count("\n", last_counted_idx, start_idx)) then reports a line number that's too high byone for every such character encountered so far in the file -- a monotonically growing drift that
never resets, since each phantom newline permanently shifts everything downstream of it.
Confirmed evidence
language-crucible/data/assembly/cosmopolitan/ape.S(real Cosmopolitan libc source, which uses\fform-feed characters as a deliberate page-break idiom inside its comments/ASCII-art banners)reproduces this exactly. Real Form Feed positions (
grep -n $'\f' ape.S): lines 101, 232, 590,875, 992, 1223, 1281, 1363, 1677 -- 9 total.
Real
function_data.start_linedrift observed against ctags' (correct) line numbers, growing inlockstep with the cumulative Form Feed count up to that point:
Every drift value exactly equals the number of Form Feeds encountered before that point in the
file -- confirms the mechanism precisely, not just a correlated coincidence.
Directly confirmed the mechanism itself, not just the symptom:
Prism.split_streams(text, "assembly")called on the raw file text preserves the real newline count perfectly in isolation(
text.count("\n") == result["code_stream"].count("\n"), 1888 == 1888) when a minimal repro withno Form Feed is used -- the bug only manifests with a real
line_exclusive-routed file containingone of
splitlines()'s extra boundary characters, which is exactly what happens for the realape.Sthrough the full pipeline.Suggested fix direction (not yet implemented)
Replace
text.splitlines()withtext.split("\n")in_strip_single_line_comments(prism.pyline 1081) so only real
\ncharacters are treated as line boundaries -- Form Feed, vertical tab,and the other
splitlines()-special characters would then correctly stay inline as ordinarynon-newline content within whatever line they appear on, matching how a human (and every other
tool -- ctags, tree-sitter) reads the file. Worth double-checking
\r\n-terminated files stillround-trip correctly afterward (a plain
split("\n")leaves a trailing\ron each line forCRLF input, which may need normalizing elsewhere in the pipeline -- or may already be normalized
upstream; not yet verified either way).
Scope
Confirmed live for
assembly(this investigation). Not yet confirmed for the other 19line_exclusivelanguages, but the bug is in code shared unconditionally by all of them (thetext.splitlines()call happens before any language-specific branching in the function) --whether it manifests in practice for a given language depends only on whether real corpus source
for that language happens to contain a Form Feed/vertical tab/other
splitlines()-specialcharacter, which is corpus-dependent and not yet surveyed.
Found via the
tri-comparison-ledger-sweepskill's investigation flow while validating theassemblylanguage's tri-comparison ledger (Claude direct analysis + a Gemini/agy read-onlydispatch that first identified the Form-Feed/
splitlines()mechanism, independently verified byre-reading
prism.py's real source and re-grepping the real Form Feed positions before filing).