Parser/scanner divergence: CR-spanning single-quote character constants; fail-open, low practical impact #883

Rationale

Residual strip-scanner divergence for CR-spanning single-quote character constants. The real tokenizer (parsepopcharconst) accepts a raw CR inside 'x' constants (legal 1-char or 4-char OSType consts spanning lines); the strip scanner is per-line (flinstring resets each line), so on the continuation line the closing quote OPENS a phantom literal — literal-state inversion. Near-zero practical occurrence (authors write '\r'), not a smuggling vector, fail-open only for round-trip fidelity. Also two fail-closed divergences: stripper accepts empty '' and >4-char runs the tokenizer rejects.

Current State

Strip-scanner divergence identified (2026-08-10):

Fail-open case (CR-spanning character constants):

• Real tokenizer (parsepopcharconst): accepts raw CR inside 'x' constants (e.g., 'FOO\r' or multi-line constants)
• Strip scanner: per-line processing (flinstring resets each line)
• On continuation line: closing quote OPENS a phantom literal
• Result: literal-state inversion, different parse state
• Practical occurrence: near-zero (authors write '\r' escape, not raw CR)
• Not a smuggling vector; fail-open only for round-trip fidelity

Fail-closed cases:

1. Stripper accepts empty '' — tokenizer rejects
2. Stripper accepts >4-char runs — tokenizer rejects

Impact:

• Round-trip fidelity gap: exported .ut files may not parse identically
• Parser differential creates potential for silent divergence
• Low practical risk (rare for code to use raw CR in constants)
• Same class as langscan comment-strip scanner ignored single-quoted char constants: 0xC7 inside 'x' literals corrupts scripts cumulatively on install (was misattributed to newScriptObject size/reinstall) #866 (parser/scanner misalignment)

Desired Outcome

Fix options:

1. Document limitation: Note that strip scanner has per-line scope; CR-spanning constants may diverge
2. Fix scanner: Carry unterminated-single-quote state across lines in scanner
3. Both: Document + fix for full fidelity

Recommendation

Option A (minimal, documentation):

• Add comment at delimiter-parsing code: "Note: strip scanner per-line; CR-spanning 'x' constants may diverge"
• Reference langscan comment-strip scanner ignored single-quoted char constants: 0xC7 inside 'x' literals corrupts scripts cumulatively on install (was misattributed to newScriptObject size/reinstall) #866 (same class of parser/scanner divergence)
• Update ODB_SCRIPT_EDITING.md: document the limitation

Option B (fix):

• Modify flinstring to carry quote state across lines
• Handle CR-spanning character constants correctly
• Stripper matches tokenizer behavior (fail-closed for empty '' and >4-char)

Recommended: Option A (documentation) + flag for future Option B (if round-trip fidelity becomes critical)

Exit Criteria

• Limitation documented in code comment and ODB_SCRIPT_EDITING.md
• Reference to langscan comment-strip scanner ignored single-quoted char constants: 0xC7 inside 'x' literals corrupts scripts cumulatively on install (was misattributed to newScriptObject size/reinstall) #866 (same class of divergence)
• Clear guidance: CR-spanning constants (rare in practice) may diverge
• Decision: pursue Option B fix if round-trip fidelity becomes requirement

Related Context

• Found by: langscan comment-strip scanner ignored single-quoted char constants: 0xC7 inside 'x' literals corrupts scripts cumulatively on install (was misattributed to newScriptObject size/reinstall) #866 security gate, 2026-08-10
• Related issues: langscan comment-strip scanner ignored single-quoted char constants: 0xC7 inside 'x' literals corrupts scripts cumulatively on install (was misattributed to newScriptObject size/reinstall) #866 (parser/scanner divergences), verify_virgin_root_sync.sh covers only Virgin.root, not guest-DB mirrors; silent drift accumulates #869 (ODB<->.ut fidelity)
• Code: Strip scanner (flinstring, per-line), real tokenizer (parsepopcharconst)
• Practical occurrence: near-zero (authors write '\r' escape)
• Smuggling risk: none (fail-open only)
• Round-trip impact: only for exotic CR-spanning constants

Fail Cases Summary

All three are edge cases; practical impact is minimal. The CR-spanning case is the most exotic (authors rarely use raw CR inside constants).