Test YAML runner writes UTF-8, mangles UserTalk 0xC7 marker; source of #866 false encoding-trap diagnosis #880

Rationale

tests/integration/runner.py:339 writes YAML script files with platform-default encoding (UTF-8), so a literal 0xC7 (UserTalk comment marker) in YAML source arrives as TWO bytes at runtime and breaks single-quoted character constants. This is the exact mangling that produced #866's false "reinstall destroys scripts" report. Fix: document \xc7 escape convention in docs/TESTING_GUIDE.md and consider making the runner write MacRoman or reject non-ASCII with clear error.

Current State

YAML encoding issue identified (2026-08-10):

• tests/integration/runner.py:339 writes YAML with platform-default encoding (UTF-8)
• UserTalk comment marker 0xC7 is a single byte in MacRoman
• UTF-8 encoding of 0xC7 produces TWO bytes (0xC3 0x87)
• Runtime receives mangled bytes → breaks character constants, scripts fail in ways that mimic install-time corruption
• This is the EXACT issue that produced langscan comment-strip scanner ignored single-quoted char constants: 0xC7 inside 'x' literals corrupts scripts cumulatively on install (was misattributed to newScriptObject size/reinstall) #866's false "newScriptObject destroys scripts" report

Consequence:

• YAML test scripts with 0xC7 produce corrupted runtime behavior
• Symptoms mimic install-path corruption (langscan comment-strip scanner ignored single-quoted char constants: 0xC7 inside 'x' literals corrupts scripts cumulatively on install (was misattributed to newScriptObject size/reinstall) #866 false alarm)
• Diagnosis becomes impossible without knowing the encoding trap
• Future test scripts will hit this silently

Desired Outcome

Prevent encoding mangling of UserTalk-specific bytes:

1. Document 0xC7 escape convention in docs/TESTING_GUIDE.md
2. Consider making runner write MacRoman (UserTalk native encoding)
3. OR reject non-ASCII in YAML with clear error message
4. Verify: test scripts with UserTalk comment markers work correctly

Recommendation

Option A (minimal, documentation):

• Document in docs/TESTING_GUIDE.md: "UserTalk comment marker 0xC7 must be escaped as \xc7 in YAML test scripts (UTF-8 mangling risk)"
• Add example test showing correct escape
• Reference langscan comment-strip scanner ignored single-quoted char constants: 0xC7 inside 'x' literals corrupts scripts cumulatively on install (was misattributed to newScriptObject size/reinstall) #866 as a case study

Option B (runner change):

• Modify runner to write YAML in MacRoman encoding (UserTalk native)
• Requires output encoding override in Python
• More robust but may affect other YAML tooling

Option C (validation):

• Reject non-ASCII bytes in YAML with clear error: "YAML script contains non-ASCII; use \xNN escapes for UserTalk-specific bytes"
• Prevents silent corruption

Recommended: A + C (document + validate)

Exit Criteria

• docs/TESTING_GUIDE.md documents \xc7 escape convention
• Example test shows correct escaping of UserTalk comment markers
• tests/integration/runner.py validates non-ASCII or writes MacRoman (no silent corruption)
• Verification: test script with 0xC7 produces correct behavior (not corruption)
• Reference langscan comment-strip scanner ignored single-quoted char constants: 0xC7 inside 'x' literals corrupts scripts cumulatively on install (was misattributed to newScriptObject size/reinstall) #866 as a case study of encoding-trap diagnosis difficulty

Related Context

• Found by: langscan comment-strip scanner ignored single-quoted char constants: 0xC7 inside 'x' literals corrupts scripts cumulatively on install (was misattributed to newScriptObject size/reinstall) #866 investigation, 2026-08-10
• Related issues: langscan comment-strip scanner ignored single-quoted char constants: 0xC7 inside 'x' literals corrupts scripts cumulatively on install (was misattributed to newScriptObject size/reinstall) #866 (false root-cause from encoding trap), verify_virgin_root_sync.sh covers only Virgin.root, not guest-DB mirrors; silent drift accumulates #869 (test infrastructure quality)
• Code: tests/integration/runner.py:339 (YAML writer), tests/integration/test_cases/*.yaml (scripts)
• Root cause: UTF-8 encoding of 0xC7 (UserTalk comment marker) produces 0xC3 0x87 (two bytes)
• Impact: langscan comment-strip scanner ignored single-quoted char constants: 0xC7 inside 'x' literals corrupts scripts cumulatively on install (was misattributed to newScriptObject size/reinstall) #866's false "newScriptObject destroys scripts" diagnosis was triggered by this trap
• Encoding: MacRoman (UserTalk native, single byte for 0xC7) vs UTF-8 (double byte)

Methodology Note

This is a classic encoding trap: single-byte MacRoman values when passed through UTF-8 encoding become multi-byte sequences, causing subtle corruption of binary data and character literals. The trap was invisible until #866's investigation required tracing the exact bits.