Repository navigation
Cast recordings corrupt multi-byte UTF-8 characters at 4096-byte chunk boundaries (mojibake with oh-my-posh / opencode TUI / bulk output) #5
Description
Summary
Start-InteractiveTranscript corrupts multi-byte UTF-8 characters in the recorded .cast file whenever more than 4096 bytes of output are pending in the PTY output pipe. The truncated UTF-8 sequences at each 4096-byte read boundary decode to U+FFFD (replacement character, EF BF BD) and are written permanently into the cast. The live console passthrough is unaffected (raw bytes are forwarded untouched), so everything looks fine during recording - the corruption only appears on playback (Show-InteractiveTranscript, asciinema play, agg renders).
This hits unicode-heavy workflows exactly like oh-my-posh prompts (powerline/nerd-font glyphs) and full-screen TUIs such as opencode, plus everyday commands that emit bulk unicode text.
Environment
- Bsn.Host 0.1.0 (PowerShell Gallery), Bsn.Common 0.1.0
- Windows 11 10.0.26300, Windows Terminal
- Recording host: PowerShell 7.6.9 (also reproduced with Windows PowerShell 5.1 as the caller)
- Verified with oh-my-posh 29.0.2 and opencode 2.2.5 sessions
Repro A - deterministic bulk write (interactive session)
Start-InteractiveTranscript # recorded nested shell
# inside the recorded session:
[Console]::Out.Write([string][char]0x2580 * 60000) # 180 KB, 3 bytes per char
exitResult on my machine: cast contains 50 output events; 20 of them are corrupted, 36 U+FFFD replacement characters in total. The corruption pattern is exactly the chunk boundary: one event ends with U+FFFD (dangling first byte of a split sequence) and the next event starts with U+FFFD (orphaned continuation bytes).
Repro B - everyday command (no synthetic writes)
Inside a recorded session, print a 291 KB UTF-8 file in a single write:
Get-Content .\big_utf8.txt -Raw # box-drawing, arrows, check marks, CJKResult: cast has 74 output events, 54 of them contain U+FFFD, 86 replacement characters total. Playback shows mojibake lines like ??┌─┬─? where every split glyph is destroyed.
Why TUI / prompt users hit it
- The ConPTY renderer flushes frames continuously, so paced output stays under the 4096-byte read buffer and survives. In my recordings an opencode full-screen paint at 120 columns was 3631 bytes - just under the limit. At typical wider terminal widths the same paint exceeds 4096 bytes and gets split mid-glyph.
- oh-my-posh prompts render fine when each burst is small, but any burst > 4096 bytes (long prompts with many segments, PSReadLine prediction-list redraws while typing/pasting fast, output scrolling) splits at byte 4096 with 2-in-3 probability per multi-byte char boundary.
Root cause
src/Bsn.Host/src/PtyRecorder.cs, PumpOutput() reads raw bytes into a 4096-byte buffer and hands each chunk to CastWriter.WriteEvent(), which decodes each chunk independently:
// PumpOutput
byte[] buffer = new byte[4096];
...
cast.WriteEvent(clock.Elapsed.TotalSeconds, false, chunk);
// CastWriter.WriteEvent
string text = Escape(Encoding.UTF8.GetString(chunk));Encoding.UTF8.GetString() on a chunk that ends mid-sequence replaces the truncated bytes with U+FFFD; the next chunk's leading continuation bytes produce another U+FFFD. Both the v2 and v3 writers share this code path, and input recording (-RecordInput) has the same problem for i events.
Suggested fix
Keep a persistent Decoder (or carry the trailing incomplete byte sequence) across reads in PumpOutput, and only hand complete sequences to CastWriter. The live proxy path already forwards raw bytes, so only the cast write path needs to change. A Pester test that records a > 4096-byte multi-byte burst (via ConvertTo-BsnAsciicast/Get-BsnCastTimeline fixtures) and asserts zero U+FFFD in the events would lock this in.
Artifacts from my runs (cast files with the corruption): C:\Users\sooro\AppData\Local\Temp\opencode\bsnhost-bugreport\ (h2_utf8.cast, I_getcontent_raw.cast). Happy to attach them if useful.
Activity
soroshsabz commentedon Nov 6, 2026
Related findings from the same investigation session (independently reproducible, separate root causes): #6 (cast writer buffering loses the header/tail on abnormal termination), #7 (-Command mangled by the cmd.exe wrapper), #8 (empty cast + success result when the caller has redirected stdio, e.g. running Start-InteractiveTranscript from the opencode TUI shell).
soroshsabz commentedon Nov 6, 2026
Evidence artifacts attached as a gist (secret link, accessible to anyone with the URL): https://gist.github.com/soroshsabz/c5fe45f2150347b335222fa10038b625
h2_utf8.cast- deterministic 180 KB multi-byte burst: 50 events, 20 corrupted, 36 U+FFFDI_getcontent_raw.cast- everydayGet-Content -Rawcommand: 74 events, 54 corrupted, 86 U+FFFDbig_utf8.txt/graw.ps1- the exact input usedverify.ps1- byte scanner, e.g.powershell -File .\verify.ps1 .\h2_utf8.castRepro tooling used for the whole investigation session (console input-injection helper + recording runners): https://gist.github.com/soroshsabz/8e74e9664da1e60241a1b73cf8443666
soroshsabz commentedon Nov 6, 2026
Reduce test duplication (SonarCloud quality gate: 17.9% -> under 3%)
Distribute issue tests into existing function-specific test files
Add real-world multi-byte stress tests: oh-my-posh (Windows) and vim …
Release Bsn.Host 0.2.1 (bug fixes for issues #5-#8) (#14)
Fix #5 (second root cause): bootstrap UTF-8 console codepage in recor…
soroshsabz commentedon Nov 6, 2026
Reopening with field evidence: 0.2.1 recordings of OpenCode still showed corruption. Root cause analysis found a SECOND mechanism, distinct from the chunk-boundary decoder bug fixed in #12:
Diagnosis (probes against the real ConPTY engine)
chcp.comoutput shows "Active code page: 437").cmd /c type- emit UTF-8 bytes, so one box-drawing character (U+2500, UTF-8E2 94 80) arrives as three CP-decoded characters (CP437Γ ö Ç) before the recorder sees anything.WriteConsoleW), which is why synthetic pwsh probes passed while real byte-writing TUIs corrupted. This matches the field symptom exactly: ASCII text fine, border/border-glyph lines mojibake.Note: the corruption pattern in the field report (valid mojibake letters, not U+FFFD) is the codepage signature; the chunk-boundary bug fixed earlier produces U+FFFD replacement instead. Both are now covered.
Fix (PR #16): the recorded command line bootstraps
[Console]::OutputEncoding/InputEncodingto UTF-8 (the programmaticchcp 65001) before the user's command or interactive shell runs - verified end to end with a byte-writing child and a real OpenCode session recording (zero mojibake, borders intact).New per-scenario system tests in
InteractiveTranscript.System.Tests.ps1:[Console]::OpenStandardOutput().Write) asserted character-for-character (60 border chars, 4 corners, zero U+FFFD, zero Latin-1-range mojibake)opencodeis not installed)Fix PR checks: cross-platform test, Sonar duplication, loader coverage
Fix #5 (playback): switch the console output encoding to UTF-8 in Sho…
Fix #5 (live view + playback): UTF-8 console output codepage manageme…