From cbc19e93e20c2887ad58baae3f096fa08cf31ad0 Mon Sep 17 00:00:00 2001 From: Seyyed Soroosh Hosseinalipour Date: Sat, 7 Nov 2026 23:34:56 +0330 Subject: [PATCH 1/3] Fix #5 (live view): switch the console to UTF-8 while recording Field regression from v0.2.2: the CAST is clean (every capture test passes), but the LIVE view - what the user watches while recording - still turns box-drawing text into CP437 mojibake on OEM-codepage consoles (cmd, Windows PowerShell 5.1 hosts, fresh hidden consoles). Reported with OpenCode after installing 0.2.2 from the gallery. Root cause: PtyRecorder proxies the ConPTY stream to the real console as raw UTF-8 bytes (WriteFile on the console handle). The console decodes those bytes with its own OUTPUT codepage; at the OEM codepage the UTF-8 box borders (E2 94 80 ...) render as 'Gamma-ohmega-acute'. The recorder never re-encodes, so no cast-side fix can help: the outer console is the decoder. Fix: the engine saves the console output codepage, switches it to UTF-8 (65001) for the duration of the recording - the proxied stream is UTF-8, matching the cast - and restores the original value in finally, alongside the existing console-input-mode restore. Consoles already at 65001 (pwsh 7 hosts) are untouched. Test: new Windows system test reproduces the exact topology end to end. conlog wraps a runner whose console sits at the OEM codepage while a byte-writing child records; the conlog capture IS the live view the user would see. Verified locally against a pre-fix build (test red: corner count 0, mojibake soup) and the fixed build (green: glyphs byte-for-byte, console codepage restored to 437). Runs on CI whenever the console-logger build step succeeds; skips gracefully otherwise. Also adds the redirected-stdio skip guard the existing console-logger cross-check was missing (conlog cannot run when its own stdin is redirected). Bsn.Host 0.2.3 (C# engine + docs + version stamp); Bsn.Common and Bsn.SystemManagement are unchanged. --- AGENTS.md | 5 +- CHANGELOG.md | 26 ++++ docs/en-US/Start-InteractiveTranscript.md | 9 ++ src/Bsn.Host/Bsn.Host.psd1 | 4 +- .../Public/Start-InteractiveTranscript.ps1 | 5 + src/Bsn.Host/en-US/about_Bsn.Host.help.txt | 6 + src/Bsn.Host/src/PtyRecorder.cs | 23 ++++ .../InteractiveTranscript.System.Tests.ps1 | 118 ++++++++++++++++++ 8 files changed, 193 insertions(+), 3 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 91aa8ba..8654d7f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -152,7 +152,10 @@ before: `src` directory when the Bsn.Common sibling exists) BEFORE dot-sourcing function files - keep that self-heal in place. - The Bsn.Host ConPTY engine (`src/Bsn.Host/src/PtyRecorder.cs`) must - restore the console input mode in `finally`, close the PTY-side pipe + restore the console input mode AND the console output codepage in + `finally` (the recording temporarily switches the real console to + UTF-8 so the live proxy view decodes correctly on OEM-codepage + consoles), close the PTY-side pipe handles after CreatePseudoConsole (EchoCon pattern), and emit cast files as UTF-8 without BOM with LF line endings. Cast files support asciicast v2 (absolute event times) and v3 (intervals since the previous event, diff --git a/CHANGELOG.md b/CHANGELOG.md index c22f417..2f3544a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,32 @@ All notable changes to the modules in this repository are documented here. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and the modules adhere to [Semantic Versioning](https://semver.org/). +## [Bsn.Host 0.2.3] - 2026-11-07 + +### Fixed + +- The LIVE recording view (what you watch while the recording runs) no + longer turns box-drawing text into CP437 mojibake on OEM-codepage + consoles (cmd, Windows PowerShell 5.1 hosts, fresh hidden consoles). + The cast file was always correct - every 0.2.2 capture test passed - + but the ConPTY proxy writes the stream to the real console as raw + UTF-8 bytes, and the console decodes those bytes with its own output + codepage (OEM 437), rendering `E2 94 80` as `Gamma-ohmega-acute`. + The engine now switches the console output codepage to UTF-8 for the + duration of the recording and restores the original value afterwards + (same restore discipline as the console input mode). Reportedly hit + with OpenCode after installing 0.2.2 from the gallery. + [#5](https://github.com/BSVN/Lmt.Commons/issues/5) + +### Added + +- A live-view system test reproducing the exact topology: conlog wraps a + runner whose console sits at the OEM codepage while a byte-writing + child records - the captured live view must render the box characters + themselves with zero CP437 mojibake, and the console codepage must be + restored after the recording. + [#5](https://github.com/BSVN/Lmt.Commons/issues/5) + ## [Bsn.Host 0.2.2] - 2026-11-07 ### Fixed diff --git a/docs/en-US/Start-InteractiveTranscript.md b/docs/en-US/Start-InteractiveTranscript.md index 33207ae..a5b3ec7 100644 --- a/docs/en-US/Start-InteractiveTranscript.md +++ b/docs/en-US/Start-InteractiveTranscript.md @@ -78,6 +78,15 @@ events. `-RecordInput` records keystrokes as `i` events (off by default - it can capture secrets; the fact is noted in the cast header; Windows only this release). +On Windows the recorded session runs under a UTF-8 console codepage (the +programmatic `chcp 65001` via `[Console]::OutputEncoding`/`InputEncoding`), +and the engine temporarily switches the calling console's output codepage +to UTF-8 for the duration of the recording, restoring it afterwards. This +keeps the live view of byte-writing TUIs (OpenCode and other Go/Node +applications) free of CP437 mojibake even on OEM-codepage consoles such as +cmd or the Windows PowerShell 5.1 host - the cast file itself is UTF-8 +encoded regardless. + ## Why not record the outer session like Start-Transcript? PowerShell exposes **no public hook, event or handler** for intercepting the diff --git a/src/Bsn.Host/Bsn.Host.psd1 b/src/Bsn.Host/Bsn.Host.psd1 index 9b89f24..b374d4e 100644 --- a/src/Bsn.Host/Bsn.Host.psd1 +++ b/src/Bsn.Host/Bsn.Host.psd1 @@ -1,6 +1,6 @@ @{ RootModule = 'Bsn.Host.psm1' - ModuleVersion = '0.2.2' + ModuleVersion = '0.2.3' GUID = '9d84a7b8-1f2b-4d60-a4bd-3d7ef1d9f337' Author = 'BSVN' CompanyName = 'BSVN' @@ -21,7 +21,7 @@ Tags = @('BSVN', 'Lmt', 'Host', 'Transcript', 'Recording', 'Terminal', 'Session', 'Asciicast', 'Asciinema', 'TUI', 'Audit', 'Screencast') LicenseUri = 'https://github.com/BSVN/Lmt.Commons/blob/main/LICENSE' ProjectUri = 'https://github.com/BSVN/Lmt.Commons' - ReleaseNotes = 'Bug fix release: recordings of byte-writing TUI applications (OpenCode and other Go/Node TUIs) are no longer corrupted to codepage mojibake - the recorded session bootstraps the console codepage to UTF-8; interop assembly loading is loader-managed (Import-Module -Force reuses the loaded assembly; cross-version imports get clear guidance and a session restart is required to switch versions). Also includes the 0.2.1 fixes: multi-byte UTF-8 survives chunk boundaries; cast files stay valid when the recording process dies; -Command values round-trip safely through pwsh -EncodedCommand; recording with both stdin and stdout redirected is a terminating error. Fixes issues #5, #6, #7, #8 and #15.' + ReleaseNotes = 'Bug fix release: the LIVE recording view (what you watch while recording) no longer turns box-drawing text into CP437 mojibake on OEM-codepage consoles (cmd, Windows PowerShell 5.1 hosts) - the engine switches the console to UTF-8 for the recording and restores it afterwards; the cast file was always correct. Also includes the 0.2.2 fixes: recordings of byte-writing TUI applications (OpenCode and other Go/Node TUIs) bootstrap the console codepage to UTF-8; interop assembly loading is loader-managed (Import-Module -Force reuses the loaded assembly; cross-version imports get clear guidance). Fixes issues #5 and #15.' } } } diff --git a/src/Bsn.Host/Public/Start-InteractiveTranscript.ps1 b/src/Bsn.Host/Public/Start-InteractiveTranscript.ps1 index 50c28cf..a8a13e1 100644 --- a/src/Bsn.Host/Public/Start-InteractiveTranscript.ps1 +++ b/src/Bsn.Host/Public/Start-InteractiveTranscript.ps1 @@ -138,6 +138,11 @@ using module Bsn.Common process output around the pseudo-console. Run from an interactive console session for full capture. - Non-console hosts receive a warning that recording may not work. + - On Windows, the console output codepage is temporarily switched to + UTF-8 while recording (the live proxy writes UTF-8 bytes) and + restored afterwards, so the live view renders byte-writing TUIs + correctly even on OEM-codepage consoles (cmd, Windows PowerShell + 5.1 hosts). Implements https://github.com/PowerShell/PowerShell/issues/28353 following the parameter and format semantics of PowerShell-RFC#421 at diff --git a/src/Bsn.Host/en-US/about_Bsn.Host.help.txt b/src/Bsn.Host/en-US/about_Bsn.Host.help.txt index 0be3592..ae70f62 100644 --- a/src/Bsn.Host/en-US/about_Bsn.Host.help.txt +++ b/src/Bsn.Host/en-US/about_Bsn.Host.help.txt @@ -83,6 +83,12 @@ LONG DESCRIPTION process output around the pseudo-console. Run from an interactive console session for full capture. + On Windows the console output codepage is temporarily switched to + UTF-8 while recording (the live proxy writes UTF-8 bytes) and + restored afterwards, so the live view renders byte-writing TUIs + correctly even on OEM-codepage consoles such as cmd or the Windows + PowerShell 5.1 host. + SCENARIOS - Audit and compliance records of privileged interactive sessions diff --git a/src/Bsn.Host/src/PtyRecorder.cs b/src/Bsn.Host/src/PtyRecorder.cs index e30c66e..9eba31e 100644 --- a/src/Bsn.Host/src/PtyRecorder.cs +++ b/src/Bsn.Host/src/PtyRecorder.cs @@ -118,6 +118,12 @@ private struct SECURITY_ATTRIBUTES [DllImport("kernel32.dll", SetLastError = true)] private static extern bool SetConsoleMode(IntPtr hConsoleHandle, uint dwMode); + [DllImport("kernel32.dll", SetLastError = true)] + private static extern uint GetConsoleOutputCP(); + + [DllImport("kernel32.dll", SetLastError = true)] + private static extern bool SetConsoleOutputCP(uint wCodePageID); + [DllImport("kernel32.dll", SetLastError = true)] private static extern bool CreatePipe(out IntPtr hReadPipe, out IntPtr hWritePipe, ref SECURITY_ATTRIBUTES lpPipeAttributes, uint nSize); @@ -177,6 +183,22 @@ public static RecordingResult Run(string commandLine, string castPath, bool reco uint savedInputMode; bool hadConsoleMode = GetConsoleMode(hRealIn, out savedInputMode); + // The live proxy writes the ConPTY output stream to the real console as + // raw UTF-8 bytes (the same bytes that go into the cast). The console + // decodes those bytes with its OUTPUT CODEPAGE: on an OEM-codepage + // console (437 - cmd, Windows PowerShell 5.1, fresh hidden consoles) + // the UTF-8 text of byte-writing TUIs renders as CP437 mojibake + // ('Gamma-ohmega-acute') in the LIVE view even though the cast is + // clean, because the console - not the recorder - decodes the bytes. + // Switch the console to UTF-8 for the duration of the recording and + // restore the original codepage afterwards; this is the console-host + // counterpart of the wrapper's codepage bootstrap on the pseudo + // console side (issue #5). A return value of 0 means the process has + // no attached console - nothing to switch or restore. + uint savedOutputCodePage = GetConsoleOutputCP(); + bool changedOutputCodePage = (savedOutputCodePage != 0) && (savedOutputCodePage != 65001); + if (changedOutputCodePage) { SetConsoleOutputCP(65001); } + short width = 120; short height = 30; CONSOLE_SCREEN_BUFFER_INFO bufferInfo; @@ -326,6 +348,7 @@ public static RecordingResult Run(string commandLine, string castPath, bool reco state.Stop = true; clock.Stop(); if (hadConsoleMode) { SetConsoleMode(hRealIn, savedInputMode); } + if (changedOutputCodePage) { SetConsoleOutputCP(savedOutputCodePage); } if (cast != null) { cast.Dispose(); } if (childThread != IntPtr.Zero) { CloseHandle(childThread); } if (childProcess != IntPtr.Zero) { CloseHandle(childProcess); } diff --git a/src/Bsn.Host/tests/system/InteractiveTranscript.System.Tests.ps1 b/src/Bsn.Host/tests/system/InteractiveTranscript.System.Tests.ps1 index 3e79216..0a8c273 100644 --- a/src/Bsn.Host/tests/system/InteractiveTranscript.System.Tests.ps1 +++ b/src/Bsn.Host/tests/system/InteractiveTranscript.System.Tests.ps1 @@ -415,6 +415,9 @@ Describe 'InteractiveTranscript system tests' -Tag 'SystemTest' { if (-not $script:HasConsoleLogger) { Set-ItResult -Skipped -Because 'conlog.exe is not on PATH (build from https://github.com/rhubarb-geek-nz/console-logger).' } + if ([Console]::IsInputRedirected -or [Console]::IsOutputRedirected) { + Set-ItResult -Skipped -Because 'conlog requires console-attached stdio (it reads its own console input); run locally in a terminal - the live-view codepage test covers conlog end to end under CI-safe hidden consoles.' + } $scenarioPath = Get-BsnScenarioScript -Directory $TestDrive @@ -780,6 +783,121 @@ opencode $hasKnownUiText = $rawStream.Contains('New session') -or $rawStream.Contains('Ask anything') -or $rawStream.Contains('Build') $hasKnownUiText | Should -BeTrue -Because 'the OpenCode UI text must be captured (proves the recorded session is the TUI)' } + + It 'keeps the LIVE proxied view clean on an OEM-codepage console (conlog)' -Tag 'RequiresConLog' { + # https://github.com/BSVN/Lmt.Commons/issues/5 + # Field regression from the v0.2.2 rollout: the CAST was clean (all + # capture tests pass) yet the user still watched mojibake during the + # recording. Cause: the engine proxies the ConPTY stream to the REAL + # console as raw UTF-8 bytes, and the console decodes those bytes + # with its own OUTPUT codepage - OEM 437 in cmd, Windows PowerShell + # 5.1 hosts and fresh hidden consoles. UTF-8 box borders became + # CP437 soup ('Gamma-ohmega-acute') in the live view while every + # cast-asserting test stayed green. This test reproduces that + # topology: conlog wraps the runner in its own pseudoconsole, the + # runner forces its console to the OEM codepage like the reporting + # user's, and the capture shows exactly what that console rendered. + if (-not $script:CanRunSystemTests) { + Set-ItResult -Skipped -Because 'pwsh is required to run recorded scenarios and was not found.' + } + if ($PSVersionTable.PSVersion.Major -lt 7) { + Set-ItResult -Skipped -Because 'system tests require PowerShell 7 console infrastructure.' + } + if (-not $script:IsWindowsPlatform) { + Set-ItResult -Skipped -Because 'the live-view codepage test targets the Windows ConPTY engine.' + } + if (-not $script:HasConsoleLogger) { + Set-ItResult -Skipped -Because 'conlog.exe is not on PATH (build from https://github.com/rhubarb-geek-nz/console-logger).' + } + + # Probe: UTF-8 box border plus ASCII markers, written as RAW BYTES + # to the console output handle by the recorded child - byte-writer + # semantics, the shape every Go/Node TUI produces. + $liveProbePath = Join-Path -Path $TestDrive -ChildPath 'liveview-probe.bin' + $thinRule = [string][char]0x2500 + $liveProbeText = 'LIVE-HEAD' + "`r`n" + + ([string][char]0x256D) + ($thinRule * 20) + ([string][char]0x256E) + "`r`n" + + ([string][char]0x2502) + ' LIVE-MARKER ' + ([string][char]0x2502) + "`r`n" + + ([string][char]0x2570) + ($thinRule * 20) + ([string][char]0x256F) + "`r`n" + + 'LIVE-TAIL' + [System.IO.File]::WriteAllBytes($liveProbePath, [System.Text.Encoding]::UTF8.GetBytes($liveProbeText)) + + $liveScenarioPath = Join-Path -Path $TestDrive -ChildPath 'liveview-child.ps1' + @' +$payload = [System.IO.File]::ReadAllBytes($env:BsnLiveViewProbeFile) +$sink = [Console]::OpenStandardOutput() +$sink.Write($payload, 0, $payload.Length) +$sink.Flush() +'@ | Set-Content -LiteralPath $liveScenarioPath -Encoding UTF8 + + $env:BsnLiveViewProbeFile = $liveProbePath + try { + # The runner executes INSIDE the conlog pseudoconsole (so its + # stdio are console handles and in-process recording works - + # same reason the hidden-console runner exists for CI), forces + # its console output codepage to OEM 437 like the reporting + # user's, records the byte-writing child, and reports the + # console codepage from before and after the recording: the + # engine must put it back. + $liveRunnerPath = Join-Path -Path $TestDrive -ChildPath 'liveview-runner.ps1' + $liveResultPath = Join-Path -Path $TestDrive -ChildPath 'liveview-result.json' + $liveCastPath = Join-Path -Path $TestDrive -ChildPath 'liveview.cast' + @" +param(`$RepoRoot, `$ScenarioPath, `$CastPath, `$ResultPath) +`$ErrorActionPreference = 'Stop' +chcp.com 437 | Out-Null +Import-Module (Join-Path `$RepoRoot 'src/Bsn.Common/Bsn.Common.psd1') -Force +Import-Module (Join-Path `$RepoRoot 'src/Bsn.Host/Bsn.Host.psd1') -Force +`$codePageBefore = ([regex]::Match((chcp.com | Out-String), '(\d+)')).Groups[1].Value +`$recording = Start-InteractiveTranscript -Command ('pwsh -NoProfile -File ' + `$ScenarioPath) -Path `$CastPath +`$codePageAfter = ([regex]::Match((chcp.com | Out-String), '(\d+)')).Groups[1].Value +@{ Exit = `$recording.ExitCode; CodePageBefore = `$codePageBefore; CodePageAfter = `$codePageAfter } | + ConvertTo-Json | Set-Content -LiteralPath `$ResultPath +"@ | Set-Content -LiteralPath $liveRunnerPath -Encoding UTF8 + + $liveCapturePath = Join-Path -Path $TestDrive -ChildPath 'liveview-capture.txt' + $conlogCommand = Get-Command -Name 'conlog' -ErrorAction 'Stop' + # Start-Process (not direct invocation) so conlog owns a fresh + # hidden console: conlog reads its stdin as a console handle and + # needs stdout OR stderr attached to a console device, which CI's + # redirected stdio cannot provide (same trick as the + # hidden-console recording runner). Only stdout is redirected to + # the capture file - conlog's stdout IS the live view: what the + # runner console actually rendered. stderr stays on the hidden + # console (dropped). + $conlogProcess = Start-Process -FilePath $conlogCommand.Source ` + -ArgumentList @("pwsh -NoProfile -ExecutionPolicy Bypass -File $liveRunnerPath -RepoRoot $($script:RepoRoot) -ScenarioPath $liveScenarioPath -CastPath $liveCastPath -ResultPath $liveResultPath") ` + -WindowStyle Hidden -RedirectStandardOutput $liveCapturePath -PassThru + if (-not $conlogProcess.WaitForExit(120000)) { + $conlogProcess.Kill() + throw 'conlog live-view runner timed out after 120 seconds.' + } + if (-not (Test-Path -LiteralPath $liveResultPath)) { + throw ("conlog live-view runner produced no result (exit code {0}); capture: {1}" -f $conlogProcess.ExitCode, (Get-Content -LiteralPath $liveCapturePath -Raw)) + } + + $liveOutcome = Get-Content -LiteralPath $liveResultPath -Raw | ConvertFrom-Json + $liveOutcome.Exit | Should -Be 0 -Because 'the recorded scenario must succeed' + $liveOutcome.CodePageBefore | Should -Be '437' -Because 'the runner console must sit at the OEM codepage to reproduce the reported environment' + $liveOutcome.CodePageAfter | Should -Be '437' -Because 'the engine must restore the original console codepage after the recording (issue #5 live view)' + + $liveStream = Get-Content -LiteralPath $liveCapturePath -Raw + $liveStream.Contains('LIVE-MARKER') | Should -BeTrue -Because 'the proxied live view must carry the payload text' + + $liveGlyphCount = { param($codePoint) ([regex]::Matches($liveStream, [regex]::Escape([string][char]$codePoint))).Count } + (& $liveGlyphCount 0x256D) | Should -Be 1 -Because 'the live view must render the rounded corner itself, not its CP437-decoded byte soup' + (& $liveGlyphCount 0x2500) | Should -Be 40 -Because 'both border rows must survive into the live view' + + # CP437-decoded UTF-8 shows up as Latin-1-range letters plus Greek + # Gamma (0xE2 is 'Gamma' in CP437, the lead byte of every 3-byte + # UTF-8 sequence) - none of that may appear in the live view. + $liveMojibake = ([regex]::Matches($liveStream, '[\u00C0-\u00FF\u0393]')).Count + $liveMojibake | Should -Be 0 -Because ('the live proxied view must be free of codepage mojibake on an OEM-codepage console (found {0}); the console decodes the recorder''s raw UTF-8 bytes with its output codepage (issue #5 live view)' -f $liveMojibake) + } + finally { + Remove-Item 'Env:BsnLiveViewProbeFile' -ErrorAction 'SilentlyContinue' + } + } } Context 'Issue #7: metacharacters survive a REAL ConPTY recording' { From b877b19889355e55a0d228aa090bf33fb09d69bd Mon Sep 17 00:00:00 2001 From: Seyyed Soroosh Hosseinalipour Date: Sat, 7 Nov 2026 23:54:36 +0330 Subject: [PATCH 2/3] Fix #5 (playback): switch the console output encoding to UTF-8 in Show-InteractiveTranscript Second half of the viewer bug: the recorded cast is lossless UTF-8, but Show-InteractiveTranscript in Windows PowerShell 5.1 (.NET Framework) writes [Console] output through the console OUTPUT CODEPAGE (WriteFile), so heavy box-drawing characters that do not exist in CP437 - OpenCode's U+2503 border and U+2579 corner - degraded to literal '?' in the viewer while ASCII and CP437 blocks (the logo, the text) survived. pwsh 7 uses WriteConsoleW and was never affected, which is why the pwsh-only tests did not catch it. Show-InteractiveTranscript now saves the console output encoding, switches it to UTF-8 for the playback, and restores the original afterwards (same save/switch/restore discipline the recording engine uses for the live proxy view). Consoles already at UTF-8 are untouched; hosts that refuse the switch fall back to the previous behavior via a guarded setter. Tests: - Unit: playback restores the console output encoding it found. - Unit (discriminating): a child pwsh with redirected stdout encodes [Console]::Write through [Console]::OutputEncoding - the same lossy path as an interactive 5.1 host. With a cast containing U+2503/U+2579 and the console forced to CP437, the output must contain the UTF-8 characters and no '?' (red on the unfixed cmdlet, green with the fix). - System (end to end): a real recording played back through powershell.exe 5.1 inside a conlog pseudoconsole at the OEM codepage - corners, borders and markers must render with zero rendered '?', and the console codepage must be restored (red unfixed, green fixed). - Also adds the redirected-stdio skip guard the console-logger cross-check test was missing (conlog cannot run when its own stdin is redirected). --- AGENTS.md | 6 +- CHANGELOG.md | 16 +++ .../Public/Show-InteractiveTranscript.ps1 | 73 ++++++++++--- src/Bsn.Host/en-US/about_Bsn.Host.help.txt | 6 +- .../InteractiveTranscript.System.Tests.ps1 | 102 ++++++++++++++++++ .../unit/Show-InteractiveTranscript.Tests.ps1 | 71 ++++++++++++ 6 files changed, 257 insertions(+), 17 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 8654d7f..5dc8373 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -157,7 +157,11 @@ before: UTF-8 so the live proxy view decodes correctly on OEM-codepage consoles), close the PTY-side pipe handles after CreatePseudoConsole (EchoCon pattern), and emit cast files - as UTF-8 without BOM with LF line endings. Cast files support asciicast + as UTF-8 without BOM with LF line endings. + `Show-InteractiveTranscript` follows the same save/switch/restore + discipline for its playback writes (Windows PowerShell 5.1 encodes + `[Console]` output through the console output codepage, which turns + CP437-unrepresentable characters into '?'). Cast files support asciicast v2 (absolute event times) and v3 (intervals since the previous event, millisecond precision, trailing 'x' exit event) - keep both writers in sync with ConvertTo-BsnAsciicast. `ConvertTo-BsnAsciicast` and diff --git a/CHANGELOG.md b/CHANGELOG.md index 2f3544a..30684fb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -20,6 +20,15 @@ and the modules adhere to [Semantic Versioning](https://semver.org/). (same restore discipline as the console input mode). Reportedly hit with OpenCode after installing 0.2.2 from the gallery. [#5](https://github.com/BSVN/Lmt.Commons/issues/5) +- `Show-InteractiveTranscript` playback no longer degrades characters + outside CP437 (e.g. OpenCode's heavy box-drawing border `U+2503`) + to literal `?` on OEM-codepage consoles: Windows PowerShell 5.1 + hosts write `[Console]` output through the console output codepage, + which mangled exactly those characters while ASCII and CP437 blocks + survived. The playback now switches the console output encoding to + UTF-8 for its duration and restores it afterwards. The cast file + was always lossless UTF-8 - pwsh 7 playback was never affected. + [#5](https://github.com/BSVN/Lmt.Commons/issues/5) ### Added @@ -29,6 +38,13 @@ and the modules adhere to [Semantic Versioning](https://semver.org/). themselves with zero CP437 mojibake, and the console codepage must be restored after the recording. [#5](https://github.com/BSVN/Lmt.Commons/issues/5) +- A playback system test playing a real recording back through + powershell.exe 5.1 inside a conlog pseudoconsole at the OEM codepage: + corners, borders and text must render and no CP437-unrepresentable + character may degrade to `?`; plus unit tests asserting the console + output encoding is restored after playback and that heavy box + characters survive a CP437-encoding console losslessly. + [#5](https://github.com/BSVN/Lmt.Commons/issues/5) ## [Bsn.Host 0.2.2] - 2026-11-07 diff --git a/src/Bsn.Host/Public/Show-InteractiveTranscript.ps1 b/src/Bsn.Host/Public/Show-InteractiveTranscript.ps1 index 0df2e80..8305f09 100644 --- a/src/Bsn.Host/Public/Show-InteractiveTranscript.ps1 +++ b/src/Bsn.Host/Public/Show-InteractiveTranscript.ps1 @@ -12,6 +12,12 @@ Input ('i') events are not replayed; resize ('r') events are ignored (the current console keeps its own size). + On Windows the console output encoding is temporarily switched to + UTF-8 for the playback and restored afterwards: Windows PowerShell + 5.1 hosts write [Console] output through the console output codepage, + which would otherwise render characters outside CP437 (e.g. heavy + box-drawing borders) as literal '?'. + .PARAMETER Path The asciicast v2 (.cast) file to play back. @@ -57,25 +63,62 @@ function Show-InteractiveTranscript { process { $timeline = Get-BsnCastTimeline -Path $Path - $previousTime = 0.0 - foreach ($timelineEvent in $timeline.Events) { - $eventTime = [double]$timelineEvent[0] - $delaySeconds = $eventTime - $previousTime - if (($IdleTimeLimit -gt 0) -and ($delaySeconds -gt $IdleTimeLimit)) { - $delaySeconds = $IdleTimeLimit + # Windows PowerShell 5.1 (.NET Framework) writes [Console] output + # through the console OUTPUT CODEPAGE (WriteFile), so playback + # characters outside that codepage - e.g. OpenCode's heavy + # box-drawing border U+2503 on an OEM-codepage (437) console - are + # replaced with literal '?' before rendering, while ASCII and CP437 + # blocks survive (the recorded cast itself is lossless UTF-8; pwsh 7 + # writes via WriteConsoleW and is unaffected). Switch the console + # output encoding to UTF-8 for the duration of the playback and + # restore the original afterwards - the same restore discipline the + # recording engine uses for the live proxy view (issue #5). + $previousOutputEncoding = [Console]::OutputEncoding + $changedOutputEncoding = $false + if ($previousOutputEncoding.CodePage -ne 65001) { + try { + [Console]::OutputEncoding = [System.Text.Encoding]::UTF8 + $changedOutputEncoding = $true } - $delaySeconds = $delaySeconds / $Speed - - if ($delaySeconds -gt 0) { - Start-Sleep -Milliseconds ([int]($delaySeconds * 1000)) + catch { + # No attached console or a host that refuses the switch (some + # IDE/redirected setups): playback continues with the current + # encoding, which is what previous versions did. + Write-Verbose -Message ('Could not switch the console output encoding to UTF-8 for playback: ' + $_.Exception.Message) } + } - $eventType = [string]$timelineEvent[1] - if ($eventType -eq 'o') { - [Console]::Write([string]$timelineEvent[2]) + try { + $previousTime = 0.0 + foreach ($timelineEvent in $timeline.Events) { + $eventTime = [double]$timelineEvent[0] + $delaySeconds = $eventTime - $previousTime + if (($IdleTimeLimit -gt 0) -and ($delaySeconds -gt $IdleTimeLimit)) { + $delaySeconds = $IdleTimeLimit + } + $delaySeconds = $delaySeconds / $Speed + + if ($delaySeconds -gt 0) { + Start-Sleep -Milliseconds ([int]($delaySeconds * 1000)) + } + + $eventType = [string]$timelineEvent[1] + if ($eventType -eq 'o') { + [Console]::Write([string]$timelineEvent[2]) + } + + $previousTime = $eventTime + } + } + finally { + if ($changedOutputEncoding) { + try { + [Console]::OutputEncoding = $previousOutputEncoding + } + catch { + Write-Verbose -Message ('Could not restore the console output encoding: ' + $_.Exception.Message) + } } - - $previousTime = $eventTime } } } diff --git a/src/Bsn.Host/en-US/about_Bsn.Host.help.txt b/src/Bsn.Host/en-US/about_Bsn.Host.help.txt index ae70f62..a4c3817 100644 --- a/src/Bsn.Host/en-US/about_Bsn.Host.help.txt +++ b/src/Bsn.Host/en-US/about_Bsn.Host.help.txt @@ -38,7 +38,11 @@ LONG DESCRIPTION Show-InteractiveTranscript plays recordings back in the console with original or adjusted timing (-Speed, -IdleTimeLimit). Recordings also play anywhere the open asciicast v2 ecosystem works, for example - 'asciinema play .cast'. + 'asciinema play .cast'. On Windows the console output encoding + is temporarily switched to UTF-8 during playback and restored + afterwards, so characters outside CP437 (heavy box-drawing borders) + do not degrade to '?' on OEM-codepage consoles or in Windows + PowerShell 5.1 hosts. RECORDING OPTIONS (RFC 421 SEMANTICS) diff --git a/src/Bsn.Host/tests/system/InteractiveTranscript.System.Tests.ps1 b/src/Bsn.Host/tests/system/InteractiveTranscript.System.Tests.ps1 index 0a8c273..cb88995 100644 --- a/src/Bsn.Host/tests/system/InteractiveTranscript.System.Tests.ps1 +++ b/src/Bsn.Host/tests/system/InteractiveTranscript.System.Tests.ps1 @@ -898,6 +898,108 @@ Import-Module (Join-Path `$RepoRoot 'src/Bsn.Host/Bsn.Host.psd1') -Force Remove-Item 'Env:BsnLiveViewProbeFile' -ErrorAction 'SilentlyContinue' } } + + It 'keeps PLAYBACK lossless on an OEM-codepage console in Windows PowerShell 5.1 (conlog)' { + # https://github.com/BSVN/Lmt.Commons/issues/5 + # Second half of the viewer bug: Show-InteractiveTranscript in + # Windows PowerShell 5.1 writes [Console] output through the + # console output CODEPAGE (WriteFile), so heavy box characters + # (U+2503, U+2579, U+256D - none exist in CP437) degraded to + # literal '?' in the playback while ASCII and CP437 blocks + # survived - exactly the reported OpenCode viewer screenshot. + # The playback must switch the console output encoding to UTF-8 + # for its duration and restore it. This test plays a real + # recording back through powershell.exe 5.1 inside a conlog + # pseudoconsole forced to the OEM codepage: the capture is what + # that console rendered. + if (-not $script:CanRunSystemTests) { + Set-ItResult -Skipped -Because 'pwsh is required to run recorded scenarios and was not found.' + } + if (-not $script:IsWindowsPlatform) { + Set-ItResult -Skipped -Because 'the playback codepage test targets the Windows ConPTY engine.' + } + if (-not $script:HasConsoleLogger) { + Set-ItResult -Skipped -Because 'conlog.exe is not on PATH (build from https://github.com/rhubarb-geek-nz/console-logger).' + } + + # Record a cast that contains corner characters (not in CP437): + # the byte-writing probe child makes the recording deterministic. + $probePath = Join-Path -Path $TestDrive -ChildPath 'playback-probe.bin' + $rule = [string][char]0x2500 + $probeText = 'PB-HEAD' + "`r`n" + + ([string][char]0x256D) + ($rule * 18) + ([string][char]0x256E) + "`r`n" + + ([string][char]0x2502) + ' PB-MARKER ' + ([string][char]0x2502) + "`r`n" + + ([string][char]0x2570) + ($rule * 18) + ([string][char]0x256F) + "`r`n" + + 'PB-TAIL' + [System.IO.File]::WriteAllBytes($probePath, [System.Text.Encoding]::UTF8.GetBytes($probeText)) + + $childPath = Join-Path -Path $TestDrive -ChildPath 'playback-child.ps1' + @' +$payload = [System.IO.File]::ReadAllBytes($env:BsnPlaybackProbeFile) +$sink = [Console]::OpenStandardOutput() +$sink.Write($payload, 0, $payload.Length) +$sink.Flush() +'@ | Set-Content -LiteralPath $childPath -Encoding UTF8 + + $env:BsnPlaybackProbeFile = $probePath + try { + $castPath = Join-Path -Path $TestDrive -ChildPath 'playback.cast' + $null = Invoke-BsnSystemRecording -ScenarioPath $childPath -CastPath $castPath + + # Play the cast back through powershell.exe 5.1 inside a conlog + # pseudoconsole whose console sits at the OEM codepage: conlog's + # capture is the rendered playback, exactly what the reporting + # user's viewer showed. + $playRunnerPath = Join-Path -Path $TestDrive -ChildPath 'playback-runner51.ps1' + $playResultPath = Join-Path -Path $TestDrive -ChildPath 'playback-result.json' + @" +param(`$RepoRoot, `$CastPath, `$ResultPath) +`$ErrorActionPreference = 'Stop' +chcp.com 437 | Out-Null +Import-Module (Join-Path `$RepoRoot 'src/Bsn.Common/Bsn.Common.psd1') -Force +Import-Module (Join-Path `$RepoRoot 'src/Bsn.Host/Bsn.Host.psd1') -Force +`$codePageBefore = ([regex]::Match((chcp.com | Out-String), '(\d+)')).Groups[1].Value +Show-InteractiveTranscript -Path `$CastPath +`$codePageAfter = ([regex]::Match((chcp.com | Out-String), '(\d+)')).Groups[1].Value +@{ Before = `$codePageBefore; After = `$codePageAfter } | ConvertTo-Json | Set-Content -LiteralPath `$ResultPath +"@ | Set-Content -LiteralPath $playRunnerPath -Encoding UTF8 + + $playCapturePath = Join-Path -Path $TestDrive -ChildPath 'playback-capture.txt' + $conlogCommand = Get-Command -Name 'conlog' -ErrorAction 'Stop' + $conlogProcess = Start-Process -FilePath $conlogCommand.Source ` + -ArgumentList @("powershell.exe -NoProfile -ExecutionPolicy Bypass -File $playRunnerPath -RepoRoot $($script:RepoRoot) -CastPath $castPath -ResultPath $playResultPath") ` + -WindowStyle Hidden -RedirectStandardOutput $playCapturePath -PassThru + if (-not $conlogProcess.WaitForExit(120000)) { + $conlogProcess.Kill() + throw 'conlog playback runner timed out after 120 seconds.' + } + if (-not (Test-Path -LiteralPath $playResultPath)) { + throw ("conlog playback runner produced no result (exit code {0}); capture: {1}" -f $conlogProcess.ExitCode, (Get-Content -LiteralPath $playCapturePath -Raw)) + } + + $playOutcome = Get-Content -LiteralPath $playResultPath -Raw | ConvertFrom-Json + $playOutcome.Before | Should -Be '437' -Because 'the playback console must sit at the OEM codepage to reproduce the reported viewer' + $playOutcome.After | Should -Be '437' -Because 'the playback must restore the console output encoding it found (issue #5 viewer bug)' + + $playStream = Get-Content -LiteralPath $playCapturePath -Raw + $playStream.Contains('PB-MARKER') | Should -BeTrue -Because 'the playback must carry the payload text' + + $playGlyphCount = { param($codePoint) ([regex]::Matches($playStream, [regex]::Escape([string][char]$codePoint))).Count } + (& $playGlyphCount 0x256D) | Should -Be 1 -Because 'the corner must render in the playback, not degrade to a question mark (issue #5 viewer bug)' + (& $playGlyphCount 0x2500) | Should -Be 36 -Because 'both border rows must survive playback' + + # Count RENDERED question marks: strip VT sequences first (the + # passthrough stream legitimately contains '?' inside CSI + # private-mode sequences like ESC[?25l - those are interpreted, + # never rendered). + $plainPlayback = Convert-BsnCastPlainText -RawStream $playStream + $playQuestionMarks = ([regex]::Matches($plainPlayback, '\?')).Count + $playQuestionMarks | Should -Be 0 -Because ('CP437-unrepresentable characters must not degrade to literal question marks during playback (found {0}; issue #5 viewer bug)' -f $playQuestionMarks) + } + finally { + Remove-Item 'Env:BsnPlaybackProbeFile' -ErrorAction 'SilentlyContinue' + } + } } Context 'Issue #7: metacharacters survive a REAL ConPTY recording' { diff --git a/src/Bsn.Host/tests/unit/Show-InteractiveTranscript.Tests.ps1 b/src/Bsn.Host/tests/unit/Show-InteractiveTranscript.Tests.ps1 index 9e543da..35a9bb3 100644 --- a/src/Bsn.Host/tests/unit/Show-InteractiveTranscript.Tests.ps1 +++ b/src/Bsn.Host/tests/unit/Show-InteractiveTranscript.Tests.ps1 @@ -85,4 +85,75 @@ Describe 'Show-InteractiveTranscript' { It 'accepts the cast path from the pipeline' { { $script:ShowPath | Show-InteractiveTranscript -Speed 100 } | Should -Not -Throw } + + It 'restores the console output encoding after playback' { + # https://github.com/BSVN/Lmt.Commons/issues/5 + # The playback temporarily switches the console output encoding to + # UTF-8 (Windows PowerShell 5.1 hosts write [Console] output through + # the codepage, which mangles non-CP437 box characters to '?'); the + # original encoding must come back afterwards. + if ('System.Text.CodePagesEncodingProvider' -as [type]) { + [System.Text.Encoding]::RegisterProvider([System.Text.CodePagesEncodingProvider]::Instance) + } + $previousEncoding = [Console]::OutputEncoding + try { + [Console]::OutputEncoding = [System.Text.Encoding]::GetEncoding(437) + + Show-InteractiveTranscript -Path $script:ShowPath -Speed 100 + + [Console]::OutputEncoding.CodePage | Should -Be 437 -Because 'the playback must restore the console output encoding it found' + } + finally { + [Console]::OutputEncoding = $previousEncoding + } + } + + It 'writes heavy box characters losslessly even when the console encodes them as question marks' { + # https://github.com/BSVN/Lmt.Commons/issues/5 + # Reproduces the reported OpenCode viewer bug: a cast with heavy + # box-drawing characters (U+2503 border, U+2579 corner - NOT in + # CP437) played back by a host whose [Console]::Write goes through + # the console output codepage. Before the fix those characters + # reached the screen as literal '?'; the playback must switch the + # console output encoding to UTF-8 so they arrive intact. A child + # pwsh with redirected stdout encodes [Console]::Write through + # [Console]::OutputEncoding - the same lossy path as an interactive + # Windows PowerShell 5.1 host. + if ('System.Text.CodePagesEncodingProvider' -as [type]) { + [System.Text.Encoding]::RegisterProvider([System.Text.CodePagesEncodingProvider]::Instance) + } + + $unicodeCastPath = Join-Path -Path $TestDrive -ChildPath 'show-unicode.cast' + $eventText = 'A' + [string][char]0x2503 + 'B' + [string][char]0x2579 + 'C' + $eventJson = ConvertTo-Json -InputObject $eventText -Compress + $castLines = @( + '{"version": 2, "width": 80, "height": 24, "timestamp": 1793895000, "env": {"TERM": "xterm-256color"}}' + ('[0, "o", ' + $eventJson + ']') + ) + Set-Content -LiteralPath $unicodeCastPath -Value $castLines -Encoding UTF8 + + $childScriptPath = Join-Path -Path $TestDrive -ChildPath 'show-oem-child.ps1' + @' +param($ModuleRoot, $CastPath) +$env:PSModulePath = (Split-Path -Parent -Path $ModuleRoot) + [System.IO.Path]::PathSeparator + $env:PSModulePath +Import-Module (Join-Path $ModuleRoot 'Bsn.Host.psd1') -Force +[System.Text.Encoding]::RegisterProvider([System.Text.CodePagesEncodingProvider]::Instance) +[Console]::OutputEncoding = [System.Text.Encoding]::GetEncoding(437) +Show-InteractiveTranscript -Path $CastPath +'@ | Set-Content -LiteralPath $childScriptPath -Encoding UTF8 + + $childOutputPath = Join-Path -Path $TestDrive -ChildPath 'show-oem-out.txt' + $childErrorPath = Join-Path -Path $TestDrive -ChildPath 'show-oem-err.txt' + $null = Start-Process -FilePath (Get-Process -Id $PID).Path ` + -ArgumentList @('-NoProfile', '-ExecutionPolicy', 'Bypass', '-File', $childScriptPath, '-ModuleRoot', $moduleRoot, '-CastPath', $unicodeCastPath) ` + -WindowStyle Hidden -RedirectStandardOutput $childOutputPath -RedirectStandardError $childErrorPath -PassThru -Wait + + # The playback switched the console output encoding to UTF-8 for the + # write: the heavy box characters arrive as UTF-8, not as '?'. + $playbackBytes = [System.IO.File]::ReadAllBytes($childOutputPath) + $playbackText = [System.Text.Encoding]::UTF8.GetString($playbackBytes) + $playbackText.Contains([string][char]0x2503) | Should -BeTrue -Because 'the heavy vertical border must survive playback on a CP437-encoding console (issue #5 viewer bug)' + $playbackText.Contains([string][char]0x2579) | Should -BeTrue -Because 'the heavy corner must survive playback on a CP437-encoding console (issue #5 viewer bug)' + $playbackText | Should -Not -Match '\?' -Because ('characters outside CP437 must not degrade to literal question marks (got: {0})' -f $playbackText) + } } From 55bdfb2c7788af786171823d78f6ef8f3a1e28c3 Mon Sep 17 00:00:00 2001 From: Seyyed Soroosh Hosseinalipour Date: Sun, 8 Nov 2026 00:02:52 +0330 Subject: [PATCH 3/3] Unit test: Start-Process -WindowStyle is Windows-only; apply it conditionally so the lossless-playback test runs on Linux/macOS CI (both failed with NotSupportedException there). --- .../unit/Show-InteractiveTranscript.Tests.ps1 | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/src/Bsn.Host/tests/unit/Show-InteractiveTranscript.Tests.ps1 b/src/Bsn.Host/tests/unit/Show-InteractiveTranscript.Tests.ps1 index 35a9bb3..053d59a 100644 --- a/src/Bsn.Host/tests/unit/Show-InteractiveTranscript.Tests.ps1 +++ b/src/Bsn.Host/tests/unit/Show-InteractiveTranscript.Tests.ps1 @@ -144,9 +144,19 @@ Show-InteractiveTranscript -Path $CastPath $childOutputPath = Join-Path -Path $TestDrive -ChildPath 'show-oem-out.txt' $childErrorPath = Join-Path -Path $TestDrive -ChildPath 'show-oem-err.txt' - $null = Start-Process -FilePath (Get-Process -Id $PID).Path ` - -ArgumentList @('-NoProfile', '-ExecutionPolicy', 'Bypass', '-File', $childScriptPath, '-ModuleRoot', $moduleRoot, '-CastPath', $unicodeCastPath) ` - -WindowStyle Hidden -RedirectStandardOutput $childOutputPath -RedirectStandardError $childErrorPath -PassThru -Wait + # -WindowStyle is Windows-only (Unix pwsh rejects the parameter); on + # Unix the child simply inherits the harness's redirected stdio, which + # is exactly what the lossy-encoding scenario needs. + $childStartArguments = @{ + FilePath = (Get-Process -Id $PID).Path + ArgumentList = @('-NoProfile', '-ExecutionPolicy', 'Bypass', '-File', $childScriptPath, '-ModuleRoot', $moduleRoot, '-CastPath', $unicodeCastPath) + RedirectStandardOutput = $childOutputPath + RedirectStandardError = $childErrorPath + PassThru = $true + Wait = $true + } + if ($IsWindows) { $childStartArguments.WindowStyle = 'Hidden' } + $null = Start-Process @childStartArguments # The playback switched the console output encoding to UTF-8 for the # write: the heavy box characters arrive as UTF-8, not as '?'.