Repository navigation
fix(host-core): preserve BOM-marked UTF-16 text through Read and Edit - #1445
Merged
Merged
Conversation
PowerShell logs contain UTF-16 zero bytes that the binary sniff rejected before decoding. Decode BOM-marked text and retain its byte order and line endings so reading and editing ordinary logs works without changing their encoding.
The summary is outside the UTF-16 fix and duplicates the shared root path added by other ready maintenance PRs. Drop it so the PR stays scoped and subsequent conflict resolution only covers the E2E plan entries.
The latest main contains the completed session index migration and its regression coverage. Revalidate the UTF-16 fix against that integration baseline.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Windows PowerShell 5.1 commonly writes redirected logs as UTF-16LE with a BOM.
Read rejects these ordinary text files as
TOOL_BINARY_CONTENTbecause thebinary sniff treats the encoding's zero bytes as binary data. This reproduces
on upstream 0.17.0, including main at
72b5e826cb7a.Decode BOM-marked UTF-16LE and UTF-16BE before binary classification, and keep
the original encoding when Edit writes the result. The existing UTF-8 behavior,
line anchors and binary-file refusal remain intact.
Reproduction:
"build passed" > build.log.build.log.lines; editing a displayed line retains the BOM, endian and line endings.
Validation:
and pass after the fix: UTF-16LE PowerShell logs and UTF-16BE Chinese text.
cargo test -p host-core --locked -j 2 tools:: -- --test-threads=1: 92 passedon the combined candidate; the two new UTF-16 tests also pass on this standalone
PR branch.
cargo fmt --check: passed.sends decoded Chinese log content to a controlled local provider. This visual
flow was run on the combined candidate, with identical host tool code.
No model-provider, account, IPC or persistent transcript format changes.
Related work: #1296 handles image-file reads, and #1420 preserves unmatched text in legacy edits. This change handles BOM-marked UTF-16 text decoding and encoding preservation.
Validation candidate:
6f6f189c2eadfd24e360d548b2c72de9c80452a2; upstream main:72b5e826cb7a9928467091ccf745aa9b225eeb04.