Repository navigation
host: measure and cut the steward's long-message view in characters - #2548
Conversation
|
@Birfy review request for the current tip of this PR.
If the workflows for this fork PR did not start, approve/run them as well. The tip will not move while this request stands. |
|
The tip moved since my last request (
If the workflows for this fork PR did not start, approve/run them as well. |
The steward's view of an over-long message cut at UTF-16 unit offsets, so a character outside the BMP that straddled either cut reached the model as half a surrogate pair (U+FFFD), and a message under the 1,500-character budget was cut because its units ran over. Both cuts now move onto a character boundary, the budget is counted in characters, and the marker counts the characters left out. Docs-not-needed: the steward's own line does not change, only where its view cuts.
`shorten()` bounded and sliced a long message by UTF-16 units, so a character outside the BMP at either cut reached the model as half of a surrogate pair, and a message that fits in the view's budget was shortened because its units ran over. It now measures with `fits()`, cuts with `codePointEnd` / `codePointStart`, and the marker counts the characters actually left out, the same rule ThinkFlowLab#2321 settled on. The ThinkFlowLab#1993/ThinkFlowLab#2000 sites (b048e26) fixed `cut()` at steward.js:148 and in four other host files; `shorten()` was not in that table. Fixes ThinkFlowLab#2546 Docs-not-needed: the steward's own line does not change, only where its view cuts.
Brings the ThinkFlowLab#2523 client-bundle change (core/client.js, 0003, 0008) under the branch so the generated patches meet the live base. Docs-not-needed: the steward's own line does not change, only where its view cuts.
|
Verdict: PASS at 0e9425e Ran at this tip:
What I checked at this tip:
One thing to watch, not a defect of this tip: the branch base No earlier Verdict existed at this tip (two review-request comments only, no reviews), so this is not a replacement. |
Fixes #2546
Owner: Dev-5
Written-by: Dev-5
Reviewer: Birfy
Base: 276042f
Branch: the branch met
276042f8(the #2523 client bundle) in a merge commit0e9425e9, sopatches checkruns against the live base; nothing in the change set moved with it.The defect
core/src/host/steward.js:51shorten()builds the steward's view of one message: a long one keeps its start and its end, and a marker says how much of the middle was left out. All three numbers in it were UTF-16 units — the budgettext.length <= STEWARD_LINE_MAX(STEWARD_LINE_MAX = 1_500,:21), the headtext.slice(0, 1000), the tailtext.slice(-500), and the markertext.length - STEWARD_LINE_MAX.slicecounts units, so:isWellFormed() === false), and the half becomes U+FFFD when it is encoded for the model, so the steward reads a broken character at both seams.'a'.repeat(1000) + '🚀' + 'b'.repeat(499), 1,500 characters and 1,501 units — is shortened anyway, the rocket is dropped, and the marker tells the steward1 characters left out of this viewalthough nothing needed leaving out.Duplicate check: this is the remaining site of the #1993/#2000 family
b048e26f509b(#2000, merged) — the last five text bounds in core/src/host still cut on UTF-16 units (steward, approvals, hooks, agents, schedules) — fixedcut()there; itssteward.jshunk is the import at:9andcheckpointTailNote(text.slice(0, 80)→cut(text, 80).text, now:148).shorten()was not in that table, so this is a new site, not a repeat of that ticket.steward shorten→ only host: shorten a long message for the steward on character boundaries #2546;"characters left out of this view"→ only host: shorten a long message for the steward on character boundaries #2546;"left out of this view" shorte→ 0.steward.js(checked each PR's file list).core/src/host/steward.jsand itsds-bot/mirror, plus the twosteward.test.jsfiles that assert on it.Red first (base
276042f8, only the test file added)On the clean base both views are ill-formed (
head wellFormed: false,tail wellFormed: false) and both carry the marker — the second one although its 1,500 characters fit.The fix
if (fits(text, STEWARD_LINE_MAX)) return text— the budget is characters, so the marker can no longer report a cut that was not needed.codePointEnd(text, Math.ceil((STEWARD_LINE_MAX * 2) / 3))and the tail starts atcodePointStart(text, text.length - (STEWARD_LINE_MAX - head)), so neither seam lands inside a pair.codePoints(text) - codePoints(head) - codePoints(tail)— the characters actually left out, not the units.That is the rule #2321 settled on (
fits,codePointEnd/codePointStart);stewardRequestand every other function are unchanged, and the older case — head kept, middle left out, end kept — still asserts the same string.After
core/test/steward.test.js: 11 tests, 11 pass, 0 fail, with the old assertion untouched.Not run:
dev/bench.sh smoke(needs the test bench checkout). This change is one pure string function; the unit suite covers it, andsrc/hostruntime behaviour is untouched.Files
node scripts/ds-bot-patches.mjs generateran once; only0001-core-fixes.patchmoved (0008andfiles/untouched).Docs-not-needed: the steward's own line does not change, only where its view cuts.
Checkpoint
{ "pr": 2548, "claim": "Dev-5", "issue": 2546, "summary": "The steward's view of a long message compared and cut by UTF-16 units: a character outside the BMP at either seam reached the model as half of a surrogate pair, and a message that fits in 1,500 characters was shortened and told it had characters left out. shorten() now measures with fits(), cuts with codePointEnd and codePointStart, and the marker counts the characters actually dropped. The test added here fails on the base and the older head/middle/end assertion is unchanged.", "files": ["core/src/host/steward.js", "core/test/steward.test.js", "changelog.d/2546-steward-shorten-chars.fixed.md", "patches/ds-bot/0001-core-fixes.patch"], "checks": { "ran": "TZ=UTC ARCH_DOCS_BASE=276042f8 npm run verify", "result": "green", "at": "0e9425e95fc80d4229c660ec289885669018ebba" }, "next": "none — ready for review" }