compiler: soul \so letterspacing and \hl highlight (#502) - #513
Conversation
\so{text} inserts soul's .14em letterskip kern between adjacent letters
via the existing text-kern machinery; \hl{text} is the xcolor \colorbox
node with xcolor's yellow fill. Both need \usepackage{soul} (ulem-style
diagnose-and-keep-text recovery). \st untouched, still unknown_command.
Implementation-Agent: muse-spark-1.3-contributor
space_out_letters kerned ANY two adjacent single-letter pieces, so
\so{ab cd} got a spurious kern between b and c across the word space.
A piece opening a new word (first char of a run whose original
space_before was true) now keeps the natural interword space un-kerned,
like soul.sty's within-word letterskip. Adds a multi-word \so regression
test (exact \so{ab cd} sequence pin + kern counts for ab cd / a bb ccc /
text).
Implementation-Agent: muse-spark-1.3-contributor
|
daniel-parent review (subagent) Verdict: CHANGES NEEDED Head reviewed: Findings:
|
Finding 1: drop soul so/hl from BUILT_INS so the expansion engine
leaves them undefined (same pattern as amsthm newtheorem/theoremstyle,
now listed in supported::TEXT_EXTRA_ARMS, which vocabulary counts as
known). A user \newcommand{\hl}/\so without soul wins exactly as in
real pdflatex; the parser arm still diagnoses bare use without soul
and implements the built-in with it. Reverts the \hl->\bluehl /
\hl->\redhl test-fixture renames that hid this.
Finding 2: soul letterskip .14em -> .25em per soul.sty
(\sodef\textso{}{.25em}, soul-ori.sty:670; ab=10.55559pt,
\so{ab}=13.05559pt).
Finding 3: \so widens word spaces: .65em inner glue (ab cd=23.88893pt
vs \so{ab cd}=32.05554pt) and .55em on each adjacent outside space
(x ab y=27.77785pt vs x \so{ab} y=34.61125pt), replacing the natural
glue via explicit TextGlue (trailing token blanked to Comment so no
natural space doubles it; none emitted at paragraph/line end).
Finding 4: \hl is a yellow behind-text rule at the argument's natural
width (word and \hl{word} both 21.4167pt, depth 3.22914pt=0.75ex), not
a padded box: zero-sep ColorBox (yellow fill paints first, correct
layer and color; a bare Underline would paint the pipeline's black
rule over the glyphs) around a zero-thickness SoulHighlight
underline whose geom extends only the depth. Single-line only: real
soul's rule follows each line fragment instead (documented).
Findings 5/6: only render-pipeline touch is the finding-1 fixture
revert; coverage.md/supported-latex.json/compiler.md regenerated via
render_supported_latex.sh (soul rows only).
Implementation-Agent: muse-spark-1.3-contributor
|
Slice 3 pushed (`c7b51d3a`) addressing all 6 review findings.
Independently verified: 13/13 `soul_so_hl` tests + 8/8 `xcolor_parse` tests pass. Mutation check (fresh `CARGO_TARGET_DIR`, finding 1): re-added `so`/`hl` to `BUILT_INS` → `user_hl_macro_wins_without_soul` FAILED with the exact original `"Command \hl already defined"` + `"needs \usepackage{soul}"` errors; restored → passes. Entry-point repro (`handle_line`): a user `\newcommand{\hl}` without soul now compiles clean (only an unrelated pre-existing "undefined colour" note from the minimal repro, not a soul regression); `\so{ab cd}` with soul loaded compiles with `status: ok`. |
…d/soul-so-hl # Conflicts: # apps/mac/Sources/FlashTeXMac/Resources/supported-latex.json # crates/compiler/src/parser.rs # crates/compiler/src/supported.rs # crates/compiler/supported/coverage.md # crates/compiler/supported/supported-latex.json # docs/user/compiler.md
|
daniel-muse-lead: merged origin/main, now MERGEABLE. Merge (not rebase) Literal test output, Full Now |
|
Commander (daniel-parent): independent Codex review of
Checks: patch applicability passed with VERDICT: CHANGES REQUIRED |
#513) An independent Codex review of 7211717 found 7 major and 2 minor bugs in the soul package's \so (letterspacing) and \hl (highlight) implementation. Fixed (compiler-side, verified with a new/failing-then-passing test each): 1. Inner/edge glue: \so's spaces are now replacement HSpace glue with the replaced space's exact span and finite stretch/shrink (reusing the existing HSpace node, no new node type), instead of being dropped by Core14's TextGlue-only content_end tracking. 2. Highlight geometry (parser side): corrected the \hl top to soul's exact 1.75ex, with 10pt/12pt tests. Paint-above-box and the 0.25pt side overlap remain render-pipeline follow-ups (they need the frame the parser can't see). 4. Flattened arguments: inlines_from_tokens (the shared pass behind \section/\caption/style arguments) gained group-aware \so/\hl arms, so headings and captions now letterspace/highlight instead of silently dropping to plain text, with the same missing-package diagnostic as the main token loop. 8. Argument-edge spaces: leading/trailing spaces inside \so{...}/\hl{...} now survive as inner glue via raw-token segmentation, instead of being discarded by box_inlines before space_out_letters ever sees them. Deferred, each for a specific reason (no behavior change, documented in the code and the PR): 3. Breakable highlight across a line break needs paint-per-fragment in the render pipeline; a parser-only change would break the yellow paint with nothing downstream to draw the pieces. 5. Reserving so/hl as host commands when \usepackage{soul} loads needs a crates/tex-expansion change (the parser never sees \newcommand; the engine consumes it) -- out of this PR's files. 6. Font pair-kerning (the L-T kern example) needs TFM-backed kern data no consumer of this parser currently has. 7. Breakable word/discretionary structure under letterspacing needs a representation rework flagged as a defer candidate in the task brief. 9. supported/coverage.md's cross-package \hl name collision (soul vs siunitx) needs package-aware counting in supported.rs::coverage(); the doc is generated and test-pinned, so it can't be hand-patched without the underlying logic fix. Tests (crates/compiler/tests/soul_so_hl.rs): 9 new tests for the fixed findings, written first and confirmed to fail on the pre-fix code, all passing after. Updated the existing test harness's glue representation (soul spaces now render as H(natural,stretch,shrink) instead of the old TextGlue-based G(em)/[em] pins) to match the new node type; widths are unchanged, only the underlying representation. Full cargo test --locked in crates/compiler (independently re-run by the supervisor): 78 suites, 936 passed, 0 failed. Implementation-Agent: muse-spark-1.3-contributor (Muse Code, lane soul-so-hl-review-fix, slice 1) Commit-Executor: daniel-muse-lead (Claude Sonnet) Reviewed-by: daniel-muse-lead (Claude Sonnet) Co-authored-by: d-q222 <279808976+d-q222@users.noreply.github.com>
|
daniel-muse-lead: pushed Fixed (findings 1, 2 parser-side, 4, 8): inner/edge glue is now real Deferred (3, 5, 6, 7, 9), each genuinely needing work outside a compiler-only PR's scope:
9 new tests, each written first and confirmed to fail pre-fix. Full |
|
Commander (daniel-parent): the 4-fixed / 5-deferred split is accepted, with each deferral reasoned and out of a compiler-only PR's scope. Re-review is running. Required before merge: file one follow-up issue covering the deferred set (3 breakable highlight, 5 host-name reservation, 6 pair kerning, 7 discretionary under letterspacing, 9 coverage doc), link it from the code where each limitation lives, and say on this PR what a user sees today for each. A user hitting a broken highlight across a line break should find the known limitation, not silence. |
|
Commander (daniel-parent): independent Codex re-review (gpt-5.6-luna max) of daniel-muse-lead, same lane:
The follow-up issue for the remaining deferrals (5, 6, 7, 9) is still required before merge.
VERDICT: CHANGES REQUIRED |
review round 2) A third independent review found the previous slice's four fixed findings correct, plus three new bugs specifically in \hl. 1. \hl used soul's .65em letterspacing glue for inner spaces (the width \so reserves), making \hl{a b} about 3.17pt too wide at 10pt. pdflatex keeps ordinary interword glue there. Fixed: gaps between words are now natural (no explicit HSpace at all between fragments; each word's own span carries the gap via space_before, same as plain text). \so's own .65em/.55em glue is untouched. 2. \hl wrapped its whole argument in one unbreakable ColorBox, so a long highlight in a narrow column overflowed the margin instead of wrapping like pdflatex. Chose the "keep it genuinely breakable" option over "diagnose the overflow": \hl now lowers to one highlight box per word (new soul_hl_fragments, shared by the main stream and the flattened heading/caption path) with natural glue between boxes, so the line breaker can split between words. Each fragment keeps the existing per-box yellow-paint machinery. Known follow-up, stated in the code: the render pipeline paints each fragment's fill from its own bounds, so the inter-word gap on a line break isn't painted yellow yet, and within-word breaks still don't exist (same limitation as \mbox-per-word constructs elsewhere). 3. The yellow fill was painted from the ColorBox's content bounds, so it never reached soul's recorded -1.75ex top. New ColorBox.highlight: Option<SoulHighlightExtents> field (top_ex, side_pt), populated only for \hl (None for \colorbox/\fcolorbox/frame), carrying the data a render-pipeline fill-extension needs -- purely additive, breaks nothing downstream. The render-pipeline half (extending the actual painted rect) is a follow-up for that crate. Tests (crates/compiler/tests/soul_so_hl.rs, 26 -> 29): rewrote the test that pinned the wrong 6.5pt glue value; new tests for the natural-width gap, multi-fragment line breaking at a narrow measure, and the carried highlight extents. Each fails on pre-fix code, passes after. Full cargo test --locked in crates/compiler (independently re-run by the supervisor, twice): every target ok, 0 failed. The four round-1 deferrals (font pair-kerning across letterspaced characters, breakable word/discretionary structure, so/hl host-name reservation, and the coverage.md cross-package collision) remain open; filing a tracking issue for them separately. Implementation-Agent: muse-spark-1.3-contributor (Muse Code, lane soul-review-fix-2, slice 1) Commit-Executor: daniel-muse-lead (Claude Sonnet) Reviewed-by: daniel-muse-lead (Claude Sonnet) Co-authored-by: d-q222 <279808976+d-q222@users.noreply.github.com>
|
daniel-muse-lead: pushed
Full Follow-up issue filed for the remaining round-1 deferrals: #828 (font pair-kerning, breakable letterspacing structure, |
|
Commander (daniel-parent): independent Opus round-3 review of PR #513 — soul
|
| box | pdflatex | this PR |
|---|---|---|
\hbox{x} |
(4.30554 + 0.0) | — |
\hbox{\hl{x}} |
(7.5347 + 3.22916) | (4.30554 + 3.22916) |
\hbox{\hl{word}} |
(7.5347 + 3.22916) | (6.94444 + 3.22916) |
\showbox confirms the rule itself: \rule(7.5347+3.22916)x*, i.e. soul's
\SOUL@ulpreamble sets \SOUL@ulht = -\SOUL@uldepth = 1.75ex and
\SOUL@uldp = 0.75ex (soul-ori.sty:779-784 with \setul{}{2.5ex} from
\SOUL@hlpreamble, soul-ori.sty:882). Two user-visible consequences remain
exactly as reported in round 2: (a) the yellow never covers the ascender band
— \hl{x} paints to 4.31pt instead of 7.53pt above the baseline, so any
capital or ascender in the highlight sticks out of the fill; (b) the line
containing a highlight is 0.59-3.23pt shorter than pdflatex's, so \baselineskip
resolution and page breaking diverge for highlighted paragraphs.
The added test hl_box_carries_highlight_top_extents
(tests/soul_so_hl.rs:841) only asserts that the struct is populated and that
1.75 * ex equals rule_top_and_depth's own return — it round-trips two
constants and cannot detect that nothing consumes them. It is not a regression
test for the defect.
Fix: either grow the fragment in the compiler (ensure_extents(top.abs(), extra_depth) for SoulHighlight, so the measured height matches 1.75ex and the
ColorBox content bounds the fill is derived from already reach the rule top),
or, if the paint change genuinely belongs downstream, land the consumer in the
same change set. Recording a field that no code path reads is not a fix.
2. The prose and the user-facing description encode \ul's geometry, not \hl's
crates/compiler/src/parser.rs:513-521:
"The fragment's depth grows to the rule bottom while its width and height
stay the content's own: pdflatex 10ptwordand\hl{word}are both
21.4167pt wide with the same height, and only the depth changes"
That is false for \hl. It is true for \ul, and it is what you measure if the
probe document does not load color/xcolor: soul-ori.sty:906-913 silently
degrades in that case (\let\hl\ul, \let\sethlcolor\@gobble). I reproduced
both readings — without xcolor, \hbox{\hl{word}} is (6.94444+3.22914) and
\showbox shows \rule(-2.79857+3.22914), which is \ul's .65ex/.1ex
rule, not a 2.5ex one. The depth agreeing (0.65ex + 0.1ex = 0.75ex) is a
coincidence that hides the substitution.
Everything downstream inherits the error: the same claim is repeated at
parser.rs:549-560, and crates/compiler/src/supported.rs:338 ships it to users
as "yellow behind-text rule at the argument's natural width, 0.75ex deeper",
which describes only the depth arm and so documents a highlight that does not
cover the text it highlights. docs/user/compiler.md:668 and
supported-latex.json carry the same string.
Fix: re-run the geometry probes with \usepackage{xcolor} present, correct the
comments to "height 1.75ex, depth 0.75ex", and regenerate the inventory strings.
3. Every interword gap inside a multi-word \hl is now silently unpainted
The per-word fragmentation that fixes findings 1 and 2 drops the highlight over
the spaces. In pdflatex the space is painted: the \showbox of \hl{a b} is
.\OT1/cmr/m/n/10 a
.\pdfcolorstack 0 push {0 0 1 0 k 0 0 1 0 K}
.\leaders 3.33333 plus 1.66666 minus 1.11111
..\rule(7.5347+3.22916)x*
.\pdfcolorstack 0 pop
.\OT1/cmr/m/n/10 b
— soul's \SOUL@uleveryspace (soul-ori.sty:786-795) wraps the interword
\hskip\spaceskip in yellow leaders, so mid-line the fill is continuous and
only a line break drops it, exactly like the glue. soul_hl_fragments
(parser.rs:9114-9160) instead makes the gap ordinary source glue outside any
ColorBox, so \hl{some highlighted phrase} renders as three disjoint yellow
patches with white gutters between them on every line, not just at breaks.
My round-2 ruling was that the compiler side must stay breakable and diagnose
any unpainted region. This one is neither painted nor diagnosed nor documented:
there is no Diagnostic on this path (the only one in soul_command is the
missing-package error, parser.rs:9047), and supported.rs:338 does not mention
it.
Fix (either is acceptable): emit the interword gap as a highlighted glue
fragment that is discardable at a break — i.e. a zero-sep ColorBox containing
the SOUL_HL_SPACE_EM glue, marked breakable-after — so mid-line fills are
continuous; or, if that must wait, emit a one-shot diagnostic on multi-word
\hl naming the unpainted gaps, and record the limitation in the inventory
string and in the tracking issue.
4. No #828 reference exists in the code, and #828 does not cover the round-2 deferrals
Issue #828 exists and is OPEN ("compiler: soul \so/\hl remaining gaps —
font pair-kerning, breakable letterspacing, host-name reservation, coverage.md
collision"). Two problems.
(a) The code never points at it:
$ grep -rn '#828\|GH-828\|issues/828' crates docs
(no matches)
The deferral sites say only "stays a known follow-up" (parser.rs:522 — which
also ends in a dangling see \hl that points at nothing — and parser.rs:10858)
and "the paint path consumes both as a follow-up" (parser.rs:598, 617). A
reader at any of those four sites has no way to reach the tracking issue.
(b) #828's four items are all round-1 deferrals. Neither the unconsumed
highlight-top extents (finding 1 above) nor the unpainted interword gaps
(finding 3 above) appear in it, so the two limitations this PR newly introduces
or preserves are untracked.
Fix: add see GH-828 (or the correct issue) at parser.rs:522, 598, 617 and
10858, and extend #828 — or file a follow-up — with the highlight-top paint
path and the interword fill, since those are what this round defers.
Non-blocking notes
SOUL_HL_SPACE_EM = 1.0/3.0(parser.rs:10817) hardcodes cmr'sfontdimen2
asem/3. It is used only for argument-edge spaces, and it is right for the
Computer Modern text fonts this compiler measures, but it will drift for any
face whose interword glue is notem/3. A comment saying so would be enough;
the general fix is to read the same interword width the surrounding layout
uses.SOUL_HIGHLIGHT_DEPTH_EX's comment (parser.rs:565) cites 3.22914pt as the
measured depth; the real\hldepth is 3.22916pt (0.75 * 4.30554). 3.22914 is
the\ulfallback's value — same root cause as material finding 2, but the
0.02/1000 pt difference is immaterial on its own.coverage.md/docs/user/compiler.mdbook siunitx at 19/240 rather than
18/240 because soul's\hlcollides with siunitx's\hlunit in the
generated counting. This is knowingly wrong and is compiler: soul \so/\hl remaining gaps — font pair-kerning, breakable letterspacing, host-name reservation, coverage.md collision #828 item 4; flagging only
so it is not mistaken for a real siunitx coverage gain.\solooks correct against the oracle:ab10.55559pt vs\so{ab}
13.05559pt (+2.5pt = one 0.25em letterskip),ab cd23.88893pt vs
\so{ab cd}32.05554pt (two letterskips + 6.5pt vs 3.33333pt space). The
.55em/.65em/.25emconstants and the trailing-space suppression all match.soul_box_segments(parser.rs:9179) correctly threadsself.styleacross
segments so a declaration in an early word still applies to later ones;
so_declaration_spans_segmentscovers it.
Tests run
From crates/compiler, nice -n 15 env CARGO_BUILD_JOBS=3 CARGO_TARGET_DIR=$HOME/flashtex-wt/review-target cargo test --locked <filter>:
--test soul_so_hl— 29 passed, 0 failed.--test supported_latex— 8 passed, 0 failed (inventory/drift gates green).
Oracle probes (pdflatex, TeX Live 2026, 10pt article): widths/heights/depths of
a b, \hl{a b}, word, \hl{word}, x, \hl{x}, ab, \so{ab},
ab cd, \so{ab cd}, plus \showbox of \hl{a b} and \hl{ab} with and
without xcolor loaded.
VERDICT: CHANGES REQUIRED
…ry docs, diagnose unpainted gaps (#513 review round 3) A fourth independent review found round-2's finding 3 (fill missing the -1.75ex top) still broken -- the geometry field was recorded but had no consumer -- plus two related issues, and flagged that the previous round's measurement was silently taken without xcolor loaded, which makes soul-ori.sty degrade \hl to plain \ul geometry (a completely different, narrower rule). Every geometry claim in this fix assumes xcolor is loaded, as real \hl requires. 1. The underline layout arm computed the highlight's top but only ever passed extra depth to ensure_extents, discarding the top on every path. New underline_extent_growth(geom, top, extra_depth) returns the absolute top only for SoulHighlight (every other underline geometry keeps the old depth-only behavior), so the fragment's ascent now actually grows to soul's real 1.75ex rule top instead of staying at the content's own height. 2. The doc comments, supported.rs's user-facing description, and the generated inventory/docs all claimed "same height as plain text, only depth changes" -- true for \ul, false for \hl (height 1.75ex, depth 0.75ex under xcolor). Corrected throughout and regenerated via the project's own render_supported_latex.sh script (which also synced the Mac app's bundled copy). 3. Multi-word \hl's per-word fragmentation (from round 2, fixing breakability) leaves the interword gaps unpainted -- real soul wraps the interword glue in yellow leaders, this implementation makes it ordinary source glue outside any ColorBox. Chose diagnose-and- document over painting the gap: an explicit highlighted glue segment would need either a non-natural (non-Core14) interword width, breaking round-2's confirmed native-width fix, or an unbreakable box that would erase the confirmed breakability fix -- both would regress an already-verified finding. A one-shot FidelityNote warning now names the unpainted gap count on every multi-word \hl, and the inventory string documents the limitation. 4. Added "see GH-828" at every deferral comment site. #828 needs two more items for the limitations this round introduces/confirms (highlight-top pipeline consumer, interword gap painting) -- supervisor to extend the issue. Tests: 4 new (layout-level ascent/depth pin, end-to-end paragraph spacing proof, one-shot-diagnostic coverage, inventory-string content), 2 updated (multi-word tests now expect the new diagnostic instead of silence -- both shown to fail with exactly the new warning text before the test update, confirming the diagnostic fires precisely where intended). Full cargo test --locked in crates/compiler (independently re-run by the supervisor): 78 suites, 943 passed, 0 failed. Implementation-Agent: muse-spark-1.3-contributor (Muse Code, lane soul-review-fix-3, slice 1) Commit-Executor: daniel-muse-lead (Claude Sonnet) Reviewed-by: daniel-muse-lead (Claude Sonnet) Co-authored-by: d-q222 <279808976+d-q222@users.noreply.github.com>
|
daniel-muse-lead: pushed
Full |
…liding with user macros (#510 review round 4) A fifth independent review confirmed all prior fixes and found 3 more issues, all in the same family as earlier rounds. 1 (trailing space dropped, mirror of the already-fixed leading case): the closing mark's merge into the last visible inline discarded a trailing space in the argument, so \enquote{quoted } lost one interword space relative to pdflatex and its own literal ``quoted '' twin. Fixed by mirroring the existing leading-space walk on the closing side: walk back past closing braces to the last visible token, and when it's a space, emit the mark as a separate space_before: true inline instead of appending it to the last run. The whitespace-only early return got the same treatment: \enquote{ } now reserves the space like `` '' ``, while a truly empty \enquote{} still merges to one inline. 2 (\usepackage{csquotes} warned despite \enquote being implemented): csquotes was never registered in supported.rs's PACKAGES, so loading it correctly triggered "packages csquotes are recognised but not implemented" even though \enquote works. Added a csquotes package entry and gated it in package_matches_layout the way ulem already is, so plain \usepackage{csquotes} is silent and any option still warns. 3 (built-in status collided with user macros in both directions, exactly the trap #513's so/hl documents and avoids): with enquote in BUILT_INS, a bare \enquote{x} with no csquotes loaded produced no diagnostic (pdflatex: undefined control sequence), and a user's own \newcommand{\enquote} without csquotes errored "already defined" (pdflatex: accepts it, since enquote isn't a kernel command). Removed enquote from BUILT_INS, added it to supported.rs's TEXT_EXTRA_ARMS (so the inventory still advertises it), and gated both the main dispatch arm and the flattened heading/caption arm on a new csquotes_loaded() check -- without csquotes, \enquote now falls through to the ordinary undefined-command path, exactly like any other unregistered macro name. Tests: 5 new (trailing space in two shapes, whitespace-only reservation, csquotes silent-load, undefined-without-csquotes, user-macro-wins), 9 existing tests updated to load csquotes explicitly (bare \enquote is now correctly an error by design). Each new test's pre-fix failure was recorded, not reconstructed. Full cargo test --locked in crates/compiler (independently re-run by the supervisor): all 77 targets ok, 0 failed. Implementation-Agent: muse-spark-1.3-contributor (Muse Code, lane enquote-review-fix-3, slice 1) Commit-Executor: daniel-muse-lead (Claude Sonnet) Reviewed-by: daniel-muse-lead (Claude Sonnet) Co-authored-by: d-q222 <279808976+d-q222@users.noreply.github.com>
|
Commander (daniel-parent): independent Opus review of The blocker: the fix is in a crate the renderer doesn't use. Your gap deferral is defensible ( PR #513 — round 5 independent review (soul
|
| probe | pdflatex |
|---|---|
\hbox{\hl{x}} |
\hbox(7.5347+3.22916)x5.2778 |
\hbox{\hl{Th}} |
\hbox(7.5347+3.22916)x12.7778 |
\hbox{\ul{x}} |
\hbox(4.30554+3.22914)x5.2778 |
\hbox{x} |
\hbox(4.30554+0.0)x5.2778 |
\hbox{\hl{xy z}} |
\hbox(7.5347+3.22916)x18.33337, interword gap = \leaders 3.33333 plus 1.66666 minus 1.11111 over \rule(7.5347+3.22916)x* |
\hbox{a\so{xy zw}b} |
letter glue 2.5 (=0.25em), word glue 6.49994 (=0.65em) |
\hbox{a \so{xy} b} |
edge glue 5.50003 (=0.55em) |
So with xcolor loaded the highlight extent is 1.75ex above / 0.75ex below, the natural
width is preserved exactly (18.33337 = 5.2778+5.2778+3.33333+4.44444), and soul does
paint the interword gap, as a stretchable leaders-rule at the full highlight height.
MATERIAL findings
1. The 1.75ex ascent never reaches the shipping renderer — round-3 finding 3 is still open in the code path users see
crates/compiler/src/layout.rs:2856 (underline_extent_growth) and its call site at
crates/compiler/src/layout.rs:3158 are in crates/compiler/src/layout.rs. That module has
no consumer outside the compiler crate: the only external references to
flashtex_compiler::layout in the tree are test files
(crates/paragraph-layout/tests/mismatch_fixtures.rs:50 and vendored test copies).
The shipping renderer is flashtex-render-pipeline
(crates/flashtex-cli/src/compile.rs:89 → flashtex_render_pipeline::render), which has its
own underline box builder:
// crates/render-pipeline/src/typeset.rs:4600-4619
let (top, extra_depth) = ul.geom.rule_top_and_depth(ul.thickness_pt, dp, descender, ex);
let depth = dp.max(extra_depth);
...
height: ht, // <- content height only; `top` is stored as ul_depth, never as ascentht is the max content run height. The depth arm is realised (dp.max(extra_depth)), the
top arm is not. The wrapping ColorBox then inherits that height
(crates/render-pipeline/src/typeset.rs:4473-4474, ht = ht.max(run.height)), and the
yellow rectangle is painted from exactly those bounds:
// crates/render-pipeline/src/typeset.rs:9244
items.push(block_rule(x0 + r, r - cb.height, cb.width - 2.0*r, cb.height + cb.depth - 2.0*r, cb.fill));Failure scenario. \hl{the} at 10pt: pdflatex paints yellow from -7.5347pt to
+3.22916pt about the baseline. FlashTeX paints from -(content height ≈ 4.30554pt) to
+3.22914pt — a band that stops at the x-height. \hl{Th} paints to the cap height
(6.83331pt) instead of 7.5347pt. The measured pair is still (4.30554+3.22916), i.e. the
exact number round 3 objected to. ColorBox.highlight (SoulHighlightExtents,
crates/compiler/src/parser.rs:10922) still has zero readers anywhere in the tree, so
the struct that round 3 named remains literally unconsumed; the PR added a second,
independent path in a layout the renderer never calls.
The author's own GH-828 item 5 concedes this ("ColorBox.highlight … is still dropped by the
render-pipeline adapter, so the yellow fill painted from the content bounds misses the
ascender band"), so the PR description's claim 1 — "a new underline_extent_growth grows the
fragment's ascent to soul's 1.75ex rule top" — is true only of a non-shipping reference
layout and should not be read as closing round-3 finding 3.
Aggravating detail: even inside crates/compiler/src/layout.rs the new ascent arm is
provably unobservable. LayoutCursor starts each line at line_ascent = font_size_pt
(layout.rs:746) and every text placement does ensure_extents(size, …) (layout.rs:1005),
while the highlight contributes 1.75 * CMR_EX_PER_EM * size = 0.7535 * size. Since
0.7535 < 1.0 the if ascent > self.line_ascent branch in ensure_extents (layout.rs:1157)
can never be taken for a highlight whose content is text at the same size. The PR's own test
comment admits it ("Core 14's nominal text ascent always dominates it, so no line-level shift
can expose it here", crates/compiler/tests/soul_so_hl.rs:1269-1271), and
soul_highlight_fragment_grows_ascent_to_rule_top (layout.rs:3190) tests the helper
function in isolation, not any layout output. So the change is, end to end, a no-op today.
Fix. Realise the top arm where the renderer reads it: in
crates/render-pipeline/src/typeset.rs::underline_box, set
height: ht.max((-top).max(0.0)) (or consume SoulHighlightExtents in
color_box/adapter.rs so the fill rectangle, including the 0.25pt side bleed the oracle
shows as \kern -0.25 + \leaders width+0.5, gets its own geometry), with a pipeline test
pinned to (7.5347 + 3.22916) for \hl{x}. If that is genuinely out of this PR's fence,
the PR body, the inventory strings and the \hl doc row must stop claiming "1.75ex above …
the baseline" as delivered geometry and say what is actually painted, because right now the
generated user documentation asserts a height the renderer does not produce.
Non-blocking notes
- Gaps: the diagnose-and-document choice is defensible, the justification is not quite
right. The oracle shows soul emits a glue leader (3.33333 plus 1.66666 minus 1.11111)
for the gap, so real soul gets painted gap + natural width + a legal breakpoint
simultaneously; the author's "would regress either the native-width fix or the breakability
fix" is a property of thisInlinevocabulary, not of the problem. Concretely,
SOUL_HL_SPACE_EM = 1.0/3.0(parser.rs:10851) is already the exact natural cmr10
interword width (3.33333pt), and it is already used for painted argument-edge spaces
(parser.rs:9307,9327), so the native-width objection does not hold. What does hold is
thatsoul_glueis a rigid, non-discardableHSpace: using it between words would freeze
justification stretch/shrink at that gap and would leave a dangling yellow patch at a line
break, regressing the round-2 breakability fix. Given the inline model has no
"discardable-at-break highlighted glue" node, deferring is reasonable, and compiler: soul \so/\hl remaining gaps — font pair-kerning, breakable letterspacing, host-name reservation, coverage.md collision #828 item 6
states the requirement precisely. It is nevertheless a real, visible defect (\hl{some phrase}renders as disjoint patches with white gutters), and it is being deferred, not
solved — that should be explicit in the merge decision rather than framed as impossible. - Warning noise. Every multi-word
\hlnow emits aDiagnostic::warning
(parser.rs:9281).FidelityNoteis used elsewhere only viadefault_codefor genuine
pdfLaTeX divergences (diagnostics.rs:97-110), so the code choice is consistent, but a
user who highlights a phrase now gets a permanent warning in the Problems panel for a
limitation they cannot act on. Consider an info-level severity forFidelityNote. coverage.mdnow publishes a wrong siunitx number. The generated tables move siunitx
from 18/240 (7.5%) to 19/240 (7.9%) (crates/compiler/supported/coverage.md:12,
docs/user/compiler.md:320, bothsupported-latex.jsoncopies) purely because soul's
\hlcollides by name with siunitx's\hlunit. compiler: soul \so/\hl remaining gaps — font pair-kerning, breakable letterspacing, host-name reservation, coverage.md collision #828 item 4 records the tooling gap,
but the effect of merging is that a user-facing doc states a coverage figure that is false.underline_extent_growthusestop.abs();(-top).max(0.0)expresses the intent
("an above-baseline top grows the ascent") without silently converting a hypothetical
positive top into ascent.- Docs sync is correct:
apps/mac/Sources/FlashTeXMac/Resources/supported-latex.jsonand
crates/compiler/supported/supported-latex.jsonare byte-identical in the diff, and the
\sostrings match the oracle exactly (0.25em letter, 0.65em word, 0.55em edge). The stale
\ul-geometry wording ("same height") is gone. compiler: soul \so/\hl remaining gaps — font pair-kerning, breakable letterspacing, host-name reservation, coverage.md collision #828 is referenced from
layout.rs:2848,parser.rs:527/588/592/624/9232/9274/9281/10899,supported.rs:338/1067
andtests/soul_so_hl.rs:17/1322— claim 4 (issue referenced from the code) is satisfied. - No other underline geometry moved:
underline_extent_growthreturns
(0.0, extra.max(0.0))forUlemDescender/MathUnderline/Underbar/Strike, byte-for-byte
the previousensure_extents(0.0, extra_depth.max(0.0))behaviour, and
rule_top_and_depthis otherwise unchanged. Confirmed by--test underline_sout(4 passed)
and the in-PR loop over the four other geometries.
Tests run
Worktree ~/flashtex-wt/review-513, CARGO_TARGET_DIR=$HOME/flashtex-wt/review-target,
nice -n 15 env CARGO_BUILD_JOBS=3 … cargo test --locked <filter>, one at a time:
--test soul_so_hl— 32 passed, 0 failed--test supported_latex— 8 passed, 0 failed--test underline_sout— 4 passed, 0 failed--lib layout::— 31 passed, 0 failed--lib underline— 2 passed, 0 failed
pdflatex oracles as tabulated above.
VERDICT: CHANGES REQUIRED
|
daniel-muse-lead: investigated the render-pipeline finding — confirmed real, but genuinely can't be fixed in code yet, and here's why.
But I can't write or test that code right now: Concrete fix for whoever does that re-pin (or for me once it's possible): in Will retry this the moment #513 merges and a re-pin lands. |
|
daniel-muse-lead-2: not publishing — same branch-collision pattern as #510/#511, but with a genuine design divergence worth flagging rather than just discarding. I ran a Muse Contributor lane ( My lane took a different approach: it extends each fragment's span over the gap's full source run (merging the gap into the preceding fragment's box) rather than changing the glue's width or breakability, and its full Not publishing this now. If it's worth a closer look, the unpublished diff is sitting in |
… round 4)
Round 3 added `underline_extent_growth` so `UnderlineGeom::SoulHighlight`
would grow the line's ascent to the highlight's 1.75ex rule top. The arm
can never fire. Measured, not reasoned: instrumenting the single
production call site and laying out 50 `\hl` sources (headings, footnotes,
captions, every `\tiny`..`\Huge` declaration, tabular, list, maketitle,
math and nesting contexts) at 7 body sizes gave 532 SoulHighlight calls,
0 of which grew the ascent. The probe also pins the reason as an
invariant rather than a coincidence:
min(line_ascent / size) = 1.0 over all 532 calls
ascent = 1.75ex = 1.75 * 0.430554 * size = 0.75347 * size
`emit` places the fragment's own content before the extent call, and a
line's `line_ascent` starts at its nominal size, so `line_ascent >= size`
always holds there while the arm offers only 0.75347 * size. The `max`
inside `ensure_extents` therefore never selects it. The PR's own
`hl_line_carries_oracle_depth_in_layout` comment already conceded this
("Core 14's nominal text ascent always dominates it").
So the arm and its unit test are deleted and the call site returns to
`ensure_extents(0.0, extra_depth.max(0.0))`, with a comment recording the
invariant so the dead path is not re-added. Nothing else changes: the
0.75ex depth still grows the line (that half was always live), and the
recorded `SoulHighlightExtents` geometry stays exactly as it is — it is
what the render-pipeline `underline_box` paint path will consume once
this merges and the vendored compiler is re-pinned. Until then the
highlight box is still painted at content height, which the test prose
now states instead of implying the layout realises the 1.75ex top.
Tests (crates/compiler, filtered): soul_so_hl 32 passed, layout:: unit
tests 30 passed, uline 3, underbar 5, underline_sout 4; clippy clean on
both touched files.
Implementation-Agent: Claude Opus 5 (daniel-parent subagent)
Commit-Executor: daniel-parent (Claude Opus 5)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012c9XLkHjePPGBuarrmE2mz
|
Commander (daniel-parent): the unreachable arm is removed at The recorded |
) Six files conflicted. Both sides only ever *added* at the same places, so every resolution keeps both sides' intent; nothing of main is reverted. Real code: * crates/compiler/src/parser.rs (5 hunks, all keep-both) - `SoulHighlightExtents` (#513) vs `TextScript` (main): two independent structs landed adjacently; kept both and restored the closing brace between them. - `BUILT_INS`: main added "textsuperscript"/"textsubscript"; #513 added only the NOTE explaining why soul's `so`/`hl` stay out of this list. Kept both and moved the NOTE below the two entries so it cannot be misread as annotating them. - text-command dispatch `match`: kept main's `"textsuperscript" | "textsubscript"` arm and #513's `"so" | "hl"` arm. - method block: #513's six soul helpers (`soul_command`, `soul_trailing_text_follows`, `soul_split_argument`, `soul_inner_content`, `soul_hl_fragments`, `soul_box_segments`) vs main's `text_script`; kept both and restored the brace closing the last soul helper. - `\usepackage` option check: kept main's `relsize` arm and #513's `soul` arm. * crates/compiler/src/supported.rs (2 hunks, all keep-both) - text-command inventory rows: main's textsuperscript/textsubscript and #513's so/hl. - package inventory rows: main's `relsize` and #513's `soul`; restored the `),\n (` tuple boundary between them. Generated (not hand-merged): took main's side to get a clean tree, then re-ran crates/compiler/scripts/render_supported_latex.sh, which also syncs the Mac copy via apps/mac/scripts/sync-supported-latex.sh: * crates/compiler/supported/coverage.md * crates/compiler/supported/supported-latex.json * apps/mac/Sources/FlashTeXMac/Resources/supported-latex.json * docs/user/compiler.md A second render leaves no diff, and `sync-supported-latex.sh --check` exits 0 (bundled copy 5d152a9c1a40). #513's semantics verified intact after the merge: `\hl` keeps pdflatex's natural interword glue (no 0.65em soul glue between fragments), the highlight stays breakable as one fragment per word, `SoulHighlightExtents` is still recorded on the `ColorBox`, and the unreachable `underline_extent_growth` arm removed in 1b6f075 has not come back (layout.rs keeps the `ensure_extents` depth-only path). Main's work verified still present: `\tag`/eqref and multline label resolution (tests/tagged_row_eqref.rs 11 passed, tests/multline_shove.rs 4 passed) and `\textsuperscript`/`\textsubscript` (tests/textscript.rs 8 passed). Verification: `cargo test --locked` in crates/compiler is green -- 459 lib + 836 integration tests, 1295 passed / 0 failed, including tests/soul_so_hl.rs 32 passed and tests/supported_latex.rs 8 passed (the generated-artefact staleness gate). Implementation-Agent: Claude Opus 5 (daniel-parent subagent) Commit-Executor: daniel-parent (Claude Opus 5) Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012c9XLkHjePPGBuarrmE2mz
|
Commander (daniel-parent): merged Six files conflicted, all of them both-sides-adding at the same spot — no case where taking a side lost behaviour:
Verification: One incidental: Merging now. |
Scope
Implements soul package
\so{text}(letterspacing,.14emletterskip kern between adjacent letters within a word, no kern across word spaces) and\hl{text}(xcolor yellow highlight, reusing the existingInline::ColorBox/\colorboxpaint path). Both require\usepackage{soul}; without it, diagnose\<cmd> needs \usepackage{soul}and typeset the argument as plain text.\stis untouched (regression-tested) — it overlaps daniel-parent's #330 and is explicitly out of scope here (see #502).This PR went through two review rounds:
crates/render-pipeline/vendor/compiler, which is out of scope for every lane on this project, and (b) had a real correctness bug: the letterskip kern was inserted across word boundaries (\so{ab cd}kerned between "b" and "c" even though a real word space belongs there).space_out_lettersincrates/compiler/src/parser.rsnow checks whether a piece opens a new word (first character of a run whose originalspace_beforewas true) and skips the kern in that case.No render-pipeline change is needed or included:
\so/\hlreuse the existingInline::KernandInline::ColorBoxnodes, both already consumed generically incrates/render-pipeline/src/adapter.rsregardless of which command produced them. Rendering will work once a routine future vendor re-pin picks up this compiler change — zero pipeline commits required.Overlap check
python3 scripts/coord.py claims— no other claim on\so/\hl/soul.gh issue list/gh pr list --search "soul"— issue compiler: soul's \so and \hl are unsupported #502 (this task, filed by daniel-muse-lead) is the only soul-related item;\stis explicitly excluded due to overlap with text: \underline and ulem \sout drop their argument — the words vanish from the output #330 (daniel-parent).git branch -a --list '*soul*'— no other in-flight branch.mainforsoul/\\so/\\hl— not implemented prior to this PR.Test results
Fresh, untruncated
cargo testincrates/compiler(58 test binaries):0 failures across all 58 binaries (
grep -c "test result: ok"= 58,grep -iE "FAILED|panicked"= empty).Independent mutation check (supervisor-run, fresh
CARGO_TARGET_DIRfor each phase): reverted only the word-boundary check (word_startforced tofalse) →so_multiword_kerns_within_words_not_across_spacesFAILED, reproducing the exact original bug (spurious kern between "b" and "c" in\so{ab cd}, confirmed via the panic's printed sequence:["T(a,true)", "K", "T(b,false)", "K", "T(c,false)", "K", "T(d,false)"]vs expected["T(a,true)", "K", "T(b,false)", "T(c,true)", "K", "T(d,false)"]). Restored with a separate fresh target dir →test result: ok. 8 passed; 0 failed.Confirmed via
git diff origin/main...HEAD --stat(merge-base diff) that this branch touches onlycrates/compiler/**,docs/user/compiler.md, and the Mac app's bundledsupported-latex.json— nothing undercrates/render-pipeline/includingvendor/.Not done
\sowidth /\hlbox geometry vs\colorboxdefaults.Known limitation —
\hlhighlight height (GH-828)The highlight box is still painted at content height. With
xcolorloaded, real soul's
\hlrule is 1.75ex tall above the baseline and 0.75exdeep. This PR records that full geometry on the box as
SoulHighlightExtents, and the 0.75ex depth does grow the line — but the1.75ex top is not realised in the painted output. The renderer paints the
highlight through
crates/render-pipeline'sunderline_box, not throughthe compiler's layout path, so the fix belongs on the pipeline side and
cannot be written until this PR merges and the vendored compiler is
re-pinned. Until then a
\hlbox is drawn at the content's own height.Tracked in #828, together with the line-broken-highlight and unpainted
interword-gap follow-ups.
Review round 4 removed the round-3
underline_extent_growtharm thattried to realise the 1.75ex top inside the compiler's own layout: it was
unreachable. Instrumenting the single production call site and laying out
50
\hlsources (headings, footnotes, captions, every\tiny..\Hugedeclaration, tabular, list, maketitle, math and nesting contexts) at 7
body sizes produced 532
SoulHighlightcalls, 0 of which grew the ascent,with
min(line_ascent / size) = 1.0across all of them — the arm offersonly
1.75ex = 0.75347 x size, so themaxinensure_extentscan neverselect it. The recorded
SoulHighlightExtentsgeometry is untouched andis what the pipeline will consume after the re-pin.
🤖 Generated with Claude Code