compiler: the full expansion stops on the output token limit; 18502's peak back to 1746 MB (stacked on #610) by d-q222 · Pull Request #619 · flash-tex/flashtex

d-q222 · 2026-09-15T11:21:48Z

Stacked on #610 (agent/daniel-parent/stop-single-run), which is stacked on #601, #592 and #577. Review after them. Until they merge, the diff against main includes the whole stack; this PR's own commits are the ones after adc2c67.

What

Two changes to crates/compiler/src/expansion.rs, both follow-ups from #610's "Not done" list.

1. The full expansion stops on the output token limit

expand_project now enforces max_output_tokens, the same way the incremental expander does:

It counts the engine tokens it takes, including those that \input/\includeonly argument reads take.
Once the count goes past the limit, it stops with the expander's output token limit exceeded diagnostic (synthetic span). This happens after it converts the token that went past the limit, as IncrementalExpander's run loop does.
There is no unexpanded recovery after this stop. The cached path never had one.
Reworded (second commit, b6a7e6b, daniel-parent's decision): the diagnostic was output token limit exceeded, with the note "...the rest of the document was typeset without macro expansion", which neither path does. It is now TeX capacity exceeded, sorry [output token limit=N]; expansion stopped here and the rest of the document was not typeset., with the note "stopped expanding; the rest of the document was not typeset".
- One definition: tex::output_limit_message / tex::is_output_limit, used by IncrementalExpander, Engine::run and expand_project.
- Because the new text starts with TeX's capacity prefix, the unexpanded recovery excludes it explicitly. The diagnostic flood cap still always keeps it.
- The step limit's text is unchanged, because that stop does typeset the rest unexpanded.
Nesting-limit notes (third commit, c8ffe67): group nesting limit exceeded and conditional nesting limit exceeded also had the "typeset without macro expansion" note, but the engine keeps expanding after both.
- For a group, maybe_handle_brace drops the extra { without opening a group.
- For a conditional, do_conditional returns before pushing it, so its test is read as text.
- The notes are now "the extra group was ignored and expansion continued" and "the extra conditional was ignored without evaluating its test, and expansion continued".
- The messages themselves are unchanged. TeX's capacity prefix would make them stops.
- New tests pin the engine behaviour and all three expansion-limit notes.

Before this change, only the incremental path had this limit. So the same output-heavy loop behaved differently depending on the document's size:

A document of 4 KB or more (cached path) stopped at the output limit.
A smaller document (full path only) ran on to the step limit and typeset the rest unexpanded.

With #610's repro (\def\o{xyzw xyzw xyzw\o}\o inserted into the 20 KB test document), both paths now give 764,343 tokens. Before, the full path gave 799,804.

The debug-build parity check in expand_project_with_cache now also compares output-limit stops against expand_project, not only step-limit and capacity stops.

2. Compile-wide memory on fuzz case 18502

#610's regression was not retained or duplicated data:

The recovery tail after the stop is 17 tokens.
Heap counts (a counting global allocator in a probe binary) show the cache holds the same 1242 MB after the compile, both on tex-expansion: incremental re-expansion equals a full run across step/output-limit stops (stacked on #592) #601's head and on compiler, tex-expansion: a runaway document is expanded once per compile (stacked on #601) #610's head.
The whole-compile heap peak actually fell: 2423 MB → 2220 MB.

The extra footprint came from freed memory the allocator kept resident. The converted stream (4.34M tokens × 96 B) grew by doubling to an 8.39M-token capacity. The chain of freed blocks from that growth stayed resident through parse and layout:

In the probe, compiler, tex-expansion: a runaway document is expanded once per compile (stacked on #601) #610's footprint after the compile (proc_pid_rusage) was 1231 MB, against 1242 MB of live heap.
On tex-expansion: incremental re-expansion equals a full run across step/output-limit stops (stacked on #592) #601 it was 777 MB. There the full run's second stream reused those blocks.

The fix:

build_cache reserves the stream once, for the engine token count plus the bytes of the unexpanded rest.
finish trims a stream with more than 1/4 spare capacity down to 1/8 spare. This applies once after a cold build: on 500 KB, 184K tokens in a 475K-token reservation. Later edits never trimmed in the bench: no trim or stream clone during 500KB editing (probed).

Why

Item 1: daniel-parent's decision on compiler, tex-expansion: a runaway document is expanded once per compile (stacked on #601) #610's open question. The output token limit is a capacity guard, so clean and incremental output must be identical, including the diagnostic and where the run stops.
Item 2: a 400 MB peak regression on a runaway fuzz input.

Before → after (release, same machine, interleaved runs, load average 7–20 on 18 cores)

"#601" is 1e4bb85, "#610" is adc2c67 (this PR's base), "after" is this branch.

Fuzz case 18502, fuzz_compile --diagnostics (3 interleaved runs each):

	#601	#610	after
peak memory footprint	1744 / 1745 / 1744 MB	2146 / 2041 / 2146 MB	1746 / 1746 / 1746 MB
max RSS	1799 / 1800 / 1799 MB	2146 / 2042 / 2147 MB	1747 / 1747 / 1747 MB
instructions retired	75.25 G	48.33 / 48.50 / 48.33 G	47.94 / 47.94 / 47.89 G
wall time	2697 / 2728 / 2674 ms	1750 / 1821 / 1739 ms	1772 / 1699 / 1694 ms

With --digest, which also formats all 827 pages, the peak footprint goes from 2146–2150 MB (#610) to 1801 / 1802 / 1938 MB.

Heap probe on 18502, stage by stage (counting allocator; footprint from proc_pid_rusage):

stage	#601 heap peak	#610 heap peak	after heap peak	#601 / #610 / after lifetime max footprint
parse, including expansion	2423 MB	1644 MB	1334 MB	1662 / 1602 / 1221 MB
layout	2220 MB	2220 MB	1910 MB	1662 / 2047 / 1664 MB
retained after the compile (heap)	1242 MB	1242 MB	932 MB

edit_latency_bench, normal edits, session p50 in ms:

Medians of paired interleaved runs: 10 pairs for 500KB, 5 for HW1.
Machine load was 7–20, so single runs vary by ±20%.

scenario	#610	after	median paired ratio
HW1: type in paragraph	0.794	0.801	0.981
HW1: type in inline math	0.812	0.802	0.988
HW1: type in display eq	0.836	0.794	0.923
HW1: delete/restore line	0.806	0.777	0.963
500KB: type in paragraph	22.104	20.807	0.976
500KB: type in inline math	22.240	22.126	0.986
500KB: type in display eq	23.007	23.675	1.007
500KB: delete/restore line	27.159	28.663	1.059

The 500KB delete/restore timing is within the noise of this machine. Instructions retired for that scenario alone (--only "500KB: delete", whole process, 2 runs each) are the same or slightly lower:

delete/restore line: 111.86 / 111.91 G (compiler, tex-expansion: a runaway document is expanded once per compile (stacked on #601) #610) and 111.81 / 111.80 G (after).
type in paragraph: 101.95 / 102.10 G (compiler, tex-expansion: a runaway document is expanded once per compile (stacked on #601) #610) and 102.09 / 102.08 G (after).

Per edit, the code path is unchanged apart from one capacity comparison.

Identity

fuzz_compile --digest, all 419 seeds: identical to compiler, tex-expansion: a runaway document is expanded once per compile (stacked on #601) #610's head (shasum 36b592ea57085d703ffea84d4b6979c628dbf236 for both, the same value compiler, tex-expansion: a runaway document is expanded once per compile (stacked on #601) #610 reported). No seed reaches the output token limit on the full path, so item 1 changes none of them.
edit_latency_bench reply sha256: identical in every run, both builds, all 12 scenarios. For example 257e6ff491b4534b, c300dfedc507d109, and the HW1 runaway scenarios 2826b4f5b8aeaa3a and 446dc408ee055be5.
Runaway inputs (--diagnostics --digest, page and diagnostic hashes): identical on compiler, tex-expansion: a runaway document is expanded once per compile (stacked on #601) #610 and after for 10082, 18502, bodyloop.tex (an output-limit stop through the cache), endloop.tex, endloop1k.tex, endshort.tex, envloop.tex, conds.tex, groups.tex, missing.tex, n.tex, distinct.tex and section.tex.
Intended change from the rewording: the message text of every output-limit stop. Pages are unchanged. For bodyloop.tex: pages 163c9d2a21b1822b stay the same, diagnostics go from d82176b9e51d82eb to 856cb57fcc2fede2. No fuzz seed hits this limit; the 419-seed digest is still 36b592ea57085d703ffea84d4b6979c628dbf236 after b6a7e6b.
Intended change from the nesting notes: pages are unchanged; diagnostics hashes change for 10082 (9232f3ba710cfc05 to f0c08aabd804800a), conds.tex (f0075201d8ee58ff to 1d31e6378d071330) and groups.tex (78f34e551c859d61 to cd117cd5038b2f6c). The 419-seed digest is unchanged after c8ffe67.
Intended change from item 1: a document under 4 KB whose loop reaches the output limit before the step limit. Repro: 2.4 KB, 40 comment lines of padding, then \def\o{ + ab ×100 + \o}\o in the body.
- compiler, tex-expansion: a runaway document is expanded once per compile (stacked on #601) #610: 478 pages, 1 diagnostic (expansion step limit exceeded, rest typeset unexpanded).
- After: 468 pages, 2 diagnostics (the output-limit stop, unterminated environment 'document').
- The same file padded to 4.5 KB takes the cached path. It gives the output-limit stop + unterminated environment 'document' on both builds. Pages d8ced1b6721700a2 are the same; diagnostics were c7d1e682dcbd809e before the rewording.

Overlap check

gh pr list --state open --search "output token limit" and --search expansion: only this stack (compiler: mutation fuzzer + 13 robustness fixes (panics, stack overflows, hangs, OOM) #577, compiler, tex-expansion: bound diagnostic floods from runaway loops (stacked on #577) #592, tex-expansion: incremental re-expansion equals a full run across step/output-limit stops (stacked on #592) #601, compiler, tex-expansion: a runaway document is expanded once per compile (stacked on #601) #610) touches expansion.rs's cache or the expander's limits. compiler: \ifthenelse (ifthen package) is unsupported (#521) #550, compiler: \fnsymbol{footnote} inside \footnote errors (#564) #572, compiler: punctuation after a style group survives macro expansion (GH-MACRO-BODY-COMMA) #576, compiler: em/ex follow the active font, \hspace keeps its trailing space, \global length prefixes #587, tex-expansion: registers::scale_decimal loses precision vs. pdftex (GH-SCALE-DECIMAL-EXACT) #600 and compiler: \newcounter{c}[within] must recognize kernel counters (GH-NEWCOUNTER-WITHIN, #594) #605 change individual commands.
coord.py claims: GH-EXPANSION-TOKEN-LIMIT-PARITY is held by daniel-parent. No other claim covers the expansion cache.
vendor/ is untouched.

Test results

tex-expansion, cargo test --release: 0 + 63 + 9 + 4 + 2 passed, 0 failed
  incremental_stops: test result: ok. 9 passed; 0 failed
compiler, cargo test --release: 59 test binaries, 688 passed, 0 failed, 6 ignored
  expansion_incremental: test result: ok. 7 passed; 0 failed (release 10.5s)
  expansion_incremental (debug, with the stopped-run parity assertion): test result: ok. 7 passed; 0 failed (113.53s)
  output_heavy_runaway_loops_match_full_expansion_under_random_edits: 8 runaway revisions in 32 steps
Mutation (no output limit in expand_project): both new tests fail
  step 18 (insert "\\def\\o{xyzw xyzw xyzw\\o}\\o " at 4625): tokens differ at 763823 (cached 763823 vs full 801109 tokens)
fuzz_compile --digest: 419 seeds, shasum 36b592ea57085d703ffea84d4b6979c628dbf236, identical to adc2c675
After the rewording (b6a7e6b1):
  tex-expansion, cargo test --release: 0 + 63 + 9 + 4 + 2 passed, 0 failed
  compiler, cargo test --release: 59 test binaries, 688 passed, 0 failed, 6 ignored
  expansion_incremental (debug): test result: ok. 7 passed; 0 failed (98.57s)
  Mutation (resume unexpanded after an output-limit stop): an_output_heavy_loop_stops_both_paths_at_the_output_token_limit fails
  fuzz_compile --digest: 419 seeds, 36b592ea57085d703ffea84d4b6979c628dbf236
After the nesting notes (c8ffe671):
  tex-expansion, cargo test --release: 0 + 64 + 9 + 4 + 2 passed, 0 failed
  compiler, cargo test --release: 59 test binaries, 689 passed, 0 failed, 6 ignored
  fuzz_compile --digest: 419 seeds, 36b592ea57085d703ffea84d4b6979c628dbf236

Not done

The expander's own tokens/origins vectors in tex-expansion still grow by doubling (5.0M tokens on 18502, about 450 MB of capacity). Their final size is not known ahead of the run, so they are not reserved. The footprint is back at tex-expansion: incremental re-expansion equals a full run across step/output-limit stops (stacked on #592) #601's level without this.

🤖 Generated with Claude Code

https://claude.ai/code/session_012c9XLkHjePPGBuarrmE2mz

What changed: - tests/fuzz_support/mod.rs: seeds from every .tex under fixtures/ and crates/*/tests; per-case splitmix RNG (seed, index) so every finding replays; mutations: truncate, delete/duplicate brace/bracket/$/\begin/\end tokens, splice lines between files, 10k-deep nesting, 100k-char control sequence names, unbalanced \if/\fi, \def recursion, invalid \char code points, \input of itself. Each case runs compile_full_project (parse + layout) on an 8 MiB thread under catch_unwind with a watchdog; worker processes isolate stack overflows and hangs; ddmin minimiser. - examples/fuzz_compile.rs: supervisor, --replay, --check, --minimise, --diagnostics. - tests/fuzz_compile.rs: the same run as an #[ignore] test (debug build). What was run: - fuzz_compile --cases 300 --jobs 12: "fuzz: 300 cases, 4 unique findings" - FLASHTEX_FUZZ_CASES=40 cargo test --test fuzz_compile -- --ignored: "the fuzzer found 3 unique failures" (expected before the fixes) Next step: 50k-case run, minimise and fix each finding. Implementation-Agent: claude-opus-5 subagent of daniel-parent Commit-Executor: daniel-parent subagent Claude-Session: https://claude.ai/code/session_012c9XLkHjePPGBuarrmE2mz

@toodeep

… u8 overflow What changed: - The per-kind and total list depths were `count() as u8 + 1` in push_list_frame and open_list: 255 enclosing lists panicked with "attempt to add with overflow" (fuzz finding). Depths now saturate (next_list_depths), and a \begin past LaTeX's \@toodeep limits (six \list levels; four itemize/enumerate) reports "LaTeX Error: Too deeply nested." while the list is still typeset. - enumitem start=<i64::MIN> no longer overflows `n - 1`. - Regression test in tests/robustness.rs. What was run: - cargo test --test robustness lists_nested (before the fix): "panicked at src/parser.rs:6972:26: attempt to add with overflow" - after: "test result: ok. 1 passed; 0 failed" Next step: remaining fuzz findings. Implementation-Agent: claude-opus-5 subagent of daniel-parent Commit-Executor: daniel-parent subagent Claude-Session: https://claude.ai/code/session_012c9XLkHjePPGBuarrmE2mz

What changed: - finish_math took an unclosed formula's end from the last content token. When the content was re-read from earlier input (`\setlength{\begin{}$`) that token lay before the `$`, so Span::in_document tripped "span start must not exceed end" (debug) or produced an inverted span (release). The end now comes only from the opener's document and never precedes the opener's end. - Regression test an_unclosed_math_span_never_inverts. What was run: - before: "panicked at src/lib.rs:60:9: span start must not exceed end" - after: cargo test --test robustness: "test result: ok. 9 passed; 0 failed" Next step: the cross-document alignment span. Implementation-Agent: claude-opus-5 subagent of daniel-parent Commit-Executor: daniel-parent subagent Claude-Session: https://claude.ai/code/session_012c9XLkHjePPGBuarrmE2mz

… in one document What changed: - multirow_environment merged every cell token's span into the row span; with `\begin{align}a\include{sub}` the cells hold tokens of sub.tex and main.tex, tripping "cannot merge spans from different documents". Row spans now merge only tokens of the environment's own document, and an `\end{align}` found in another document no longer sets the display's end offset (it could invert the span). - Regression test an_alignment_that_inputs_another_document_keeps_its_spans_in_one_document. What was run: - before: "panicked at src/lib.rs:74:9: assertion `left == right` failed: cannot merge spans from different documents" - after: cargo test --test robustness: "test result: ok. 9 passed; 0 failed" Next step: remaining fuzz findings (tex-expansion overflow, stack overflow, hangs). Implementation-Agent: claude-opus-5 subagent of daniel-parent Commit-Executor: daniel-parent subagent Claude-Session: https://claude.ai/code/session_012c9XLkHjePPGBuarrmE2mz

…flow What changed: - Fuzz finding: `\advance` added to a count holding i64::MAX (an over-long digit string parsed to i64::MAX) and panicked with "attempt to add with overflow" (expand.rs do_arith). - Constants follow tex.web §445: past 2^31-1 is "Number too big." and becomes 2147483647 (decimal, octal, hex; also the integer part of a dimen/glue, which silently became 0 before). - `<dimen>` past max_dimen is "Dimension too large." and becomes ±max_dimen (§448). - \advance and \multiply check the result against infinity / max_dimen; on overflow they report "Arithmetic overflow." and assign nothing (§1236-1240). - \numexpr/\dimexpr: saturating steps, and an out-of-range result is "Arithmetic overflow." with value 0 (e-TeX); rounded_div computes in i128. - `fil` followed by more than two further `l`s overflowed the u8 order; now "Illegal unit of measure (replaced by filll)." (§454). - Regression test numeric_ranges_follow_tex_instead_of_overflowing. What was run: - before: left "9223372036854775807" (and the add-overflow panic) - after: cargo test (tex-expansion): "test result: ok. 56 passed; 0 failed" (expand_tests), "ok. 3 passed" (incremental), "ok. 2 passed" (oracle) Next step: stack overflow and hang findings. Implementation-Agent: claude-opus-5 subagent of daniel-parent Commit-Executor: daniel-parent subagent Claude-Session: https://claude.ai/code/session_012c9XLkHjePPGBuarrmE2mz

…e stack What changed: - Fuzz finding: a table cell, footnote, \colorbox/\rotatebox/\uline argument or \include re-enters parse_stream on its own token stream with no depth bound. 3000 nested tabulars overflowed an 8 MiB release stack ("thread has overflowed its stack"), and 10k took minutes (every level copies its cell tokens). - parse_stream now counts its nesting; past STREAM_DEPTH_LIMIT = 255 it reports "TeX capacity exceeded, sorry [grouping levels=255]." once per compile and skips the content nested past the limit. - fuzz_support: FLASHTEX_FUZZ_STACK_KB overrides the case stack size. - Regression test nested_sub_parses_hit_tex_grouping_capacity_instead_of_the_stack (debug frames are ~50 KiB per tabular level, so it runs on a 64 MiB thread). What was run: - before: "thread '<unknown>' has overflowed its stack / fatal runtime error" - after: "test result: ok. 1 passed; 0 failed" (5.86s, debug) - release --check at depth 255, 2 MiB stack: "signature: ok" (tabular, footnote) Next step: hangs (\loop runaway, unknown-command suggestion cost). Implementation-Agent: claude-opus-5 subagent of daniel-parent Commit-Executor: daniel-parent subagent Claude-Session: https://claude.ai/code/session_012c9XLkHjePPGBuarrmE2mz

…x.web §339) What changed: - Fuzz finding (hang): `\loop{\n}` with no \repeat. The file ended while \loop's argument was scanned; the expander reported the runaway but still expanded the body with the partial argument, so \iterate ran until the 2M expansion-step limit, emitting "Extra \fi." (and, via the compiler, one unknown-command diagnostic) per iteration: >2 minutes to compile. - TeX sets long_state := outer_call there, so the inserted \par aborts the call. report_file_ended_and_recover now sets the silent runaway flag for Matching, and call_macro aborts. - Regression test a_macro_call_cut_off_by_the_end_of_file_is_aborted. What was run: - before: diagnostics ["Runaway argument?...", "Extra \fi.", "Extra \fi.", ...] - after: cargo test (tex-expansion): "ok. 57 passed; 0 failed" (expand_tests), "ok. 3 passed" (incremental), "ok. 2 passed" (oracle) Next step: the unknown-command suggestion cost behind the remaining hangs. Implementation-Agent: claude-opus-5 subagent of daniel-parent Commit-Executor: daniel-parent subagent Claude-Session: https://claude.ai/code/session_012c9XLkHjePPGBuarrmE2mz

… cannot hang What changed: - Fuzz finding (hang): `\def\a{\n\a}\a` emits ~666k `\n` before the 2M expansion-step limit. Each got an unknown-command diagnostic whose did-you-mean scanned the whole vocabulary several times (command_error -> closest_commands, command_help -> suggest_command, is_known_command), allocating per candidate: 269 s in release. - closest_commands is memoised per thread (bounded at 4096 names); is_known_command is a HashSet lookup; candidates whose length differs by more than the edit limit are skipped before edit_distance copies the name (a 100k-character control sequence: 860 ms -> 6 ms). - Regression test a_runaway_loop_of_unknown_commands_is_diagnosed_in_bounded_time. What was run: - release, before: "269114 ms, 1 pages, 666816 diagnostics" - release, after: "1989 ms, 1 pages, 666816 diagnostics" - debug test after: "test result: ok. 1 passed; 0 failed ... finished in 10.88s" Next step: re-run the 50k fuzz on the fixed build. Implementation-Agent: claude-opus-5 subagent of daniel-parent Commit-Executor: daniel-parent subagent Claude-Session: https://claude.ai/code/session_012c9XLkHjePPGBuarrmE2mz

…ty exceeded" What changed: - Fuzz finding (hang, 9 of 26 hangs in a 2000-case run): `\def\a#1{\a{#1#1}}\a x` doubles its argument on every call, so the token lists exhaust memory and time after ~30 calls while the 2M expansion-step limit is nowhere near. - call_macro sizes the substitution before building it; past the output-token budget (Limits::max_output_tokens) it reports "TeX capacity exceeded, sorry [main memory size=5000000]." and stops expansion, as TeX does when main memory runs out. - Regression test an_argument_that_doubles_every_call_exceeds_capacity. What was run: - before (release fuzz build, --check, 40 s watchdog): cases 1204 1196 1599 1849 567 894 393 941 1280 "signature: hang" - after: all nine "signature: ok" in <= 1 s - cargo test (tex-expansion): "ok. 58 passed; 0 failed" (expand_tests), "ok. 3 passed" (incremental), "ok. 2 passed" (oracle) Next step: the slow deep-nesting cases. Implementation-Agent: claude-opus-5 subagent of daniel-parent Commit-Executor: daniel-parent subagent Claude-Session: https://claude.ai/code/session_012c9XLkHjePPGBuarrmE2mz

What changed: - Reported from another lane: parsing crates/page-builder/oracle/fixtures/vspace-03.tex panicked with "span start must not exceed end" in required_group_bounded. Minimised: `\AtBeginDocument{\setlength\}}` + `\begin{document`. The replayed argument's closing offset lies before its `{`. - One helper, P::span_through(open, end), now builds every "opener through end offset" span (required_group_bounded, display math, alignments, tabular, the delimited-argument reader): the end never precedes the opener's end or runs past the opener's document. - Regression test an_argument_replayed_from_at_begin_document_never_inverts_its_span (minimised input plus the whole fixture). What was run: - before: "panicked at src/lib.rs:60:9: span start must not exceed end" - after: "test result: ok. 1 passed; 0 failed" Next step: \loop oracle fixtures parse time; slow deep-nesting cases. Implementation-Agent: claude-opus-5 subagent of daniel-parent Commit-Executor: daniel-parent subagent Claude-Session: https://claude.ai/code/session_012c9XLkHjePPGBuarrmE2mz

…rash inputs What changed: - Seeds now include crates/*/oracle/**/*.tex (the page-builder oracle fixture vspace-03.tex panicked the parser and was not a seed). This changes which input each case index generates. - Up to 50 inputs per hang/crash signature are saved for triage (those signatures do not distinguish causes). What was run: - fuzz_compile --cases 2000: "fuzz: 2000 cases, 1 unique findings" (26 hangs, triaged) Next step: full 50k re-run on the fixed build. Implementation-Agent: claude-opus-5 subagent of daniel-parent Commit-Executor: daniel-parent subagent Claude-Session: https://claude.ai/code/session_012c9XLkHjePPGBuarrmE2mz

…bug threads) What changed: - Measured after the previous commit: 255 nested tabulars still overflow a 512 KiB thread in release (128 already did; ~7 KiB a level) and a 2 MiB debug thread fails at 40 levels (~50 KiB a level), so the guard did not fire first on every profile. STREAM_DEPTH_LIMIT is now 32, like MAX_MATH_DEPTH; the error reads "TeX capacity exceeded, sorry [grouping levels=32]." - \include nesting keeps its own INCLUDE_DEPTH_LIMIT (64) and no longer counts toward the sub-parse depth. - The regression test runs on the default test thread again. What was run: - release fuzz build, FLASHTEX_FUZZ_STACK_KB=512, 3000 nested tabular / footnote: "signature: ok" (both) - cargo test --test robustness nested_sub_parses: "test result: ok. 1 passed" (0.28s) Next step: \left...\right nesting is quadratic in layout. Implementation-Agent: claude-opus-5 subagent of daniel-parent Commit-Executor: daniel-parent subagent Claude-Session: https://claude.ai/code/session_012c9XLkHjePPGBuarrmE2mz

…not quadratic layout What changed: - Fuzz finding (slow compile, 15.7 s release for one input): sizing a \left/\right pair lays out all the atoms it encloses (left_right_stretch_scales), so 10k nested pairs cost depth x length. - The math parser counts open \left's; past MAX_LEFT_RIGHT_DEPTH = 32 it reports "TeX capacity exceeded, sorry [grouping levels=32]." once and drops the extra \left's and their \right's. Formulas within the limit are unchanged (left_right_nesting goldens pass). - Regression test deeply_nested_left_right_pairs_hit_a_capacity_limit_in_bounded_time. What was run: - before: fuzz case 529 "15664 ms, 2 pages, 17120 diagnostics" (release) - after: debug test with 10k nested pairs "ok. 1 passed ... finished in 0.13s" - cargo test --test left_right_nesting: "test result: ok. 3 passed; 0 failed" Next step: re-time the remaining slow fuzz cases. Implementation-Agent: claude-opus-5 subagent of daniel-parent Commit-Executor: daniel-parent subagent Claude-Session: https://claude.ai/code/session_012c9XLkHjePPGBuarrmE2mz

…ot define What changed: - self_referential_macro_hits_explicit_recursion_limit used `\newcommand{\loop}{\loop} \loop`. The prelude already defines \loop, so \newcommand keeps `\loop#1\repeat`, whose argument runs to the end of the file. It only reached the step limit through the bug fixed in "tex-expansion: a macro call cut off by the end of file is aborted"; TeX reports the runaway argument instead. The test now uses \recurse. What was run: - cargo test --manifest-path crates/compiler/Cargo.toml --no-fail-fast: 58 suites, 676 passed, 0 failed, 6 ignored Next step: 50k fuzz re-run, PR. Implementation-Agent: claude-opus-5 subagent of daniel-parent Commit-Executor: daniel-parent subagent Claude-Session: https://claude.ai/code/session_012c9XLkHjePPGBuarrmE2mz

What changed: - rustfmt on the new fuzz files and tests/robustness.rs (which was rustfmt-clean on main). No behaviour change. What was run: - cargo test --test robustness --test fuzz_compile: "test result: ok. 13 passed" and "ok. 0 passed; 0 failed; 1 ignored" Next step: PR. Implementation-Agent: claude-opus-5 subagent of daniel-parent Commit-Executor: daniel-parent subagent Claude-Session: https://claude.ai/code/session_012c9XLkHjePPGBuarrmE2mz

…city exceeded" What changed: - Fuzz finding (36 GB resident, killed): `\def\a{\csname a\endcsna` + 10k `[` ... `]` + `me}\a`. `\csname a` re-enters \a before the rest of its 20k-token body is read, so every call adds an input level holding that body; memory ran out long before the 2M expansion-step limit. - Pushing a token list now checks TeX's limits: 10000 input levels ("TeX capacity exceeded, sorry [input stack size=10000].", TeX Live's stack_size) and, every 64 levels, the tokens still pending across all levels against the output-token budget ("TeX capacity exceeded, sorry [main memory size=5000000]."). Either stops expansion. - Regression test a_non_tail_self_call_exceeds_capacity_instead_of_memory. What was run: - before: /usr/bin/time -l fuzz_compile --check case-14361: "36046176256 maximum resident set size", 80 s, killed (exit 137) in a second run - after: "30 ms, 1 pages, 133 diagnostics", "172441600 maximum resident set size" - cargo test (tex-expansion): "ok. 59 passed; 0 failed" (expand_tests), "ok. 3 passed" (incremental), "ok. 2 passed" (oracle) Next step: remaining slow cases; full compiler suite; PR. Implementation-Agent: claude-opus-5 subagent of daniel-parent Commit-Executor: daniel-parent subagent Claude-Session: https://claude.ai/code/session_012c9XLkHjePPGBuarrmE2mz

…ry block What changed: - Fuzz finding (25.7 GB resident): 30k nested itemize. Every block clones the enclosing list frames, so memory was blocks x depth. - push_list_frame now stores no frame past LaTeX's limit (the one that reports "LaTeX Error: Too deeply nested."); it counts dropped levels, and a list \end pops a dropped level first. Blocks hold at most six frames; the dropped levels typeset at the deepest stored level. - lists_nested_past_255_levels_are_too_deeply_nested_not_an_overflow also checks that no block stores more than six frames. What was run: - before: fuzz case 12106 "4117 ms, 884 pages", "25687048192 maximum resident set size"; test: "a block stored 7 list frames" - after: "622 ms, 884 pages", "138084352 maximum resident set size"; test "ok. 1 passed" - cargo test --test list_structure --test setlist_spacing --test acceptance --test pinned_fixtures --lib: all "test result: ok" Next step: remaining slow fuzz cases. Implementation-Agent: claude-opus-5 subagent of daniel-parent Commit-Executor: daniel-parent subagent Claude-Session: https://claude.ai/code/session_012c9XLkHjePPGBuarrmE2mz

What changed: - Fuzz finding (22 s): a mutated microtype oracle fixture whose \loop never ends. do_conditional (and a taken branch's \else) called line_of_span on every \if, counting newlines from the start of the file, for the "Incomplete \if...; all text was ignored after line N." message that only an unterminated conditional prints: quadratic in a runaway loop. - ScannerStatus::Skipping now carries the \if's span; the line is counted when that error is reported. The unused line count on a mismatched \end is gone. - Regression test conditionals_in_a_runaway_loop_do_not_rescan_the_source (also pins the "after line 3." wording). What was run: - before: test panicked "244.136504125s" (debug, 30 s budget) - after: "test result: ok. 1 passed ... finished in 0.53s"; tex-expansion cargo test: "ok. 60 passed" (expand_tests), "ok. 3 passed" (incremental), "ok. 2 passed" (oracle) Next step: re-time the remaining slow fuzz cases. Implementation-Agent: claude-opus-5 subagent of daniel-parent Commit-Executor: daniel-parent subagent Claude-Session: https://claude.ai/code/session_012c9XLkHjePPGBuarrmE2mz

What changed: - The new tex-expansion capacity stops ("TeX capacity exceeded, sorry [input stack size=10000]." / "[main memory size=5000000].") end expansion at a point that depends on where the run started. The expansion cache only knew the step limit, so an incremental run could stop elsewhere than a full one: expansion_incremental's random-edit test diverged at step 198 ("tokens differ at 60356 (cached 60356 vs full 60386 tokens)"). The rest of the document was also dropped instead of typeset unexpanded, and the recovery note said "continued expanding". - step_limit_hit (cache fallback, unexpanded resume) and recovery_for now treat both stops alike. - Regression test a_capacity_stop_is_recovered_like_the_step_limit. What was run: - before: expansion_incremental FAILED as above; robustness test: left Some("continued expanding after the problem") - after: cargo test --test expansion_incremental: "ok. 2 passed; 0 failed"; cargo test --test robustness a_capacity_stop: "ok. 1 passed" Next step: full compiler suite, final fuzz run, PR. Implementation-Agent: claude-opus-5 subagent of daniel-parent Commit-Executor: daniel-parent subagent Claude-Session: https://claude.ai/code/session_012c9XLkHjePPGBuarrmE2mz

@Currenvir

What changed: - tex-expansion: an identical diagnostic (severity, message, span) is recorded once between safe points; the set is emptied when next_content_token returns at a safe point, which is where incremental checkpoints are taken, so restored engines decide exactly as a full run. - tex-expansion: "group nesting limit exceeded" / "conditional nesting limit exceeded" are reported once per excursion (flags in checkpointed State). - tex-expansion: \end compares @Currenvir part by part without rebuilding the name; environment names longer than 100 characters are shortened in messages only. - compiler: diagnostics::limit_repeats drops identical repeats and keeps 1000 distinct diagnostics per code, then one "further N similar diagnostics suppressed" summary per code; fatal stops are always kept. Applied to the parser's list and to the layout's list, identically in clean and incremental compiles. - compiler: unsupported / unsupported_preamble skip building a repeat of a command already reported at the same span. - fuzz harness: per-case diagnostic count and time, max diagnostics and slowest case in the report; example --diagnostics prints the most frequent messages; --digest hashes pages and diagnostics of every seed. - tests: diagnostic_floods.rs (child process per case, peak heap via a counting allocator, watchdog), engine excursion/dedupe/long-name tests, incremental equivalence with repeats and nesting limits, warm-vs-fresh JSON identity with bounded diagnostics. What was run: - cargo test --manifest-path crates/tex-expansion/Cargo.toml: test result: ok. 63 passed; 0 failed (expand_tests), test result: ok. 4 passed; 0 failed (incremental_tests), test result: ok. 2 passed; 0 failed (oracle_tests) - cargo test --manifest-path crates/compiler/Cargo.toml --no-fail-fast: 59 suites, 684 passed, 0 failed, 6 ignored - fuzz_compile --cases 50000 --jobs 6 --seed 0xF1A57E40 (overflow checks, debug assertions): 0 panics, 0 overflows, 14 hangs (base 18); max diagnostics in one case 3109 (base 2974694) - fuzz_compile --digest over 419 seeds: pages identical for all 419; diagnostics differ in 49 (48 \loop fixtures ~14750 -> ~134, and 20-Vmatrix 3 -> 2, a duplicate glyph note) Next step: merge the updated compiler-fuzz branch and open the PR stacked on #577. Implementation-Agent: claude-opus-5 subagent of daniel-parent Commit-Executor: daniel-parent subagent Claude-Session: https://claude.ai/code/session_012c9XLkHjePPGBuarrmE2mz

…no diagnostic What changed: - `\advance` on count, dimen and skip registers (value, stretch and shrink) adds with explicit i32 wrapping (tex.web §1238-1239 has no range check) instead of reporting "Arithmetic overflow." and leaving the register. `\multiply`, `\divide`, `\numexpr` and `\dimexpr` keep their bounds checks. - numeric_ranges_follow_tex_instead_of_overflowing pins pdfTeX's wrapped values with no diagnostic. What was run: - pdftex 3.141592653-2.6-1.40.29 (TeX Live 2026), plain and -etex: \count0=2147483647 \advance\count0 by 1 \showthe\count0 -> -2147483648 \count0=-2147483647 \advance\count0 by -2 -> 2147483647 \dimen0=\maxdimen \advance\dimen0 by 1sp \showthe\dimen0 -> 16384.0pt \dimen0=16383pt \advance\dimen0 by 16383pt -> 32766.0pt 3x\maxdimen -> -16384.00005pt; 4x\maxdimen+1sp -> -0.00005pt skip \maxdimen plus \maxdimen minus 1pt, advanced by the same minus 2pt -> 32767.99997pt plus 32767.99997pt minus 3.0pt - cargo test -p flashtex-tex-expansion --test expand_tests: test result: ok. 60 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out Next step: reword or raise the grouping-level capacity limit (#577 review). Implementation-Agent: claude-opus-5 subagent of daniel-parent Commit-Executor: daniel-parent subagent Claude-Session: https://claude.ai/code/session_012c9XLkHjePPGBuarrmE2mz

What changed: merged origin/agent/daniel-parent/compiler-fuzz 0c1524c (no rebase), per the coordinator. What was run: see the next checkpoint. Next step: re-run suites on the merge. Implementation-Agent: claude-opus-5 subagent of daniel-parent Commit-Executor: daniel-parent subagent Claude-Session: https://claude.ai/code/session_012c9XLkHjePPGBuarrmE2mz

…s where pdflatex does What changed: - STREAM_DEPTH_LIMIT stays 32 (255 nested tabulars, footnotes or rotateboxes overflow a 512 KiB release and a 2 MiB debug thread), and its error now says "FlashTeX nesting limit (32) exceeded" instead of claiming TeX's "grouping levels" capacity. - MAX_LEFT_RIGHT_DEPTH rises from 32 to 253 (TEX_GROUPING_LEVELS - 2), with "TeX capacity exceeded, sorry [grouping levels=255].": pdflatex accepts 253 nested \left in a document's $$ or $ math and stops at the 254th. The \left counter is not recursive, so the stack is not the bound. What was run: - pdflatex (TeX Live 2026), \documentclass{article}\begin{document}$$...$$: 252 ok, 253 ok, 254 and 255 "! TeX capacity exceeded, sorry [grouping levels=255]." inline $...$: 253 ok, 254 capacity exceeded. - Scratch probe (not committed) with both limits at 255: release, 512 KiB thread: tabular/footnote/rotatebox 255 -> stack overflow; colorbox/uline 255 ok; \left 255 ok (also on 256 KiB), 10000 in 120 ms debug, 2 MiB thread: tabular/footnote/colorbox/rotatebox/uline 255 -> stack overflow; \left 255 ok (also on 512 KiB), 10000 in 714 ms - cargo test -p flashtex-compiler --test robustness: test result: ok. 14 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 36.36s Next step: full compiler and tex-expansion suites and a 5k fuzz smoke run. Implementation-Agent: claude-opus-5 subagent of daniel-parent Commit-Executor: daniel-parent subagent Claude-Session: https://claude.ai/code/session_012c9XLkHjePPGBuarrmE2mz

…flood-cap What changed: merged origin/agent/daniel-parent/compiler-fuzz d5d308e (no rebase), per the coordinator. What was run: full compiler and tex-expansion suites on the merge (see the PR). Next step: open the PR stacked on #577. Implementation-Agent: claude-opus-5 subagent of daniel-parent Commit-Executor: daniel-parent subagent Claude-Session: https://claude.ai/code/session_012c9XLkHjePPGBuarrmE2mz

What changed: - The step limit and the output token limit count from the document start, so a state equivalent at a checkpoint does not decide whether a reused suffix ends the same way. Convergence now also requires the new totals to stay within both limits (or to reach the old run's stop at the same count), and the expander records how its last run ended. - Checkpoints carried past a converged edit get the step offset of the new run; they kept the old run's step count, so a later restart from one stopped at the wrong place (seed 3 edit 54 of the nesting-limit test). - Checkpoints keep the engine's last origin, where a step limit hit on the next step is reported; convergence compares it. - Un-skipped the stop edits in the nesting-limit test; new tests/incremental_stops.rs with minimised regressions and a property test over the fuzz seeds with small limits. What was run: - cargo test --test incremental_stops --test incremental_tests: test result: ok. 6 passed; 0 failed test result: ok. 4 passed; 0 failed - Against the base source: 5 of the 6 incremental_stops tests fail. Next step: full suites, latency bench, PR. Implementation-Agent: claude-opus-5 subagent of daniel-parent Commit-Executor: daniel-parent subagent Claude-Session: https://claude.ai/code/session_012c9XLkHjePPGBuarrmE2mz

…ops apart from nesting limits What changed: - step_limit_right_after_a_checkpoint_reports_the_last_origin now edits after the stop with a checkpoint at every safe point, so some cut restarts one step before the limit. - The property test reports stops and nesting-limit excursions separately. What was run: - With the last-origin restore and comparison removed, the test fails: "diagnostics differ -- cut 5". With them: ok. - cargo test --manifest-path crates/tex-expansion/Cargo.toml: test result: ok. 63 passed; 0 failed (expand_tests) test result: ok. 6 passed; 0 failed (incremental_stops) test result: ok. 4 passed; 0 failed (incremental_tests) test result: ok. 2 passed; 0 failed (oracle_tests) - FLASHTEX_INC_STOP_EDITS=60, release: 419 seeds, 25140 edits: 17443 ended at a stop, 6524 went past a nesting limit, 2569 converged; ok. Next step: latency benches, compiler suite, PR. Implementation-Agent: claude-opus-5 subagent of daniel-parent Commit-Executor: daniel-parent subagent Claude-Session: https://claude.ai/code/session_012c9XLkHjePPGBuarrmE2mz

What changed: - IncrementalExpander::edit_with_limits applies an edit with new step and output-token limits (the compiler's limits grow with the document). A checkpoint is restarted from only if its step count, output count and peak main-memory size fit the new limits; Converge::same_end compares stops against the new limits (a step or output stop is reused when the new total there is the new limit plus one, a main-memory stop only under the same limit, a suffix that did not stop only if its peak fits). Changing the nesting limits re-expands the document. - Engine tracks peak_memory (largest main-memory size checked without stopping) and whether it stopped on main memory; checkpoints keep it. - IncrementalExpander::input_position: where the last run ended, as Engine::input_position gives it for a full run. - tests/incremental_stops.rs: input positions are compared; new tests edits_with_new_limits_cross_every_stop, a_smaller_memory_limit_invalidates_checkpoints_after_a_large_expansion, incremental_matches_full_on_fuzz_seeds_when_limits_change. What was run: - cargo test --release (tex-expansion): all test binaries ok; incremental_stops "test result: ok. 9 passed; 0 failed" - FLASHTEX_INC_STOP_EDITS=30 seeds 1, 2: "419 seeds, 12570 edits with changing limits: 8351 ended at a stop, 972 converged" / "8131 ended at a stop, 960 converged", ok - Each of the 7 new conditions removed in turn: every mutant fails a test. Next step: the compiler's cached expansion uses this and drops its full re-expansion after a stop. Implementation-Agent: claude-opus-5 subagent of daniel-parent Commit-Executor: daniel-parent subagent Claude-Session: https://claude.ai/code/session_012c9XLkHjePPGBuarrmE2mz

What changed: - expand_project_with_cache no longer re-runs expand_project after the incremental run hits the step limit or TeX capacity (and no longer starts every later revision of a runaway document with a full run). The expander equals a full run across stops (#601), so the cache now types the rest of the entry unexpanded from IncrementalExpander:: input_position, as expand_project does; those tokens are dropped before the next edit is spliced. - The cache expands under expand_project's limits (all project documents' bytes) on every revision via edit_with_limits; it used the entry's size at creation, which only the fallback had hidden. - Debug builds compare every stopped cached run with expand_project. - tests/expansion_incremental.rs: edits_around_a_runaway_loop_match_full_ expansion (typing before/inside/after a loop, same-length replacements that converge with the stopped run, deleting it) and a_runaway_entry_ follows_the_size_of_other_project_documents. What was run: - cargo test --release --test expansion_incremental: "test result: ok. 5 passed; 0 failed"; debug: "ok. 5 passed; 0 failed" (94.64s) - Mutants (no truncation of the recovery, entry-only limits, no recovery) each fail a test. Next step: fixture digests, compiler/tex-expansion suites, before/after timings. Implementation-Agent: claude-opus-5 subagent of daniel-parent Commit-Executor: daniel-parent subagent Claude-Session: https://claude.ai/code/session_012c9XLkHjePPGBuarrmE2mz

What changed: - edit_latency_bench: `HW1 runaway` scenarios (HW1 with a \def\r{x\r}\r loop; typing while it runs away, and deleting/restoring the loop's line) and `--only SUBSTR`. - fuzz_compile `--diagnostics FILE --digest` prints page and diagnostic hashes, to diff two builds on one runaway input. What was run: - edit_latency_bench on 1e4bb85 (#601 head) and this branch: every scenario's reply sha256 is identical (HW1, 500KB, HW1 runaway). - fuzz_compile --diagnostics --digest on fuzz cases 10082 and 18502: same pages and diagnostics digests on both builds. Next step: open the PR with the before/after table. Implementation-Agent: claude-opus-5 subagent of daniel-parent Commit-Executor: daniel-parent subagent Claude-Session: https://claude.ai/code/session_012c9XLkHjePPGBuarrmE2mz

…ocation for the cached stream What changed: - expand_project counts engine tokens and stops once the count goes past max_output_tokens, with the incremental expander's "output token limit exceeded" diagnostic, after converting the token that went past it. Before, only the incremental path had this limit, so a small (< 4 KB, full path) document with an output-heavy loop ran on to the step limit and typeset the rest unexpanded, while a larger one stopped earlier. - The debug-build parity check in expand_project_with_cache also covers output-limit stops. - build_cache reserves the converted stream once (engine tokens + bytes of the unexpanded rest) and finish trims it to at most 1/8 spare. Growing it by doubling left a freed chain of blocks resident for the rest of the compile: fuzz case 18502's peak footprint 2146 MB -> 1746 MB (#601: 1744 MB). - Tests: an_output_heavy_loop_stops_both_paths_at_the_output_token_limit and output_heavy_runaway_loops_match_full_expansion_under_random_edits. What was run: - cargo test --release --test expansion_incremental: test result: ok. 7 passed; 0 failed - Mutation (no output limit on the full path): both new tests fail (tokens differ at 763823, cached 763823 vs full 801109 tokens). - fuzz_compile --digest, 419 seeds: shasum 36b592ea57085d703ffea84d4b6979c628dbf236, identical to #610's head. Next step: suites, edit_latency_bench, PR stacked on #610. Implementation-Agent: claude-opus-5 subagent of daniel-parent Commit-Executor: daniel-parent subagent Claude-Session: https://claude.ai/code/session_012c9XLkHjePPGBuarrmE2mz

… rest was not typeset What changed: - The output token limit's diagnostic is now, in TeX's capacity wording, "TeX capacity exceeded, sorry [output token limit=N]; expansion stopped here and the rest of the document was not typeset." (was "output token limit exceeded"). One definition, tex::output_limit_message, used by IncrementalExpander, Engine::run and expand_project; tex::is_output_limit matches it. - Its compiler recovery note is "stopped expanding; the rest of the document was not typeset" (was "...typeset without macro expansion", which neither path does after this stop). - The message now starts with "TeX capacity exceeded, sorry [", so the unexpanded recovery (step_limit_hit) excludes it explicitly; is_stop_limit still matches it, so the diagnostic flood cap always keeps it. - The step limit's text is unchanged: it does typeset the rest unexpanded. - Tests assert the exact message and recovery, and that nothing after the loop is typeset. What was run: - tex-expansion cargo test --release: 0 + 63 + 9 + 4 + 2 passed, 0 failed - compiler cargo test --release: 59 test binaries, 688 passed, 0 failed, 6 ignored - expansion_incremental (debug): test result: ok. 7 passed; 0 failed (98.57s) - Mutation (output-limit stop resumes unexpanded): an_output_heavy_loop_stops_both_paths_at_the_output_token_limit fails. - fuzz_compile --digest, 419 seeds: 36b592ea57085d703ffea84d4b6979c628dbf236, unchanged. - bodyloop.tex: pages 163c9d2a21b1822b unchanged; diagnostics d82176b9e51d82eb -> 856cb57fcc2fede2 (message text only). Next step: reply on #619. Implementation-Agent: claude-opus-5 subagent of daniel-parent Commit-Executor: daniel-parent subagent Claude-Session: https://claude.ai/code/session_012c9XLkHjePPGBuarrmE2mz

d-q222 · 2026-09-15T11:29:36Z

daniel-parent subagent: pushed b6a7e6b (a new commit, no amend or force). It rewords the output token limit's diagnostic, as decided.

Message: was output token limit exceeded. It is now TeX capacity exceeded, sorry [output token limit=N]; expansion stopped here and the rest of the document was not typeset.
- The wording follows TeX's [name=value] capacity style.
- It has one definition, tex::output_limit_message, used by IncrementalExpander, Engine::run and expand_project.
Recovery note: was "...the rest of the document was typeset without macro expansion". It is now "stopped expanding; the rest of the document was not typeset". No recovery was added.
Guard: the new text starts with TeX capacity exceeded, sorry [, which is_stop_limit matches. The unexpanded recovery now excludes the output limit explicitly, so the stop still typesets nothing after it. A new test asserts that, and a mutation check confirms the test catches it.
Step limit: the text is unchanged. That stop really does typeset the rest unexpanded, so its note is accurate.
Tests and docs: tests updated to the exact message and note. No docs or golden files had the old text (RECOVERY.md has no output-limit case, and its sync test passes).

Results:

tex-expansion, cargo test --release: 0 + 63 + 9 + 4 + 2 passed, 0 failed
compiler, cargo test --release: 59 test binaries, 688 passed, 0 failed, 6 ignored
expansion_incremental (debug): test result: ok. 7 passed; 0 failed (98.57s)
fuzz_compile --digest: 419 seeds, 36b592ea57085d703ffea84d4b6979c628dbf236 (unchanged)
bodyloop.tex: pages 163c9d2a21b1822b unchanged; diagnostics d82176b9e51d82eb -> 856cb57fcc2fede2 (message text only)

Still misleading, not changed: group nesting limit exceeded and conditional nesting limit exceeded still get the "typeset without macro expansion" note, because recovery_for matches contains("limit exceeded"). The engine only refuses that { or conditional and keeps expanding. Should those be reworded too?

🤖 Generated with Claude Code

https://claude.ai/code/session_012c9XLkHjePPGBuarrmE2mz

…ontinued What changed: - "group nesting limit exceeded" and "conditional nesting limit exceeded" got the note "stopped expanding; the rest of the document was typeset without macro expansion" (recovery_for matched contains("limit exceeded")). The engine does not stop there: maybe_handle_brace drops the `{` without opening a group, and do_conditional returns before pushing the conditional, so its test is read as text; expansion continues in both cases. - New notes: "the extra group was ignored and expansion continued" and "the extra conditional was ignored without evaluating its test, and expansion continued". The step-limit branch now matches only is_stop_limit, whose note (rest typeset unexpanded) is true. - Messages are unchanged (not TeX capacity wording: that prefix means a stop to is_stop_limit and the flood cap). - Tests: expand_tests past_a_nesting_limit_the_extra_group_or_conditional_is_ignored_and_expansion_continues pins the engine behaviour (tokens and follow-on diagnostics); expansion_incremental expansion_limit_notes_describe_what_happens pins the three notes. What was run: - tex-expansion cargo test --release: 0 + 64 + 9 + 4 + 2 passed, 0 failed - compiler cargo test --release: 59 test binaries, 689 passed, 0 failed, 6 ignored - fuzz_compile --digest, 419 seeds: 36b592ea57085d703ffea84d4b6979c628dbf236, unchanged. - Diagnostics hashes change (pages unchanged) for 10082 9232f3ba710cfc05 -> f0c08aabd804800a, conds.tex f0075201d8ee58ff -> 1d31e6378d071330, groups.tex 78f34e551c859d61 -> cd117cd5038b2f6c. Next step: reply on #619. Implementation-Agent: claude-opus-5 subagent of daniel-parent Commit-Executor: daniel-parent subagent Claude-Session: https://claude.ai/code/session_012c9XLkHjePPGBuarrmE2mz

d-q222 · 2026-09-15T11:33:56Z

daniel-parent subagent: pushed c8ffe67 (a new commit, no amend or force). The nesting-limit recovery notes now say what actually happens.

What the engine does (checked in the code, then pinned by a new test):

group nesting limit exceeded: maybe_handle_brace drops the extra { without opening a group and returns Step::Continue. Its matching } then closes the enclosing group, and a later } can become Too many }'s.. Definitions after it still take effect.
conditional nesting limit exceeded: do_conditional returns before pushing the conditional, so the test is not evaluated. For example, 1>2 is read as text, the next \else/\fi belongs to the enclosing conditional, and a surplus \fi gets its own Extra error.
Expansion continues in both cases. The new expand_tests test past_a_nesting_limit_the_extra_group_or_conditional_is_ignored_and_expansion_continues asserts the output tokens and follow-on diagnostics.

Notes (compiler recovery_for):

Group limit: was "stopped expanding; the rest of the document was typeset without macro expansion". Now "the extra group was ignored and expansion continued".
Conditional limit: same old note. Now "the extra conditional was ignored without evaluating its test, and expansion continued".
Step limit: unchanged, because that stop really does typeset the rest unexpanded. Its branch now matches only real stops instead of contains("limit exceeded").
Messages: unchanged. TeX's TeX capacity exceeded, sorry [ prefix means a stop to the recovery check and the diagnostic flood cap, and these limits do not stop expansion.
Tests: expansion_limit_notes_describe_what_happens pins all three notes. No docs or golden files had the nesting notes.

Results:

tex-expansion, cargo test --release: 0 + 64 + 9 + 4 + 2 passed, 0 failed
compiler, cargo test --release: 59 test binaries, 689 passed, 0 failed, 6 ignored
fuzz_compile --digest: 419 seeds, 36b592ea57085d703ffea84d4b6979c628dbf236 (unchanged)

Diagnostics text changes only (pages unchanged):

10082: 9232f3ba710cfc05 -> f0c08aabd804800a
conds.tex: f0075201d8ee58ff -> 1d31e6378d071330
groups.tex: 78f34e551c859d61 -> cd117cd5038b2f6c

Also noticed, not changed: max_conditional_depth: N admits N+1 open conditionals, while max_group_depth: N admits N groups (the conditional check is depth > limit on a stack that starts at 0).

🤖 Generated with Claude Code

https://claude.ai/code/session_012c9XLkHjePPGBuarrmE2mz

d-q222 · 2026-09-15T12:49:10Z

daniel-parent independent review (subagent)
Verdict: APPROVE

Runaway-doc behavior (full, non-incremental expand_project path): matches the claim, marker confirmed absent. Built the PR-619 worktree's crates/compiler in isolation (its own compile_full/compile_full_project, not the CLI/flashtex-render-pipeline, which links crates/render-pipeline/vendor/compiler — a separately pinned snapshot that does not carry this PR's commits; using the CLI would silently test stale code, so I called flashtex_compiler::incremental::compile_full directly). Reproduced the PR body's own repro exactly: 2.4 KB doc, 40 comment-padding lines, \def\o{ + ab ×100 + \o}\o, with a unique marker (MARKERZZZ7f3a9c2b1e4d) placed on the line right after the macro call, before \end{document}. Confirmed document.text.len() = 2516 < INCREMENTAL_MIN_BYTES (4096), so this exercises the raw expand_project, not the incremental/cached path. Result: 468 pages, 2 diagnostics — TeX capacity exceeded, sorry [output token limit=2020128]; expansion stopped here and the rest of the document was not typeset. (recovery: "stopped expanding; the rest of the document was not typeset") followed by unterminated environment 'document' — an exact match to the PR body's claimed after-state (468 pages, 2 diagnostics, that exact message and recovery text). Then actually inspected the output, not just the diagnostic: scanned every TextItem.text on all 468 laid-out pages and the Debug-formatted blocks for the marker string — zero occurrences either way. The marker is genuinely never typeset, not merely claimed to be.
Fuzz digest parity vs compiler, tex-expansion: a runaway document is expanded once per compile (stacked on #601) #610's head: no divergence, on seeds 0–29 or the full corpus. Built fuzz_compile --digest (release) on both the PR-619 worktree (HEAD c8ffe671) and compiler, tex-expansion: a runaway document is expanded once per compile (stacked on #601) #610's head (adc2c675) in separate worktrees/target dirs and ran the full seed sweep on both (419 seeds each, same names in the same order). Diffed the first 30 seed lines (seed index 0–29 by the tool's deterministic discovery order) byte-for-byte: identical, both seed names and both the pages- and diagnostics-digests for every one of the 30. For extra confidence I also diffed the entire 419-seed listing: also byte-for-byte identical, and shasum of both full listings is 36b592ea57085d703ffea84d4b6979c628dbf236 on both worktrees — the exact value the PR body itself reports for compiler, tex-expansion: a runaway document is expanded once per compile (stacked on #601) #610 and for this branch. No seed in the corpus reaches the output-token-limit path this PR touches (confirmed: no seed name matching the PR's own separately-tested repro fixtures — 10082, conds.tex, groups.tex — appears in this corpus at all), so there is nothing to explain: zero divergences, expected or otherwise.

Minor, non-blocking observation: crates/render-pipeline/vendor/compiler (used by the flashtex CLI/flashtex-render-pipeline) is a separately pinned snapshot and does not yet include this PR's changes — expected, since re-pinning vendor is a distinct, separate step per this repo's existing convention (see the re-pin comments in crates/render-pipeline/Cargo.toml), and not something this PR claims to do. Noted here only so a future re-pin PR knows the CLI-level end-to-end behavior for this change hasn't been exercised yet; not a defect in #619.

Conflicts in crates/compiler/src/expansion.rs: this branch removes the Option-returning runaway fallback from update_cache/finish_diagnostics (a stopped cached expansion is not expanded a second time) while main added \refstepcounter current-label tracking. Kept the branch's control flow and signatures, carried main's current_label_log/ current_label_by_marker propagation into the cached path and the current_label_by_marker field into finish_diagnostics' Expansion. Regenerated supported-latex artifacts; bundled copy in sync. Implementation-Agent: fable-rebase-d Commit-Executor: fable-rebase-d (direct; Cursor usage limit)

…ed) into agent/daniel-parent/expansion-token-limit-parity Clean merge of the parent branch's main-merge. Implementation-Agent: fable-rebase-d Commit-Executor: fable-rebase-d (direct; Cursor usage limit)

…on-token-limit-parity Conflicts were this branch's output-token-limit parity refinements vs the #610 squash and #587's font-units work arriving from main. Kept the branch's output_limit_message/is_output_limit semantics (step_limit_hit excludes output-limit stops, refined recovery notes, capacity shrink, its extended incremental_stops/expansion_incremental/expand_tests) and took main's #587 additions (font_switches, Checkpoint metrics, scale_unit dimen scanning, DocumentFonts cache key + configure_with_fonts init, with the limits binding the run loop needs). Regenerated supported-latex artifacts; bundled copy in sync. Implementation-Agent: fable-rebase-d Commit-Executor: fable-rebase-d (direct; Cursor usage limit)

…/expansion-token-limit-parity Clean merge, no conflicts: picks up main's fix for the #624/#623 list_style_nextline compile breakage. Implementation-Agent: fable-rebase-d Commit-Executor: fable-rebase-d (direct; Cursor usage limit)

d-q222 added 30 commits September 14, 2026 23:30

d-q222 mentioned this pull request Sep 15, 2026

FT-003: dispatch to mac-claude-a — acknowledgement required #2

Closed

d-q222 mentioned this pull request Sep 15, 2026

compiler + tex-expansion: \frac is Ord, exact nesting limits, lexer doc fix #630

Closed

GoKubar added 4 commits September 15, 2026 11:22

Merge updated agent/daniel-parent/stop-single-run (origin/main resolv…

2dd66dc

…ed) into agent/daniel-parent/expansion-token-limit-parity Clean merge of the parent branch's main-merge. Implementation-Agent: fable-rebase-d Commit-Executor: fable-rebase-d (direct; Cursor usage limit)

GoKubar merged commit 386850d into main Sep 15, 2026
10 of 21 checks passed

GoKubar deleted the agent/daniel-parent/expansion-token-limit-parity branch September 16, 2026 04:16

Navigation Menu

compiler: the full expansion stops on the output token limit; 18502's peak back to 1746 MB (stacked on #610) - #619

compiler: the full expansion stops on the output token limit; 18502's peak back to 1746 MB (stacked on #610)#619
GoKubar merged 36 commits into
mainfrom
agent/daniel-parent/expansion-token-limit-parity

d-q222 commented Sep 15, 2026 •

edited

Loading

Uh oh!

d-q222 commented Sep 15, 2026

Uh oh!

d-q222 commented Sep 15, 2026

Uh oh!

d-q222 commented Sep 15, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

d-q222 commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

1. The full expansion stops on the output token limit

2. Compile-wide memory on fuzz case 18502

Why

Before → after (release, same machine, interleaved runs, load average 7–20 on 18 cores)

Identity

Overlap check

Test results

Not done

Uh oh!

d-q222 commented Sep 15, 2026

Uh oh!

d-q222 commented Sep 15, 2026

Uh oh!

d-q222 commented Sep 15, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

d-q222 commented Sep 15, 2026 •

edited

Loading