perf: lex each tag once, while finding its end (0142)

[rmorozov]

Requested by Ruslan · project thread

Before: Load read each tag three times. The splitter scanned it by characters to find its end delimiter. Then lexertk lexed the same text, and Lexer::Preprocess converted lexertk's tokens into the parser's. Every token also carried an InternalValue, which was built, copied and destroyed per token. The fine parse ran only after the whole template had been split.

After: TagLexer lexes a tag once, from its start, and stops at the first end delimiter (with an optional -/+) that starts a token outside brackets. That is how Jinja2's lexer finds the end too. The fine parse takes each block as soon as the splitter closes it. Load is 17-30% cheaper in instructions. Output, error codes and positions are unchanged, except for one message text described below.

How

• TagLexer (src/lexer.h) is a port of lexertk's scanning plus Lexer::Preprocess that makes Tokens straight from the source. lexertk's generator and the Lexer class (with its std::function) are no longer used.
• Some tags still use the old scan by characters for their end, and are then lexed on their own with the same results as before:
  • end delimiters that could start inside a token (TagLexer::UsableEnd: letters, digits, = < > * / . _ - +, quotes)
  • lexing errors and ;
  • unbalanced brackets
  • a tag never closed (checked after 256 tokens)
  • a decimal literal ending in its exponent (1e1e-}})
• DoRoughParsing calls the fine parse from PushCurrentBlock, so the m_textBlocks list is gone. A splitter error is still the only error reported. RecursionLimitExceeded still stops the fine parse, while the splitter carries on to the end.
• Token is now four plain fields. A literal's value is read from the source when the parser asks for it, through LexScanner::GetValue. This change was half the gain: EatIfEqual and the error paths copy tokens, and the list is cleared for every tag.
• Phase 6's Load cost: ValueRef names go to the arena's symbol table as a string_view of the source (LexScanner::GetAsView), with no std::string copy. On its own this is -0.5..-1.4% Load.
• SplitLines now runs only when an error or the metadata needs line numbers. Keywords are compared without memcmp, and string literals with nothing to convert skip the conversion.
• Merging master gave two conflicts, both in src/expression_parser.cpp, and each one kept master's structure:
  • perf: parse binary operators by precedence climbing (0152) #452's single-string shortcut read tok.value. The merge reads the value through LexScanner::GetValue instead.
  • In perf: one access seam for InternalValue outside wave 2's files (0140 P0a) #453's TakeString-based ParseAdjacentStrings, the merge reads the token's value through LexScanner::GetValue.

Measured: bench/count.py, Release build, against master 523d59e (with #452). Preflight --perf against master 2af42b6 (with #453) gives the same numbers within ±0.1%.

Case	Load instructions	Change
Load/many_tags	12,357,016 → 9,522,452	-22.9%
Load/chat_llama	517,464 → 374,187	-27.7%
Load/substitute	10,148 → 7,778	-23.4%
Load/mitsuhiko_table	71,133 → 49,918	-29.8%
Load/plain_text	4,015 → 3,655	-9.0%

Every other Load case moved by -17.5% (filters) to -28.8% (large_static). Peak memory and the number of allocations of Load are lower in every case. Retained memory is unchanged except for chat_llama (+64 B) and chat_qwen (+112 B). I have not found the cause of those two. Load/plain_text meets 0140's open condition (≤ 4,014).

Render was not touched. Every case is within ±0.27% except substitute, which is +27 instructions (+1.0%). Callgrind shows Render itself is 17 cheaper, but the ~InternalValue of the parameter conversion temporary is now an out-of-line call (+44). On master it was inlined. This is GCC's inlining budget in template.cpp's translation unit, which includes the parser. A noinline hint on the conversion made it worse (+36). The perf track accepted it as a named exception.

Behaviour: the verifier compared the parent and this branch, narrow and wide, over about 1.6M generated templates. These covered random, mutated and enumerated tags, custom delimiters, line statements, raw/meta blocks, every number form, literals, trans contexts and caller detection. Output, error codes and positions matched. One message text changed: an Identifier token with an empty range now always prints as '<<Identifier>>'. Before, it printed '' ({{}} in a required block), or the text of a string literal the parser had copied the token from (e.g. {% import 'x' as 'x' as y %} said Expected: 'x').

Tests:

• BasicTests.TagEndsFoundWithoutTheLexer covers custom ends handled by the old scan.
• BasicTests.MalformedTagEnds covers the malformed cases round 1 found. All of them also pass on master.
• Two new corpus cases, errors.semicolon_in_tag and errors.float_second_exponent, pin two lexer leniencies this PR keeps on purpose ({{ x ; 1 }} renders, and so does {{ 1e1e-1 }}). They are filed as task 0161.

Process:

• Roles that ran: verifier, 2 rounds.
• Round 1 found a crash and an accepted template on malformed custom-delimiter input, a 4x memory cost on a never-closed tag, and 4 clang-tidy hits. All are fixed in d4801a8 and adbe10c.
• Round 2 found no differences beyond the message text above.
• ASan+UBSan ctest and driver runs were clean, and preflight --perf passes.
• Red CI pushes: none.

🤖 Generated with Claude Code

https://claude.ai/code/session_01L1F5DBn1mFcgA83WbjqtiT