perf: InternalValue is 32 bytes, a trivially copyable variant plus one intrusive owner (0140 P2)

[rmorozov]

Requested by Ruslan · project thread

Before: InternalValue was 72 bytes. Every copy, move and destroy went through std::variant's jump tables plus a shared_ptr for parent data, even for an integer.

After: InternalValue is 32 bytes. A trivially copyable std::variant (24 B) holds scalars, strings of up to 15 narrow characters, string views and typed pointers to what the value owns. One boost::intrusive_ptr<const ValueObject> (8 B) owns those objects. Copying a value is a memcpy plus one refcount increment. Render instructions drop 8-27% on every benchmark case.

Task 0140, step P2 of docs/design/0140-internal-value-plan.md.

How

• Long strings, list and map accessors, pairs, callables, renderers and undefined info are ValueObjects with an atomic intrusive count.
• The variant stores a typed pointer to the object. The owner can be bigger than that object:
  • a loop value points at the accessor inside its LoopFrame, which is now refcounted;
  • a lent nested host item points into the 0154 ItemAdapters block, which is now a ValueObject. This is the obligation recorded in 0154.
• This replaces the aliasing shared_ptrs. ListRef/MapRef carry the owner, so ToAdapter() works on any view.
• SetParentData makes an Anchor object that holds the value's own object and its parent. The value keeps pointing where it did.
• Strings carry the Markup flag. A Slot's unbound bit is a flag on the undefined alternative.
• namedtuple field names moved into a wrapping accessor.
• Visitors get a short string as a std::basic_string built in its SSO buffer, never as a view into the value. The first run of the suite showed why: Value conversions were keeping views into dead short strings.
• unique sorts positions instead of the items. A moved-from value is now undefined, and libc++'s stable_sort leaves moved-from items behind when the comparator throws on mixed types, which unique catches before falling back.

Deliberate changes

• SetMarkup on a non-string does nothing. No caller marks a non-string.
• InternalValue("literal") is now deleted. It would otherwise convert to bool.

Counts (local, GCC 13 Release, bench/count.py vs 6471f17)

case	Render	allocs/render
mitsuhiko_table	-20.95%	=
macros	-20.10%	=
for_loop_vars	-26.60%	=
for_filter_if	-16.50%	=
config_file	-13.41%	1,124 → 1,285 (+14%)
expressions	-12.02%	=
chat_mistral	-11.80%	421 → 496 (+18%)
dict_ops	-11.71%	=
many_tags	-11.48%	=
strings	-10.18%	979 → 1,289 (+32%)
chat_qwen	-8.49%	320 → 397 (+24%)
inheritance	-8.31%	=
html_autoescape	-8.02%	938 → 1,100 (+17%)
chat_llama	-7.94%	355 → 426 (+20%)
large_static	+0.10%	=

Load is -1..-6% everywhere, except mitsuhiko_table_wide (+0.94%) and plain_text (+0.33%).

P2 gates, from the CI "instruction counts vs base" job on 60b4b7e (base ea60b00, which includes 0160):

• Render/large_static 21,101, gate ≤ 21,472.
• Render: chat_qwen -8.74%, inheritance -8.51%, chat_llama -8.18%, all below base.
• Load/expressions 44,394, gate ≤ 46,953.
• Load/plain_text 3,667, gate ≤ 4,014.

CI allocations per render: strings 975 → 1,285, chat_qwen 318 → 395, chat_llama 350 → 421, chat_mistral 416 → 491, html_autoescape 932 → 1,094, config_file 1,120 → 1,281.

Trade to accept or refuse: allocations rise where long strings are made (plan contradiction 2). A string over 15 characters takes its StringObject as well as its buffer. That misses the plan's +5% allocation bar. Bytes allocated fall on most of these cases. Task 0151, which builds strings straight into their object, is the fix.

Roles and checks

• Main session (Opus) designed and implemented.
• Verifier: one round. It found two blocking issues; both are fixed here:
  • the MSVC shared build could not link CloneObject, which is now exported;
  • clang-tidy hits in the new test.
• The verifier also checked:
  • Debug, Release, shared and C++20 builds;
  • clang-18 ASan+UBSan+LSan on the suite;
  • about 11k generated and mutated templates, byte-identical to base, narrow and wide;
  • the new tests against Python Jinja2.
• scripts/preflight.sh is clean.
• ci-triage read each red job.
• Red CI pushes: 2.
  • a4a5995 failed to build on clang and AppleClang at C++20 and later. An in-class ListAdapter::ToValueList body instantiated ~vector<InternalValue> while InternalValue was still incomplete. GCC accepts this, and preflight builds with GCC only. Fixed by defining it after the class.
  • 983aede failed only on macOS, in parity case filters.unique_mixed_types. This was a latent bug that the new move semantics exposed: unique caught the comparator's throw and then read the moved-from elements stable_sort left behind. Fixed by sorting positions.

🤖 Generated with Claude Code

https://claude.ai/code/session_012zgKj9cL7pJGk1EKHf5xPR

---

Generated by Claude Code