Skip to content

AOT: fixes for RegExp and String handling - #1083

Merged
sosukesuzuki merged 69 commits into
mainfrom
claude/elegant-darwin-6vxpab
Nov 21, 2026
Merged

sosukesuzuki merged 69 commits into
mainfrom
claude/elegant-darwin-6vxpab

Conversation

@sosukesuzuki

@sosukesuzuki sosukesuzuki commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Fixes for RegExp and String handling in programs that are compiled ahead of time, one commit for each. The commit messages say what went wrong.

What changes

  • Memory safety: reads out of bounds through stale match bounds, false length and range proofs and ropes flattened in front of loops; uses after free in caches of strings and in hoisted addresses of keys; the buffer of a growing text in heap snapshots.
  • Wrong answers: hashed substrings of a base that was widened, folded keys of constants, joins moved across handlers, flat programs in unicode mode, tail loops on long subjects.
  • Facts of the program: ways to RegExp.prototype.compile, the legacy statics and the matchAll() iterator that nobody noted.
  • Super-linear time: flat programs, image code, ropes and growing texts at run time; plans of literals and rehearsals at build time.
  • Some cleanup of error handling in the runtime, WTF and Yarr.

RegExp.prototype.compile now stops a program whose facts say that nobody calls it (ProgramTables::neverRecompilesRegExps), as the legacy statics do. A read under an unknown key off an instance whose prototype was changed is still not counted: counting it added about 800 KB to small programs. Such a program stops where it used to read out of bounds.

A loop that updates a global through a holder that is not expected to be globalThis (the root[key] of a UMD wrapper) takes the generic path: about 190 instructions a turn, where it took 14.

Needs a change in Bun

oven-sh/bun#49366 pins the preview build.

  • Two expectations under test/aot/from-source/walk/loop-invariants/ follow "a rope is resolved in front of a loop only where the first turn reads it": rope-behind-loop-exit.ir and reads-of-strings-in-front-of-loops-that-may-not-begin.ir.
  • StringImpl::convertTo{Lower,Upper}caseWithoutLocale() return RefPtr.
  • The compiler takes about 20 KB more, which passes the bound in test/bundler/compile-aot-size.test.ts.

Testing

Locally Linux x64 only, with the system's ICU 74. ARM64 runs in CI.

  • testaot all (linked with Bun's table): 5055 passed, 0 failed, 45 todo.
  • run-javascriptcore-tests --memory-limited, before the last three commits: 128,549 runs, 20 failures in four BigInt tests, which pass the 600 MB limit on this machine without the branch too. With them: the 13,668 runs of the RegExp and String stress tests pass.
  • Bun: all 22,557 programs of test/aot/parallel pass or are skipped (first build of each). Of 1,447 test files, six fail, and fail alike without the branch.
  • The new .ir tests fail without their fixes. Some fixes have no test.

Generated by Claude Code

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Findings marked 🟡 are optional suggestions and need no follow-up push.

Comment on lines +1 to +4
//@ memoryHog!
//@ runDefault("--useRegExpImageMode=1")

if (/~~[^]{1073741824}/.exec("ሴ".repeat(2 ** 30 + 64)) !== null)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 nit (optional): Anyone running the stress suite without --memory-limited gets a test that allocates a 2 GiB string and runs far past the 200 ms limit JSTests/README.md sets for new tests. JSTests/stress/regexp-image-start-search-large-minimum-size.js:4 builds "ሴ".repeat(2 ** 30 + 64) (16-bit, over 2 GiB) and scans it with /~~[^]{1073741824}/; memoryHog! only skips it under $memoryLimited and runs it exclusively otherwise. Fix: keep the new test within the README's 200 ms rule, for example by reducing the subject to the smallest size that still exercises the claimed > maxUnitsInDisplacement path in YarrJIT.cpp, or move it out of JSTests/stress.

Why this was flagged

JSTests/AGENTS.md imports JSTests/README.md, whose rule 1 under Adding Tests says new tests must run in under 200ms in all configurations. The new file JSTests/stress/regexp-image-start-search-large-minimum-size.js:4 evaluates "ሴ".repeat(2 ** 30 + 64), a 16-bit string of 2^30+64 code units (more than 2 GiB), then executes /~~[^]{1073741824}/.exec over it with --useRegExpImageMode=1. Allocating and filling 2 GiB alone takes well over 200 ms. Tools/Scripts/run-jsc-stress-tests:1003 defines memoryHog! as exclusive! plus skip only when $memoryLimited, so on a normal full run the test still executes and consumes the time and 2 GiB of memory. The PR's own test run used --memory-limited, under which this test was skipped, so the harness never timed it. The base branch has no such test; after merge every unrestricted stress run pays this cost.

Verification: JSTests/stress/regexp-image-start-search-large-minimum-size.js:1-4 is //@ memoryHog! + "ሴ".repeat(2 ** 30 + 64), slightly over 2 GiB, which cannot be built in 200 ms. Tools/Scripts/run-jsc-stress-tests:1003-1008 defines memoryHog! as skip only if $memoryLimited, so on non-memory-limited runs the test runs. JSTests/README.md:19 says "Tests must run in under 200ms in all configurations."

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left as it is. The path is only taken from a minimum size of 2^30 units on, so this is the smallest subject that reaches it. memoryHog! is what the other tests of that size in JSTests/stress use, and it skips the test under --memory-limited.


Generated by Claude Code

Comment on lines +705 to +707
else {
RefPtr result = convertToLowercaseWithoutLocale();
RELEASE_ASSERT(result);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) WebCore text-transform callers now hit RELEASE_ASSERT(result) at StringImpl.cpp:707 and :747 where the base returned the string unchanged. convertToUppercaseWithoutLocaleStartingAtFailingIndex8Bit used to return *this when m_length plus sharp-s count exceeded MaxLength (removed at :584-585) and the 16-bit paths returned *this on a late ICU failure; all of these now return nullptr (:585-586, :668-669) and the locale wrappers assert. Fix: in convertTo{Lower,Upper}caseWithLocale fall back to returning *this (or a null String) when the locale-less conversion gives null, keeping the base's non-crashing behaviour for RenderText.cpp:1295-1297.

Why this was flagged

String::convertToUppercaseWithLocale is called from RenderText.cpp:1295 for every text run with text-transform: uppercase, with a non-Turkish, non-Greek, non-Lithuanian locale, so it takes the else branch at StringImpl.cpp:745-749. If the text is 8-bit and holds sharp-s characters such that m_length plus numberSharpSCharacters exceeds MaxLength, the base returned *this (deleted check at the old :572-573); the new code at :584-586 returns nullptr and :747 fires RELEASE_ASSERT, killing the web process. Likewise u_strToUpper or u_strToLower failing after the preflight (:668-669, :484-485) used to return *this and now returns nullptr into the same assert. The population is any renderer or accessibility caller (AccessibilityObject.cpp:1352-1369) handed attacker-sized text, which is rare but not impossible with a 2 GB text node. Remedy: have the WithLocale wrappers keep the old fallback (return *this when the result is null) rather than asserting.

Verification: Trigger: a Latin-1 text run whose length plus its count of U+00DF exceeds MaxLength (2^31-1), rendered with text-transform: uppercase under a locale that is not tr/az/el/lt. Base StringImpl.cpp lines 572-573 returned *this; HEAD lines 584-586 return nullptr, and :745-749 RELEASE_ASSERT(result); turns the base's silently-unchanged result into a web process crash.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left as it is. The trace is right for WebCore, but that holds the only caller of these two wrappers and no lane builds it. Neither JavaScriptCore nor Bun calls them.


Generated by Claude Code

@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Preview build of 0875e08: autobuild-preview-pr-1083-0875e082

The out-of-line matcher of test() cast the maximum of a run to an int32,
so /^[^b]{0,3000000000}c/ never met its limit and read past the subject.
It also subtracted the longest length to the end from an address, which
wrapped. A maximum that no string can reach is now planned as unbounded,
and the skip is made only where it is shorter than the subjects searched.
…acement

On 16-bit subjects a minimum size of 2^30 or more overflowed the 32-bit
displacement of the vector test, which read 4 GB away from the subject.
Such a pattern takes the form of the search that moves the index.
Only the groups that the program reads were cleared when the object held
another pattern than its literal, but the array is made for all groups
and names of the literal. The others came from an older match, and the
word of a duplicate name was used as an index into the array.
… place

In f.apply(), Reflect.apply() and calls of bound functions the written
arguments are not at the places of the parameters. The pattern was taken
from another operand, so the wrong one matched, and groups that it
promised were cut out with the bounds of an older match.
Nothing follows what a closure does with a RegExp read from a private
field, so it could be recompiled or frozen while the compiler kept the
pattern and dropped the checks. Captures count as escapes for fields.
The bytecode interpreter clears the start of an absent group and leaves
its end. With a fitting end the comparison read at offset 0xFFFFFFFF.
…esolved

FlattenString gives the empty string for a rope unless all its conditions
hold, but a read moved to the preheader under the entry test alone. It was
held against the length of the rope and read off the empty string.
A test of receiver.length removed the bounds check of a character read,
also where the receiver is a number that ToString converts first.
Conditions made of the values that enter a loop were built among the
reloads and asked again behind each loud turn. Where they failed, the turn
read off the empty string with an index below the length of the rope.
The longest text has 2^31 pieces, which wrapped to a negative number under
a range that checks were removed by. The operation gives the place of the
last piece, and the walk adds one.
…arts

Errors format their stacks behind the sweep of that cache, and resolve
names of dead functions. The dead cell stayed in the cache, and a later
substring took the characters of whatever lived there by then.
The base may have adopted a 16-bit atom since the substring was made. The
stub compared bytes of the wide buffer and found keys that were not there.
…itive

o[K.name] was turned into an access under the text that one layout holds,
although K may be a string or a number, whose member is another key.
The room behind the text is memory that nobody has written, and it was part
of the name of the cell in a snapshot and of the inspector's preview.
A join in a try block that feeds a loop behind it failed where the loop
reads it, so the RangeError escaped the catch, which was dropped as dead.
Each join to a text that did not fit its buffer, or to a deep rope, copied
the whole text again, which takes quadratic time and memory.
The table of learned names is no root. The structure was read from it
behind the allocation of a record that was then refused.
A refused write to a built-in under a long key ended the process while it
built the message.
The short cut compares at every place, which is quadratic for a long text.
It bounded the depth alone, and needs more stack than an error handling
scope leaves.
An atomic run is taken from below the depth at which the walk stops, and
was matched by code units in unicode mode.
/()\1{4000000000}/ ran to the match limit at every match.
Seven walks were made for each term, alternative or text without a bound,
so compiling a pattern took time quadratic in its length.
It ran out of them on a long subject and reported no match.
…tself

Lengths found above primitives, lengths that change behind the price and
conversions that the engine makes were not paid for, so a function that is
never called kept the build from ending.
… nobody called it

A read under an unknown key off a list whose prototype is a RegExp finds
compile, while the facts say that no RegExp is compiled again. Groups were
then cut out with the bounds of an older match. The image now records the
fact, and compile checks it. The facts stay as they are, so no executable
grows.
They were looked through for ASCII and then copied. Another thread may
write in between, and bytes that are no UTF-8 came out as Latin-1.
…its class

The result of Object.create() has any prototype. RegExp.prototype.compile
was found through it while the facts said that no RegExp is compiled again.
Function.prototype.call.call(Reflect.get, ..) and __lookupGetter__ read
under keys that nobody noted, such as compile or the legacy statics.
It may read the record of the last match, which ended the process where
the facts said that nobody does.
The row said nothing of the class, so the iterator helpers that are called
on the result were left out of the executable.
Whoever climbs to it from a set of classes holds an unknown value, and
native code reads under any name. Reading $1 ended the process.
The match went on to the end of the text before it was thrown away, so a
global match took time quadratic in the length of the subject.
With a long addition at the bottom each join walked all earlier additions
and left the rope one level deeper.
endsWith(), a search for one unit and short slices resolved the whole rope,
where the engine walks down to the part that holds the range.
@sosukesuzuki
sosukesuzuki force-pushed the claude/elegant-darwin-6vxpab branch from ce9c280 to 4f28766 Compare November 21, 2026 02:14

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I re-reviewed the PR after the new commits and found no new bugs; given the size of the change (116 files, roughly 8,000 lines across Yarr, WTF string casing and the AOT walk/runtime) a human look is still worthwhile.

What was reviewed:

  • The flat-program required-text path: leastLengthLookedThroughFirst is gone and matchOutOfLine now always hands a constant budget of stepsBeforeLookingForText (9) to matchForgetting, so the pre-scan work no longer grows with the subject — this covers the concern I raised earlier on YarrFlatProgram.h.
  • The StringImpl::convertTo{Lower,Upper}caseWithoutLocale() change to RefPtr with tryCreateUninitialized and the ICU U_BUFFER_OVERFLOW_ERROR handling; the RELEASE_ASSERT in the WithLocale wrappers and the 2 GiB stress test are unchanged, so those two inline threads stay open as posted.
  • Not done: running or timing the new JSTests/stress files against the 200 ms rule in JSTests/README.md, since this review reads code only.
Extended reasoning...

The PR touches the Yarr flat-program matcher and builder, WTF StringImpl case conversion, LiteralParser, RegExp.cpp, and a large set of AOT codegen/walk/runtime files plus 40-odd new .ir answer files; it is memory-safety and regexp-engine work, not auth or injection surface. Since the prior review, over 60 commits were pushed; the step-budget concern from the earlier inline comment is addressed in the code, while the other two inline threads (test runtime and RELEASE_ASSERT fallback) are unchanged. This run's bug hunt ran dry with no findings. The size, the memory-safety scope and the two open threads decide defer over approve.

Still open from earlier reviews (2):

  • Unresolved: 2 minor or pre-existing.

Every hoisted address was asked on each turn whether it lies in the global
object. Now only an access whose holder is expected to be globalThis finds
a place there; behind any other the global object takes the generic path.
It was written out at every startsWith(), endsWith() and short slice, which
made programs 2% larger and a flat string slower to read.
…rst again

The look is bounded, and is left out where the first unit does not fit.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review found no issues

No high-confidence issues detected in this change.

Still open from earlier reviews (2):

  • Unresolved: 2 minor or pre-existing.

@sosukesuzuki
sosukesuzuki merged commit 59e584e into main Nov 21, 2026
89 checks passed
robobun added a commit to oven-sh/bun that referenced this pull request Nov 22, 2026
Not for merge. The preview build of oven-sh/WebKit#1048 has WebKit main in
it, and with it oven-sh/WebKit#1083, which Bun's main does not pin yet.
These are the three test changes that #49366 makes for that
engine. The commit goes when that pull request is in main.

Co-authored-by: SUZUKI Sosuke <14838850+sosukesuzuki@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant