Repository navigation
AOT: fixes for RegExp and String handling - #1083
Conversation
| //@ memoryHog! | ||
| //@ runDefault("--useRegExpImageMode=1") | ||
|
|
||
| if (/~~[^]{1073741824}/.exec("ሴ".repeat(2 ** 30 + 64)) !== null) |
There was a problem hiding this comment.
🟡 nit (optional): Anyone running the stress suite without --memory-limited gets a test that allocates a 2 GiB string and runs far past the 200 ms limit JSTests/README.md sets for new tests. JSTests/stress/regexp-image-start-search-large-minimum-size.js:4 builds "ሴ".repeat(2 ** 30 + 64) (16-bit, over 2 GiB) and scans it with /~~[^]{1073741824}/; memoryHog! only skips it under $memoryLimited and runs it exclusively otherwise. Fix: keep the new test within the README's 200 ms rule, for example by reducing the subject to the smallest size that still exercises the claimed > maxUnitsInDisplacement path in YarrJIT.cpp, or move it out of JSTests/stress.
Why this was flagged
JSTests/AGENTS.md imports JSTests/README.md, whose rule 1 under Adding Tests says new tests must run in under 200ms in all configurations. The new file JSTests/stress/regexp-image-start-search-large-minimum-size.js:4 evaluates "ሴ".repeat(2 ** 30 + 64), a 16-bit string of 2^30+64 code units (more than 2 GiB), then executes /~~[^]{1073741824}/.exec over it with --useRegExpImageMode=1. Allocating and filling 2 GiB alone takes well over 200 ms. Tools/Scripts/run-jsc-stress-tests:1003 defines memoryHog! as exclusive! plus skip only when $memoryLimited, so on a normal full run the test still executes and consumes the time and 2 GiB of memory. The PR's own test run used --memory-limited, under which this test was skipped, so the harness never timed it. The base branch has no such test; after merge every unrestricted stress run pays this cost.
Verification: JSTests/stress/regexp-image-start-search-large-minimum-size.js:1-4 is //@ memoryHog! + "ሴ".repeat(2 ** 30 + 64), slightly over 2 GiB, which cannot be built in 200 ms. Tools/Scripts/run-jsc-stress-tests:1003-1008 defines memoryHog! as skip only if $memoryLimited, so on non-memory-limited runs the test runs. JSTests/README.md:19 says "Tests must run in under 200ms in all configurations."
There was a problem hiding this comment.
Left as it is. The path is only taken from a minimum size of 2^30 units on, so this is the smallest subject that reaches it. memoryHog! is what the other tests of that size in JSTests/stress use, and it skips the test under --memory-limited.
Generated by Claude Code
| else { | ||
| RefPtr result = convertToLowercaseWithoutLocale(); | ||
| RELEASE_ASSERT(result); |
There was a problem hiding this comment.
🟡 (optional) WebCore text-transform callers now hit RELEASE_ASSERT(result) at StringImpl.cpp:707 and :747 where the base returned the string unchanged. convertToUppercaseWithoutLocaleStartingAtFailingIndex8Bit used to return *this when m_length plus sharp-s count exceeded MaxLength (removed at :584-585) and the 16-bit paths returned *this on a late ICU failure; all of these now return nullptr (:585-586, :668-669) and the locale wrappers assert. Fix: in convertTo{Lower,Upper}caseWithLocale fall back to returning *this (or a null String) when the locale-less conversion gives null, keeping the base's non-crashing behaviour for RenderText.cpp:1295-1297.
Why this was flagged
String::convertToUppercaseWithLocale is called from RenderText.cpp:1295 for every text run with text-transform: uppercase, with a non-Turkish, non-Greek, non-Lithuanian locale, so it takes the else branch at StringImpl.cpp:745-749. If the text is 8-bit and holds sharp-s characters such that m_length plus numberSharpSCharacters exceeds MaxLength, the base returned *this (deleted check at the old :572-573); the new code at :584-586 returns nullptr and :747 fires RELEASE_ASSERT, killing the web process. Likewise u_strToUpper or u_strToLower failing after the preflight (:668-669, :484-485) used to return *this and now returns nullptr into the same assert. The population is any renderer or accessibility caller (AccessibilityObject.cpp:1352-1369) handed attacker-sized text, which is rare but not impossible with a 2 GB text node. Remedy: have the WithLocale wrappers keep the old fallback (return *this when the result is null) rather than asserting.
Verification: Trigger: a Latin-1 text run whose length plus its count of U+00DF exceeds MaxLength (2^31-1), rendered with text-transform: uppercase under a locale that is not tr/az/el/lt. Base StringImpl.cpp lines 572-573 returned *this; HEAD lines 584-586 return nullptr, and :745-749 RELEASE_ASSERT(result); turns the base's silently-unchanged result into a web process crash.
There was a problem hiding this comment.
Left as it is. The trace is right for WebCore, but that holds the only caller of these two wrappers and no lane builds it. Neither JavaScriptCore nor Bun calls them.
Generated by Claude Code
|
Preview build of 0875e08: |
The out-of-line matcher of test() cast the maximum of a run to an int32,
so /^[^b]{0,3000000000}c/ never met its limit and read past the subject.
It also subtracted the longest length to the end from an address, which
wrapped. A maximum that no string can reach is now planned as unbounded,
and the skip is made only where it is shorter than the subjects searched.
…acement On 16-bit subjects a minimum size of 2^30 or more overflowed the 32-bit displacement of the vector test, which read 4 GB away from the subject. Such a pattern takes the form of the search that moves the index.
Only the groups that the program reads were cleared when the object held another pattern than its literal, but the array is made for all groups and names of the literal. The others came from an older match, and the word of a duplicate name was used as an index into the array.
… place In f.apply(), Reflect.apply() and calls of bound functions the written arguments are not at the places of the parameters. The pattern was taken from another operand, so the wrong one matched, and groups that it promised were cut out with the bounds of an older match.
Nothing follows what a closure does with a RegExp read from a private field, so it could be recompiled or frozen while the compiler kept the pattern and dropped the checks. Captures count as escapes for fields.
The bytecode interpreter clears the start of an absent group and leaves its end. With a fitting end the comparison read at offset 0xFFFFFFFF.
…esolved FlattenString gives the empty string for a rope unless all its conditions hold, but a read moved to the preheader under the entry test alone. It was held against the length of the rope and read off the empty string.
A test of receiver.length removed the bounds check of a character read, also where the receiver is a number that ToString converts first.
Conditions made of the values that enter a loop were built among the reloads and asked again behind each loud turn. Where they failed, the turn read off the empty string with an index below the length of the rope.
The longest text has 2^31 pieces, which wrapped to a negative number under a range that checks were removed by. The operation gives the place of the last piece, and the walk adds one.
…arts Errors format their stacks behind the sweep of that cache, and resolve names of dead functions. The dead cell stayed in the cache, and a later substring took the characters of whatever lived there by then.
The base may have adopted a 16-bit atom since the substring was made. The stub compared bytes of the wide buffer and found keys that were not there.
…itive o[K.name] was turned into an access under the text that one layout holds, although K may be a string or a number, whose member is another key.
The room behind the text is memory that nobody has written, and it was part of the name of the cell in a snapshot and of the inspector's preview.
A join in a try block that feeds a loop behind it failed where the loop reads it, so the RangeError escaped the catch, which was dropped as dead.
Each join to a text that did not fit its buffer, or to a deep rope, copied the whole text again, which takes quadratic time and memory.
The table of learned names is no root. The structure was read from it behind the allocation of a record that was then refused.
A refused write to a built-in under a long key ended the process while it built the message.
The short cut compares at every place, which is quadratic for a long text.
It bounded the depth alone, and needs more stack than an error handling scope leaves.
An atomic run is taken from below the depth at which the walk stops, and was matched by code units in unicode mode.
/()\1{4000000000}/ ran to the match limit at every match.
Seven walks were made for each term, alternative or text without a bound, so compiling a pattern took time quadratic in its length.
It ran out of them on a long subject and reported no match.
…tself Lengths found above primitives, lengths that change behind the price and conversions that the engine makes were not paid for, so a function that is never called kept the build from ending.
… nobody called it A read under an unknown key off a list whose prototype is a RegExp finds compile, while the facts say that no RegExp is compiled again. Groups were then cut out with the bounds of an older match. The image now records the fact, and compile checks it. The facts stay as they are, so no executable grows.
They were looked through for ASCII and then copied. Another thread may write in between, and bytes that are no UTF-8 came out as Latin-1.
…its class The result of Object.create() has any prototype. RegExp.prototype.compile was found through it while the facts said that no RegExp is compiled again.
Function.prototype.call.call(Reflect.get, ..) and __lookupGetter__ read under keys that nobody noted, such as compile or the legacy statics.
It may read the record of the last match, which ended the process where the facts said that nobody does.
The row said nothing of the class, so the iterator helpers that are called on the result were left out of the executable.
Whoever climbs to it from a set of classes holds an unknown value, and native code reads under any name. Reading $1 ended the process.
The match went on to the end of the text before it was thrown away, so a global match took time quadratic in the length of the subject.
With a long addition at the bottom each join walked all earlier additions and left the rope one level deeper.
endsWith(), a search for one unit and short slices resolved the whole rope, where the engine walks down to the part that holds the range.
ce9c280 to
4f28766
Compare
There was a problem hiding this comment.
I re-reviewed the PR after the new commits and found no new bugs; given the size of the change (116 files, roughly 8,000 lines across Yarr, WTF string casing and the AOT walk/runtime) a human look is still worthwhile.
What was reviewed:
- The flat-program required-text path:
leastLengthLookedThroughFirstis gone andmatchOutOfLinenow always hands a constant budget ofstepsBeforeLookingForText(9) tomatchForgetting, so the pre-scan work no longer grows with the subject — this covers the concern I raised earlier on YarrFlatProgram.h. - The
StringImpl::convertTo{Lower,Upper}caseWithoutLocale()change toRefPtrwithtryCreateUninitializedand the ICUU_BUFFER_OVERFLOW_ERRORhandling; theRELEASE_ASSERTin theWithLocalewrappers and the 2 GiB stress test are unchanged, so those two inline threads stay open as posted. - Not done: running or timing the new JSTests/stress files against the 200 ms rule in JSTests/README.md, since this review reads code only.
Extended reasoning...
The PR touches the Yarr flat-program matcher and builder, WTF StringImpl case conversion, LiteralParser, RegExp.cpp, and a large set of AOT codegen/walk/runtime files plus 40-odd new .ir answer files; it is memory-safety and regexp-engine work, not auth or injection surface. Since the prior review, over 60 commits were pushed; the step-budget concern from the earlier inline comment is addressed in the code, while the other two inline threads (test runtime and RELEASE_ASSERT fallback) are unchanged. This run's bug hunt ran dry with no findings. The size, the memory-safety scope and the two open threads decide defer over approve.
Still open from earlier reviews (2):
- Unresolved: 2 minor or pre-existing.
Every hoisted address was asked on each turn whether it lies in the global object. Now only an access whose holder is expected to be globalThis finds a place there; behind any other the global object takes the generic path.
It was written out at every startsWith(), endsWith() and short slice, which made programs 2% larger and a flat string slower to read.
…rst again The look is bounded, and is left out where the first unit does not fit.
Not for merge. The preview build of oven-sh/WebKit#1048 has WebKit main in it, and with it oven-sh/WebKit#1083, which Bun's main does not pin yet. These are the three test changes that #49366 makes for that engine. The commit goes when that pull request is in main. Co-authored-by: SUZUKI Sosuke <14838850+sosukesuzuki@users.noreply.github.com>
Fixes for RegExp and String handling in programs that are compiled ahead of time, one commit for each. The commit messages say what went wrong.
What changes
RegExp.prototype.compile, the legacy statics and thematchAll()iterator that nobody noted.RegExp.prototype.compilenow stops a program whose facts say that nobody calls it (ProgramTables::neverRecompilesRegExps), as the legacy statics do. A read under an unknown key off an instance whose prototype was changed is still not counted: counting it added about 800 KB to small programs. Such a program stops where it used to read out of bounds.A loop that updates a global through a holder that is not expected to be
globalThis(theroot[key]of a UMD wrapper) takes the generic path: about 190 instructions a turn, where it took 14.Needs a change in Bun
oven-sh/bun#49366 pins the preview build.
test/aot/from-source/walk/loop-invariants/follow "a rope is resolved in front of a loop only where the first turn reads it":rope-behind-loop-exit.irandreads-of-strings-in-front-of-loops-that-may-not-begin.ir.StringImpl::convertTo{Lower,Upper}caseWithoutLocale()returnRefPtr.test/bundler/compile-aot-size.test.ts.Testing
Locally Linux x64 only, with the system's ICU 74. ARM64 runs in CI.
testaot all(linked with Bun's table): 5055 passed, 0 failed, 45 todo.run-javascriptcore-tests --memory-limited, before the last three commits: 128,549 runs, 20 failures in four BigInt tests, which pass the 600 MB limit on this machine without the branch too. With them: the 13,668 runs of the RegExp and String stress tests pass.test/aot/parallelpass or are skipped (first build of each). Of 1,447 test files, six fail, and fail alike without the branch..irtests fail without their fixes. Some fixes have no test.Generated by Claude Code