diff --git a/docs/Reference/Contributed/tbRegExp/PatternSyntax.md b/docs/Reference/Contributed/tbRegExp/PatternSyntax.md new file mode 100644 index 00000000..cc80fa80 --- /dev/null +++ b/docs/Reference/Contributed/tbRegExp/PatternSyntax.md @@ -0,0 +1,98 @@ +--- +title: Pattern syntax +parent: tbRegExp Package +permalink: /tB/Packages/Contributed/tbRegExp/PatternSyntax +has_toc: false +--- + +# Pattern syntax +{: .no_toc } + +How a pattern is written for **tbRegExp**, and the few places where that needs saying. + +A pattern written for VBScript's **RegExp** needs no change. What follows either adds to that syntax or spells out a rule VBScript follows without documenting it. + +## What tbRegExp adds + +**tbRegExp** accepts, beyond VBScript's syntax: + +- named groups, `(?...)`: [**SubMatches.Item**](SubMatches/Item) returns one by its name, and [**Replace**](RegExp/Replace) inserts one as `$`. A name starts with a letter or an underscore and goes on with letters, digits and underscores; a pattern with any other name is not valid; +- lookbehind, `(?<=...)` and `(?\d{4})-(?\d{2})" +Debug.Print re.Execute("2026-10")(0).SubMatches("month") ' 10 +Debug.Print re.Replace("2026-10", "$/$") ' 10/2026 + +re.Pattern = "(?<=\$)\d+" +Debug.Print re.Execute("costs $123")(0).Value ' 123 + +re.Pattern = "(?i:abc)DEF" +Debug.Print re.Test("ABCDEF") ' True +Debug.Print re.Test("ABCdef") ' False +``` + +## Escapes + +The escapes that name a control character or a character code are read as in VBScript: + +- `\c` followed by a letter, in either case, is a control character: `\cJ` and `\CJ` are the line feed. A `\c` followed by anything else makes the pattern invalid; +- `\0` is the character with code 0, `\0` followed by octal digits, such as `\012`, is that code, and `\8` and `\9` are the digits themselves; +- `\1` to `\9` and longer numbers are backreferences when the group they name has already been opened, that is, when its `(` stands earlier in the pattern. Otherwise the digits are read as an octal character code, so `\101` is `A` and `(a)\2` is `a` followed by the character with code 2; [**IsValid**](RegExp/IsValid) returns **True** for both; +- `\f`, `\n`, `\r` and `\t`, inside a character class as well, are the form feed, line feed, carriage return and tab. + + +A backslash before a letter that names no escape stands for the letter itself, as in VBScript. The constructs that other engines spell that way are therefore ordinary text here, not errors: `\k` matches `k`, `\A` matches `A`, and `\p{L}` matches `p{L}`, the braces being text as well. + + +```tb check_build check_run +Dim re As New RegExp +re.Pattern = "\101" +Debug.Print re.Test("A") ' True +re.Pattern = "\CJ" +Debug.Print re.Test(vbLf) ' True +re.Pattern = "\k" +Debug.Print re.IsValid ' True +Debug.Print re.Test("k") ' True +re.Pattern = "(a)\2" +Debug.Print re.Test("a" & ChrW(2)) ' True +``` + + +Digits that name a group declared *later* in the pattern are therefore not a backreference at all. `\1(a)` is the character with code 1 followed by `a`, and finds no match in the string `a`, as in VBScript's **RegExp**. A group that is open but has not yet taken part, such as the `\1` in `(\1a)`, is a real backreference and matches no characters. + + +## Character classes + +`\w` matches only the ASCII letters and digits and `_`, as in VBScript, so `\w+` finds `caf` in `café`. + +## Quantifiers and braces + +A `{` starts a quantifier when it is followed by `{n}`, `{n,}` or `{n,m}`. A `{` that starts none, such as the one in `a{`, `a{x}` or `a{,2}`, is an ordinary character, as in VBScript; `\{` for the character is accepted too but not needed. A `]` or `}` that closes nothing is an ordinary character as well. + + +`{,m}` is text and not a quantifier, because a quantifier needs a count before the comma: `a{,2}` finds the text `a{,2}` and not up to two `a`. A `{` that does start a quantifier with counts that cannot be used is still an error, though: `a{2,1}`, whose upper count is below its lower one, makes the pattern invalid. + + +```tb check_build check_run +Dim re As New RegExp +re.Pattern = "(?<_part2>\d)" +Debug.Print re.IsValid ' True +re.Pattern = "(?<2part>\d)" +Debug.Print re.IsValid ' False + +re.Pattern = "a{x}" +Debug.Print re.IsValid ' True +Debug.Print re.Execute("a{x}")(0).Value ' a{x} +re.Pattern = "a{2,1}" +Debug.Print re.IsValid ' False +``` + +## See Also + +- [Pattern](RegExp/Pattern) property -- the regular expression +- [IsValid](RegExp/IsValid) property -- whether a pattern compiles +- [LastErrorNumber](RegExp/LastErrorNumber) property -- the faults a pattern can have, with an example of each +- [tbRegExp Package](./) -- overview diff --git a/docs/Reference/Contributed/tbRegExp/RegExp/Execute.md b/docs/Reference/Contributed/tbRegExp/RegExp/Execute.md index 30cfea8d..e9acb873 100644 --- a/docs/Reference/Contributed/tbRegExp/RegExp/Execute.md +++ b/docs/Reference/Contributed/tbRegExp/RegExp/Execute.md @@ -36,8 +36,8 @@ Debug.Print re.Execute("fooBarBaz").Count ' 2 Debug.Print re.Execute("fooBarBaz")(1).FirstIndex ' 6 ``` - -A search that backtracks without end, such as `(a+)+b` on thirty `a`, gives up instead of hanging. The collection then holds the matches found before it gave up, and [**SearchAborted**](SearchAborted) returns **True**. + +A search that backtracks without end, such as `(a+)+b` on thirty `a`, gives up instead of hanging, and **Execute** then raises an error. The matches found before it gave up are not returned: a part of the matches, handed back as though it were all of them, is worse than no answer. See [**SearchStepLimit**](SearchStepLimit). Each **Match** gives the matched text ([**Value**](../Match/Value)), the position where it starts ([**FirstIndex**](../Match/FirstIndex), counted from 0), its length, and the text of each group of the pattern ([**SubMatches**](../Match/SubMatches)). A group written `(?...)` can be read by its name. diff --git a/docs/Reference/Contributed/tbRegExp/RegExp/IsValid.md b/docs/Reference/Contributed/tbRegExp/RegExp/IsValid.md index 8929b93a..d8992c2a 100644 --- a/docs/Reference/Contributed/tbRegExp/RegExp/IsValid.md +++ b/docs/Reference/Contributed/tbRegExp/RegExp/IsValid.md @@ -24,12 +24,12 @@ The property compiles the pattern if it has not been compiled since [**Pattern** Check **IsValid** before [**Execute**](Execute), [**Test**](Test), [**Replace**](Replace) or [**Split**](Split) when the pattern comes from outside the program, because each of those raises the compile error. -> [!NOTE] 1.8.0.0 -> Although nothing is raised, the compile error of an invalid pattern stays in the **Err** object after the call that compiled the pattern returns. A program that tests **Err** for other errors calls **Err.Clear** first. +Nothing is raised and nothing is left behind. The **Err** object is clear when the property returns, even for a pattern that failed to compile, so a program that runs `On Error Resume Next` and tests **Err** for its own errors does not have to call **Err.Clear** first. The number and the message are kept in [**LastErrorNumber**](LastErrorNumber) and [**LastError**](LastError). + -> [!NOTE] 1.8.0.0 -> A quantifier whose lower count is above its upper count, such as `a{2,1}`, is not found when the pattern is compiled, so **IsValid** returns **True** for it and [**LastErrorNumber**](LastErrorNumber) returns 0. [**Execute**](Execute), [**Test**](Test), [**Replace**](Replace) and [**Split**](Split) then raise error 3000 when a search reaches the quantifier, with the description that Windows gives that number, *The specified print monitor is unknown.* +A quantifier whose lower count is above its upper count, such as `a{2,1}`, is found when the pattern is compiled, like any other fault: **IsValid** returns **False**, [**LastErrorNumber**](LastErrorNumber) gives the invalid-quantifier number, and [**Execute**](Execute), [**Test**](Test), [**Replace**](Replace) and [**Split**](Split) raise it. A `{` that starts no quantifier at all, such as the one in `a{x}`, is not a fault but an ordinary character. + ## Example @@ -37,9 +37,9 @@ Check **IsValid** before [**Execute**](Execute), [**Test**](Test), [**Replace**] Dim re As RegExp Set re = New RegExp re.Pattern = "(\d+" -Debug.Print re.IsValid ' False -Debug.Print Err.Number = re.LastErrorNumber ' True -Err.Clear +Debug.Print re.IsValid ' False +Debug.Print Err.Number ' 0 -- nothing left in Err +Debug.Print re.LastErrorNumber - vbObjectError ' 3009 re.Pattern = "(\d+)" Debug.Print re.IsValid ' True diff --git a/docs/Reference/Contributed/tbRegExp/RegExp/LastError.md b/docs/Reference/Contributed/tbRegExp/RegExp/LastError.md index 5935dc1a..c63e96aa 100644 --- a/docs/Reference/Contributed/tbRegExp/RegExp/LastError.md +++ b/docs/Reference/Contributed/tbRegExp/RegExp/LastError.md @@ -20,7 +20,7 @@ Returns a **String**: the description of the compile error, or an empty string w ## Remarks -The property compiles the pattern if it has not been compiled since [**Pattern**](Pattern) or [**IgnoreCase**](IgnoreCase) last changed. The compilation traps its error, so **LastError** raises nothing, but the error stays in the **Err** object (see [**IsValid**](IsValid)). The text is a sentence that names the fault, such as *Unterminated character class: a '[' without a matching ']'.* for the pattern `[abc`. [**LastErrorNumber**](LastErrorNumber) is the number of the error and lists the faults. +The property compiles the pattern if it has not been compiled since [**Pattern**](Pattern) or [**IgnoreCase**](IgnoreCase) last changed. The compilation traps its error, so **LastError** raises nothing and leaves the **Err** object clear. The text is a sentence that names the fault, such as *Unterminated character class: a '[' without a matching ']'.* for the pattern `[abc`. [**LastErrorNumber**](LastErrorNumber) is the number of the error and lists the faults. [**Execute**](Execute), [**Test**](Test), [**Replace**](Replace) and [**Split**](Split) raise an error with this description when the pattern is invalid. diff --git a/docs/Reference/Contributed/tbRegExp/RegExp/LastErrorNumber.md b/docs/Reference/Contributed/tbRegExp/RegExp/LastErrorNumber.md index 14204816..c07e97df 100644 --- a/docs/Reference/Contributed/tbRegExp/RegExp/LastErrorNumber.md +++ b/docs/Reference/Contributed/tbRegExp/RegExp/LastErrorNumber.md @@ -22,7 +22,7 @@ Returns a **Long**: the number of the compile error, or 0 when the pattern is va The property compiles the pattern if it has not been compiled since [**Pattern**](Pattern) or [**IgnoreCase**](IgnoreCase) last changed. The compilation traps its error, so **LastErrorNumber** raises nothing. The value is the `Number` of the error that the compilation raised; [**LastError**](LastError) is its description. -The engine raises its pattern errors with numbers of the form `vbObjectError + 3000 + n`, where *n* identifies the fault, as in the example. The compilation traps the error, so **LastErrorNumber** raises nothing, but the error stays in the **Err** object (see [**IsValid**](IsValid)). +The engine raises its pattern errors with numbers of the form `vbObjectError + 3000 + n`, where *n* identifies the fault, as in the example. The compilation traps the error, so **LastErrorNumber** raises nothing and leaves the **Err** object clear. | *n* | Fault | Example | @@ -32,15 +32,21 @@ The engine raises its pattern errors with numbers of the form `vbObjectError + 3 | 3 | A range in a character class that runs backwards | `[z-a]` | | 4 | A character class without its `]` | `[abc` | | 6 | A group construct `(?<` that is neither a lookbehind nor a valid name | `(?<1a>x)` | -| 8 | A `{` that does not start a quantifier, or a count that is too large | `a{2`, `\p{L}` | +| 8 | A quantifier whose upper count is below its lower count, or a count that is too large | `a{2,1}`, `a{99999999999}` | | 9 | A group without its `)` | `(\d+` | | 11 | A `)` that closes no group | `)` | | 12 | A quantifier with nothing to repeat | `*a` | | 13 | A group name with a character that a name cannot hold | `(?x)` | | 14 | An inline option that is not `i`, `m` or `s`, which includes an atomic group and a conditional | `(?x)`, `(?>a)` | + +An atomic group `(?>...)` and a conditional `(?(`*n*`)`*yes*`|`*no*`)` fall under *n* = 14. VBScript's **RegExp** does not accept either construct, so a pattern that uses one is invalid in both. + + [**Execute**](Execute), [**Test**](Test), [**Replace**](Replace) and [**Split**](Split) raise an error with this number when the pattern is invalid. +Those four methods can also raise `vbObjectError + 3016`, which is not a compile error and so never appears here: it means a valid pattern ran out of its step budget before the search finished. See [**SearchStepLimit**](SearchStepLimit). + ## Example ```tb check_build check_run diff --git a/docs/Reference/Contributed/tbRegExp/RegExp/Pattern.md b/docs/Reference/Contributed/tbRegExp/RegExp/Pattern.md index b14a1210..a27927f1 100644 --- a/docs/Reference/Contributed/tbRegExp/RegExp/Pattern.md +++ b/docs/Reference/Contributed/tbRegExp/RegExp/Pattern.md @@ -34,7 +34,7 @@ Syntax: *object*.**Pattern** **=** *value* Setting a different pattern makes it compile again the next time it is needed. Setting the current pattern does nothing. A pattern that does not compile raises no error when it is set: [**IsValid**](IsValid), [**LastError**](LastError) and [**LastErrorNumber**](LastErrorNumber) report the failure, and [**Execute**](Execute), [**Test**](Test), [**Replace**](Replace) and [**Split**](Split) raise it. -A named group is written `(?...)`. Its text is read from the [**SubMatches**](../SubMatches/) of a [**Match**](../Match/) by name, and used in [**Replace**](Replace) as `$`. A name starts with a letter or an underscore and goes on with letters, digits and underscores. The [overview of the package](../#pattern-syntax) lists the other differences from VBScript's pattern syntax. +A named group is written `(?...)`. Its text is read from the [**SubMatches**](../SubMatches/) of a [**Match**](../Match/) by name, and used in [**Replace**](Replace) as `$`. A name starts with a letter or an underscore and goes on with letters, digits and underscores. [**Pattern syntax**](../PatternSyntax) covers the rest of the pattern syntax. ## Example diff --git a/docs/Reference/Contributed/tbRegExp/RegExp/Replace.md b/docs/Reference/Contributed/tbRegExp/RegExp/Replace.md index 0d68f492..1fa00e03 100644 --- a/docs/Reference/Contributed/tbRegExp/RegExp/Replace.md +++ b/docs/Reference/Contributed/tbRegExp/RegExp/Replace.md @@ -35,13 +35,12 @@ A `$` in *replaceString* starts a substitution item. Any other text is copied as | `$'` | The part of *sourceString* after the match. | | `$`*n* or `$`*nn* | The text of numbered group *n*, counting from 1. Two digits are read as the number when the pattern has a group with that number, otherwise one. It is an empty string when the group matched nothing. When the pattern has no group *n*, the `$` is copied, and so are the digits: `$0` and `$00` are never items, and `$3` in a pattern of two groups is the text `$3`. | | `$<`*name*`>` | The text of the group written `(?...)`. It is an empty string when the group matched nothing or no group has the name. When the pattern has no named group, or the `>` is missing, or the name is empty, the `$` is copied. | -| `$~` | Nothing. The item is dropped. | A `$` that starts no item raises no error, so a replacement string can hold a literal `$` as it is, at the end or before any other character. Before a character that would make an item, write `$$`: `$$$&` inserts a `$` and then the matched text, as the last example does. -> [!NOTE] 1.8.0.0 -> Items are read as in VBScript's **RegExp**, with two differences: `$<`*name*`>` is an addition of this package, and `$~` is dropped, where VBScript copies `$~` into the result. +> [!NOTE] 1.9.0.0 +> Items are read as in VBScript's **RegExp**, with one difference: `$<`*name*`>` is an addition of this package. A `$~`, which earlier versions dropped, is copied into the result as VBScript copies it. ## Remarks @@ -57,8 +56,8 @@ re.Pattern = "^" Debug.Print Replace(re.Replace("a" & vbLf & "b" & vbLf & "c", "> "), vbLf, "/") ' > a/> b/> c ``` - -A search that backtracks without end, such as `(a+)+b` on thirty `a`, gives up instead of hanging, and **Replace** then replaces only the matches found before it gave up. [**SearchAborted**](SearchAborted) returns **True** in that case. + +A search that backtracks without end, such as `(a+)+b` on thirty `a`, gives up instead of hanging, and **Replace** then raises an error. No partly replaced string is returned. See [**SearchStepLimit**](SearchStepLimit). An invalid pattern raises the compile error, with the number in [**LastErrorNumber**](LastErrorNumber) and the description in [**LastError**](LastError). Check [**IsValid**](IsValid) first to avoid it. diff --git a/docs/Reference/Contributed/tbRegExp/RegExp/SearchAborted.md b/docs/Reference/Contributed/tbRegExp/RegExp/SearchAborted.md deleted file mode 100644 index b66672fe..00000000 --- a/docs/Reference/Contributed/tbRegExp/RegExp/SearchAborted.md +++ /dev/null @@ -1,79 +0,0 @@ ---- -title: SearchAborted -parent: RegExp -permalink: /tB/Packages/Contributed/tbRegExp/RegExp/SearchAborted -has_toc: false ---- - -# SearchAborted -{: .no_toc } - -Returns whether the last search of this object gave up before it finished. - -Syntax: *object*.**SearchAborted** - -*object* -: *required* An object expression that evaluates to a **RegExp**. - -Returns a **Boolean**: **True** when the most recent call of [**Test**](Test), [**Execute**](Execute), [**Replace**](Replace) or [**Split**](Split) on *object* stopped because its search ran out of steps, **False** otherwise. The property is read-only. It is **False** for an object that has not searched yet. - -## Remarks - -A search is given a budget of steps, where a step is one operation of the compiled pattern. A pattern that backtracks without end, such as `(a+)+b` on a string of thirty `a`, needs twice as many steps for each `a` it is given, and the budget is what stops it. VBScript's **RegExp** has no such limit and for that string does not return in any reasonable time. - -When the budget runs out, the search stops and nothing is raised. Whatever was found before is returned, and the result can be short or empty: [**Test**](Test) returns **False**, [**Execute**](Execute) returns the matches found before the search stopped, [**Replace**](Replace) returns *sourceString* with only those matches replaced, and [**Split**](Split) splits only at them. A program that cannot tell this from a string without a match reads **SearchAborted** after the call. - -The budget is 5,000 steps for each character of the string, at least 1,000,000 and at most 50,000,000. It is not a property of the object, and a program cannot change it. It applies to the search for one match, so with [**Global**](Global) set to **True** each match has a budget of its own, and the first search that runs out ends the call. A simple pattern needs about three steps for each character it matches, so a long match is not affected: `a*` on a string of 1,000,000 `a` returns all of it in about half a second. - -Each call of **Test**, **Execute**, **Replace** or **Split** sets the value again: a call that finishes its search sets it to **False**, and so does a call that raises an error because the pattern is not valid, because no search was made. The value belongs to the object. A search on another **RegExp** does not change it, and a new object returns **False**. - - -> [!NOTE] 1.8.0.0 -> A pattern that is not catastrophic can also run out of steps, if it tries again from each start position and the string is long. `\d+x` on a string of 5,000 digits and no `x` takes a few seconds and then gives up, with the same result as for a string that has no match, and **SearchAborted** returns **True**. A string of 2,000 digits finishes in a second or two and returns **False**. - -## Example - -This example searches a string of thirty `a` for `(a+)+b`, which cannot match and takes a very long time to prove it. The search gives up after about a fifth of a second, and **SearchAborted** says that the **False** from **Test** is not an answer. A search of a string that has no `b`, where the pattern is not catastrophic, finishes. - -```tb check_build check_run -Dim re As New RegExp -re.Pattern = "(a+)+b" -Debug.Print re.SearchAborted ' False - -Debug.Print re.Test(String(30, "a")) ' False -Debug.Print re.SearchAborted ' True - -Debug.Print re.Test("aaab") ' True -Debug.Print re.SearchAborted ' False - -re.Pattern = "a+b" -Debug.Print re.Test(String(30, "a")) ' False -Debug.Print re.SearchAborted ' False -``` - -This example shows that the value is per object, and that a search which gave up still returns the matches found before it. - -```tb check_build check_run -Dim re As New RegExp -Dim other As New RegExp -re.Pattern = "x|(a+)+b" -re.Global = True -other.Pattern = "a" - -Dim matches As MatchCollection -Set matches = re.Execute("xx" & String(30, "a")) -Debug.Print matches.Count ' 2 -Debug.Print re.SearchAborted ' True - -Debug.Print other.Test("a") ' True -Debug.Print other.SearchAborted ' False -Debug.Print re.SearchAborted ' True -``` - -## See Also - -- [Test](Test) method -- returns whether a string contains a match -- [Execute](Execute) method -- returns the matches -- [Replace](Replace) method -- replaces the matches -- [Split](Split) method -- splits a string at the matches -- [RegExp](./) class -- overview diff --git a/docs/Reference/Contributed/tbRegExp/RegExp/SearchStepLimit.md b/docs/Reference/Contributed/tbRegExp/RegExp/SearchStepLimit.md new file mode 100644 index 00000000..6a8a0d67 --- /dev/null +++ b/docs/Reference/Contributed/tbRegExp/RegExp/SearchStepLimit.md @@ -0,0 +1,91 @@ +--- +title: SearchStepLimit +parent: RegExp +permalink: /tB/Packages/Contributed/tbRegExp/RegExp/SearchStepLimit +has_toc: false +--- + +# SearchStepLimit +{: .no_toc } + +Returns or sets the number of steps a search of this object may take before it gives up. + +Syntax: *object*.**SearchStepLimit** [= *steps*] + +*object* +: *required* An object expression that evaluates to a **RegExp**. + +*steps* +: *optional* A **Long**, the budget for one search, or 0 for the budget the package works out on its own. A negative value raises error 5. + +Returns a **Long**: the value that was set, or 0 when the budget is left to the package. The default is 0. + +## Remarks + +A search is given a budget of steps, where a step is one operation of the compiled pattern. A pattern that backtracks without end, such as `(a+)+b` on a string of thirty `a`, needs twice as many steps for each `a` it is given, and the budget is what stops it. VBScript's **RegExp** has no such limit and for that string does not return in any reasonable time. + +When the budget runs out, [**Test**](Test), [**Execute**](Execute), [**Replace**](Replace) and [**Split**](Split) raise `vbObjectError + 3016` and return nothing at all -- not even the matches found before the search stopped. A search that ran out of steps never established what the rest of the string holds, so a short result is a wrong answer rather than an incomplete one, and the error is the only honest report. The description names the budget that was spent, the length of the string, and what to do about it. + + +With the property left at 0, the budget is ten steps for each character of the string squared, at least 1,000,000 and at most 100,000,000. It is worked out for the search of one match, so with [**Global**](Global) set to **True** each match is given a budget of its own and the first search that runs out ends the call. The square matters: an ordinary pattern that backtracks, such as `\d+x` on a string of digits with no `x`, costs about two and a quarter steps for each character squared, so it finishes at any length a search is likely to be given. A simple pattern needs about three steps for each character it matches, so a long match is never affected: `a*` on a string of 1,000,000 `a` returns all of it in about half a second. + + +Set the property when a pattern of that kind is given a string long enough to reach the cap. The message of the error suggests a value. Nothing covers a pattern whose cost grows faster than the square of the string, and when the budget already exceeded what a pattern of that kind would need, the message says to simplify the pattern instead of naming this property. + +The value that **SearchStepLimit** returns is the one that was set, not the budget a search would be given: with the property at 0 there is no single number to return, because the budget depends on the string. Setting it does not compile the pattern again. + +## The error + +The description of the error says what was spent and what to do about it. The search of a thousand digits in the second example below, given 100,000 steps when it needs about 2,250,000, reports: + +> The regular expression search was abandoned after 100,000 steps on an input of 1,000 characters, so whether it matches is not known. This pattern backtracks heavily at this length. A pattern like this typically needs about 3,000,000 steps at this length - roughly 30 times the work already done. Set **SearchStepLimit** to 3000000 to allow it. If that still fails, the pattern's cost grows faster than the square of the input and no limit will help. + +A pattern that is worse than quadratic is told the opposite, because for it a larger budget is not the answer. `(a+)+b` on thirty `a` is given the floor of 1,000,000 steps, far more than the 2,700 a quadratic pattern of that length would need, so the message does not name this property at all: + +> The regular expression search was abandoned after 1,000,000 steps on an input of 30 characters, so whether it matches is not known. This pattern backtracks heavily at this length. It was already allowed more than a quadratic pattern needs here, so its cost grows faster than the square of the input and no step limit will cover it. Simplify the pattern. + +## Example + +This example searches a string of thirty `a` for `(a+)+b`, which cannot match and takes a very long time to prove it. The search gives up after about a third of a second and raises. The same pattern against a string it can match finishes. + +```tb check_build check_run +Dim re As New RegExp +re.Pattern = "(a+)+b" + +On Error Resume Next +re.Test String(30, "a") +Debug.Print Err.Number - vbObjectError ' 3016 +On Error GoTo 0 +Err.Clear + +Debug.Print re.Test("aaab") ' True +``` + + +This example gives a search too few steps on purpose, then raises the limit so that the same search finishes. `\d+x` on a string of 1,000 digits with no `x` costs about 2,250,000 steps. + +```tb check_build check_run +Dim re As New RegExp +re.Pattern = "\d+x" +Debug.Print re.SearchStepLimit ' 0 + +re.SearchStepLimit = 100000 +On Error Resume Next +re.Test String(1000, "1") +Debug.Print Err.Number - vbObjectError ' 3016 +On Error GoTo 0 +Err.Clear + +re.SearchStepLimit = 3000000 +Debug.Print re.Test(String(1000, "1")) ' False +Debug.Print re.SearchStepLimit ' 3000000 +``` + + +## See Also + +- [Test](Test) method -- returns whether a string contains a match +- [Execute](Execute) method -- returns the matches +- [Replace](Replace) method -- replaces the matches +- [Split](Split) method -- splits a string at the matches +- [RegExp](./) class -- overview diff --git a/docs/Reference/Contributed/tbRegExp/RegExp/Split.md b/docs/Reference/Contributed/tbRegExp/RegExp/Split.md index b1563cfa..cc45d9f3 100644 --- a/docs/Reference/Contributed/tbRegExp/RegExp/Split.md +++ b/docs/Reference/Contributed/tbRegExp/RegExp/Split.md @@ -43,8 +43,8 @@ Debug.Print parts.Count ' 3 Debug.Print parts(1) & "|" & parts(2) & "|" & parts(3) ' foo|Bar|Baz ``` - -A search that backtracks without end, such as `(a+)+b` on thirty `a`, gives up instead of hanging, and **Split** then splits only at the matches found before it gave up. [**SearchAborted**](SearchAborted) returns **True** in that case. + +A search that backtracks without end, such as `(a+)+b` on thirty `a`, gives up instead of hanging, and **Split** then raises an error. No partly split collection is returned. See [**SearchStepLimit**](SearchStepLimit). When **Pattern** is not a valid regular expression, **Split** raises the compile error. Its number is [**LastErrorNumber**](LastErrorNumber), its description is [**LastError**](LastError), and its source is `RegExp`. Read [**IsValid**](IsValid) first to avoid the error. diff --git a/docs/Reference/Contributed/tbRegExp/RegExp/Test.md b/docs/Reference/Contributed/tbRegExp/RegExp/Test.md index 49358f94..3cd8cfaf 100644 --- a/docs/Reference/Contributed/tbRegExp/RegExp/Test.md +++ b/docs/Reference/Contributed/tbRegExp/RegExp/Test.md @@ -26,8 +26,8 @@ Returns **True** if [**Pattern**](Pattern) finds a match in *sourceString*, othe [**IgnoreCase**](IgnoreCase), [**MultiLine**](MultiLine) and [**DotAll**](DotAll) apply to the matching. [**Global**](Global) makes no difference, because one match is enough. - -A search that backtracks without end, such as `(a+)+b` on thirty `a`, gives up instead of hanging, and **Test** then returns **False**. [**SearchAborted**](SearchAborted) tells that **False** from a string that has no match. + +A search that backtracks without end, such as `(a+)+b` on thirty `a`, gives up instead of hanging, and **Test** then raises an error rather than returning a value: a search that ran out of steps did not establish whether the pattern matches, so there is no answer to return. See [**SearchStepLimit**](SearchStepLimit). When **Pattern** is not a valid regular expression, **Test** raises the compile error. Its number is [**LastErrorNumber**](LastErrorNumber), its description is [**LastError**](LastError), and its source is `RegExp`. Read [**IsValid**](IsValid) first to avoid the error. diff --git a/docs/Reference/Contributed/tbRegExp/RegExp/index.md b/docs/Reference/Contributed/tbRegExp/RegExp/index.md index 413728d0..8f6a616f 100644 --- a/docs/Reference/Contributed/tbRegExp/RegExp/index.md +++ b/docs/Reference/Contributed/tbRegExp/RegExp/index.md @@ -47,7 +47,7 @@ All four are **False** in a new instance. The pattern is compiled the first time a method or one of [**IsValid**](IsValid), [**LastError**](LastError) and [**LastErrorNumber**](LastErrorNumber) needs it. It is compiled again after [**Pattern**](Pattern) or [**IgnoreCase**](IgnoreCase) is set to a different value. [**Global**](Global), [**MultiLine**](MultiLine) and [**DotAll**](DotAll) are applied when a match is made and do not cause a new compilation. -A pattern that does not compile does not raise an error at once. [**IsValid**](IsValid) returns **False**, and [**LastError**](LastError) and [**LastErrorNumber**](LastErrorNumber) describe the failure. [**Execute**](Execute), [**Test**](Test), [**Replace**](Replace) and [**Split**](Split) then raise that error, with the number in **LastErrorNumber**, the description in **LastError** and the source `RegExp`. When **IsValid**, **LastError** or **LastErrorNumber** is the call that compiles an invalid pattern, it raises nothing, but the compile error stays in the **Err** object after it returns; [**IsValid**](IsValid) says what to do about it. +A pattern that does not compile does not raise an error at once. [**IsValid**](IsValid) returns **False**, and [**LastError**](LastError) and [**LastErrorNumber**](LastErrorNumber) describe the failure. [**Execute**](Execute), [**Test**](Test), [**Replace**](Replace) and [**Split**](Split) then raise that error, with the number in **LastErrorNumber**, the description in **LastError** and the source `RegExp`. When **IsValid**, **LastError** or **LastErrorNumber** is the call that compiles an invalid pattern, it raises nothing and leaves the **Err** object clear. ## Properties @@ -60,7 +60,7 @@ A pattern that does not compile does not raise an error at once. [**IsValid**](I - [LastErrorNumber](LastErrorNumber) -- returns the number of the compile error - [MultiLine](MultiLine) -- returns or sets whether `^` and `$` match at line boundaries - [Pattern](Pattern) -- returns or sets the regular expression -- [SearchAborted](SearchAborted) -- returns whether the last search gave up before it finished +- [SearchStepLimit](SearchStepLimit) -- returns or sets the step budget that stops a runaway search ## Methods diff --git a/docs/Reference/Contributed/tbRegExp/index.md b/docs/Reference/Contributed/tbRegExp/index.md index 231c2623..18fa0adb 100644 --- a/docs/Reference/Contributed/tbRegExp/index.md +++ b/docs/Reference/Contributed/tbRegExp/index.md @@ -4,109 +4,67 @@ parent: Contributed Packages has_toc: false exclude_from_docs: - StaticRegex -indexed_from: 1.8.0.0 +indexed_from: 1.9.0.0 permalink: /tB/Packages/Contributed/tbRegExp/ --- # tbRegExp Package {: .no_toc } -**Author:** GCuser99. **Version:** 1.8.0.0. **On GitHub:** [GCuser99/tbRegExp](https://github.com/GCuser99/tbRegExp), in the folder *Package*. +**Author:** GCuser99. **Version:** 1.9.0.0. **On GitHub:** [GCuser99/tbRegExp](https://github.com/GCuser99/tbRegExp), in the folder *Package*. The **tbRegExp** contributed package provides regular expressions for twinBASIC projects, as a replacement for the **RegExp** object of VBScript. It follows the same object model. A [**RegExp**](RegExp/) is given a pattern and options, its [**Execute**](RegExp/Execute) method returns a [**MatchCollection**](MatchCollection/) of [**Match**](Match/) objects, and each **Match** holds the text captured by the groups of the pattern as a [**SubMatches**](SubMatches/) object. Positions are zero-based, as in VBScript. The matching is done by the [vba-regex](https://github.com/sihlfall/vba-regex) engine of sihlfall, which the package includes as a private module, **StaticRegex**. -Beyond VBScript's object, **RegExp** has the [**DotAll**](RegExp/DotAll) and [**Flags**](RegExp/Flags) options, the [**Split**](RegExp/Split) method, [**IsValid**](RegExp/IsValid), [**LastError**](RegExp/LastError) and [**LastErrorNumber**](RegExp/LastErrorNumber), which report an invalid pattern before it is used, and [**SearchAborted**](RegExp/SearchAborted), which reports a search that gave up. [**Replace**](RegExp/Replace) reads a `$` in the replacement string as VBScript does, and adds `$` for a named group. - -## Pattern syntax - -A pattern is written as for VBScript's **RegExp**. **tbRegExp** also accepts: - -- named groups, `(?...)`: [**SubMatches.Item**](SubMatches/Item) returns one by its name, and [**Replace**](RegExp/Replace) inserts one as `$`. A name starts with a letter or an underscore and goes on with letters, digits and underscores; a pattern with any other name is not valid; -- lookbehind, `(?<=...)` and `(?\d{4})-(?\d{2})" -Debug.Print re.Execute("2026-10")(0).SubMatches("month") ' 10 -Debug.Print re.Replace("2026-10", "$/$") ' 10/2026 - -re.Pattern = "(?<=\$)\d+" -Debug.Print re.Execute("costs $123")(0).Value ' 123 - -re.Pattern = "(?i:abc)DEF" -Debug.Print re.Test("ABCDEF") ' True -Debug.Print re.Test("ABCdef") ' False +re.Pattern = "\d+" +re.Global = True +Debug.Print re.Test("a1b22c333") ' True +Debug.Print re.Execute("a1b22c333").Count ' 3 +Debug.Print re.Execute("a1b22c333")(1).Value ' 22 +Debug.Print re.Replace("a1b22c333", "#") ' a#b#c# ``` - -> [!NOTE] 1.8.0.0 -> A pattern that holds an atomic group `(?>...)` or a conditional is not valid: [**IsValid**](RegExp/IsValid) returns **False**. A named backreference `\k`, `\A`, `\Z` and `\p{...}` are not supported either, but they do not make the pattern invalid, because a backslash before a letter that names no escape stands for the letter itself, as in VBScript: `\k` matches the text `k`, and `\A` matches `A`. The pattern `\p{L}` is invalid for another reason: `{L}` does not start a quantifier (see below). +Beyond VBScript's object, **RegExp** has the [**DotAll**](RegExp/DotAll) and [**Flags**](RegExp/Flags) options, the [**Split**](RegExp/Split) method, [**IsValid**](RegExp/IsValid), [**LastError**](RegExp/LastError) and [**LastErrorNumber**](RegExp/LastErrorNumber), which report an invalid pattern before it is used, and [**SearchStepLimit**](RegExp/SearchStepLimit), which bounds a runaway search. [**Replace**](RegExp/Replace) reads a `$` in the replacement string as VBScript does, and adds `$` for a named group. -The escapes that name a control character or a character code are read as in VBScript: +## Pattern syntax -- `\c` followed by a letter, in either case, is a control character: `\cJ` and `\CJ` are the line feed. A `\c` followed by anything else makes the pattern invalid; -- `\0` is the character with code 0, `\0` followed by octal digits, such as `\012`, is that code, and `\8` and `\9` are the digits themselves; -- `\1` to `\9` and longer numbers are backreferences when the pattern has that many groups. Otherwise the digits are read as an octal character code, so `\101` is `A` and `(a)\2` is `a` followed by the character with code 2; [**IsValid**](RegExp/IsValid) returns **True** for both; -- `\f`, `\n`, `\r` and `\t`, inside a character class as well, are the form feed, line feed, carriage return and tab. +A pattern written for VBScript's **RegExp** needs no change. **tbRegExp** also accepts named groups, lookbehind, and the `i`, `m` and `s` options inside the pattern: ```tb check_build check_run Dim re As New RegExp -re.Pattern = "\101" -Debug.Print re.Test("A") ' True -re.Pattern = "\CJ" -Debug.Print re.Test(vbLf) ' True -re.Pattern = "\k" -Debug.Print re.IsValid ' True -Debug.Print re.Test("k") ' True -re.Pattern = "(a)\2" -Debug.Print re.Test("a" & ChrW(2)) ' True +re.Pattern = "(?\d{4})-(?\d{2})" +Debug.Print re.Execute("2026-10")(0).SubMatches("month") ' 10 +Debug.Print re.Replace("2026-10", "$/$") ' 10/2026 ``` - -> [!NOTE] 1.8.0.0 -> A backreference to a group that has not yet taken part in the match, such as `\1(a)`, matches no characters, so the pattern finds `a` in the string `a`. VBScript's **RegExp** finds no match for such a pattern. - -`\w` matches only the ASCII letters and digits and `_`, as in VBScript, so `\w+` finds `caf` in `café`. - -A `{` has to start a quantifier: `{n}`, `{n,}`, `{n,m}` or `{,m}`, which means `{0,m}`. A pattern in which a `{` starts none, such as `a{` or `a{x}`, is not valid; write `\{` for the character. A `]` or `}` that closes nothing is an ordinary character, as in VBScript. - - -> [!NOTE] 1.8.0.0 -> VBScript's **RegExp** reads a `{` that starts no quantifier, and `{,m}`, as ordinary text: `a{x}` is a valid pattern there that finds the text `a{x}`, and `a{,2}` finds the text `a{,2}` and not `aa`. In **tbRegExp** the first is not valid, so a search with it raises the compile error, and `a{,2}` finds up to two `a`. - -```tb check_build check_run -Dim re As New RegExp -re.Pattern = "(?<_part2>\d)" -Debug.Print re.IsValid ' True -re.Pattern = "(?<2part>\d)" -Debug.Print re.IsValid ' False - -re.Pattern = "a{x}" -Debug.Print re.IsValid ' False -re.Pattern = "a\{x}" -Debug.Print re.Execute("a{x}")(0).Value ' a{x} -``` +[**Pattern syntax**](PatternSyntax) covers those in full, along with the escapes, the character classes and the brace rules. ## Searches that give up - -A pattern that backtracks without end, such as `(a+)+b` on a string of thirty `a`, does not hang a search as it does in VBScript's **RegExp**. Every search has a budget of steps, which a program cannot change, and when it runs out the search stops and raises no error. Its result is then short or empty, and a program that has to tell it from a true result reads [**SearchAborted**](RegExp/SearchAborted) after the call. A long match is no problem: `a*` and `^.*$` match a string of 1,000,000 `a` whole. +This is the one place where **tbRegExp** behaves differently from VBScript's **RegExp** rather than adding to it. + + +A pattern that backtracks without end, such as `(a+)+b` on a string of thirty `a`, does not hang a search as it does in VBScript's **RegExp**. Every search is given a budget of steps, and when it runs out the search raises an error rather than returning a result it never established. A long match is no problem: `a*` and `^.*$` match a string of 1,000,000 `a` whole. ```tb check_build check_run Dim re As New RegExp re.Pattern = "(a+)+b" -Debug.Print re.Test(String(30, "a")) ' False -Debug.Print re.SearchAborted ' True +On Error Resume Next +re.Test String(30, "a") +Debug.Print Err.Number - vbObjectError ' 3016 +On Error GoTo 0 +Err.Clear re.Pattern = "^.*$" Debug.Print re.Test(String(100000, "a")) ' True -Debug.Print re.SearchAborted ' False ``` +[**SearchStepLimit**](RegExp/SearchStepLimit) explains the budget, shows the message, and raises the ceiling when a search genuinely needs more. + ## Classes - [Match](Match/) -- holds one match that a RegExp found in a string: the matched text, where it starts, how many characters it has, and the text captured by its groups @@ -130,7 +88,7 @@ Debug.Print re.SearchAborted ' False - [MultiLine](RegExp/MultiLine) -- returns or sets whether `^` and `$` match at line boundaries - [Pattern](RegExp/Pattern) -- returns or sets the regular expression - [Replace](RegExp/Replace) -- replaces the matches in a string - - [SearchAborted](RegExp/SearchAborted) -- returns whether the last search gave up before it finished + - [SearchStepLimit](RegExp/SearchStepLimit) -- returns or sets the step budget that stops a runaway search - [Split](RegExp/Split) -- splits a string at the matches - [Test](RegExp/Test) -- returns whether a string contains a match - [SubMatches](SubMatches/) -- holds the strings that the capture groups of a pattern captured in one Match diff --git a/test/package-tests/tbRegExp/Sources/Expect.twin b/test/package-tests/tbRegExp/Sources/Expect.twin new file mode 100644 index 00000000..59f5c2f4 --- /dev/null +++ b/test/package-tests/tbRegExp/Sources/Expect.twin @@ -0,0 +1,155 @@ +' Expect: the checks every test case of the twinBASIC documentation's package tests makes. +' Part of each test project; https://docs.twinbasic.com. MIT licence. +' +' A case is a [TestCase] Sub of a [TestFixture] Module. It starts with Expect.Begin, checks with +' Expect.AreEqual, Expect.IsTrue, Expect.IsFalse and Expect.Fail, and ends with Expect.Done. Each +' check states what is correct, so a case that fails names a defect of the package. A failed check +' does not stop the case: Expect.Done reports every failed check of the case at once. +' +' Run the project (Sub Main in the module RunAll) to run every case: each one prints PASS, FAIL or +' SKIP in the DEBUG CONSOLE. A test runner that runs one case at a time sees a failed case as a +' failed assertion, which the case's own On Error statement cannot hide. +Module Expect + + Private mChecks As Long + Private mFailures As Long + Private mText As String + Private mCollecting As Boolean + Private mReachedDone As Boolean + Private mPassed As Long + Private mFailed As Long + Private mSkipped As Long + + ' Starts a case: forgets the checks of the case before it. + Public Sub Begin() + mChecks = 0 + mFailures = 0 + mText = "" + End Sub + + ' Checks that actual is expected. A string equals only a string, Empty only Empty, Null only + ' Null and an object only the same object; numbers, dates and Booleans compare by value. + Public Sub AreEqual(ByVal Expected As Variant, ByVal Actual As Variant, ByVal What As String) + mChecks = mChecks + 1 + If Not Same(Expected, Actual) Then Record What & ": expected " & Show(Expected) & ", got " & Show(Actual) + End Sub + + Public Sub IsTrue(ByVal Condition As Boolean, ByVal What As String) + mChecks = mChecks + 1 + If Not Condition Then Record What & ": expected True, got False" + End Sub + + Public Sub IsFalse(ByVal Condition As Boolean, ByVal What As String) + mChecks = mChecks + 1 + If Condition Then Record What & ": expected False, got True" + End Sub + + ' A failed check with no values, for a place the case must not reach. + Public Sub Fail(ByVal What As String) + mChecks = mChecks + 1 + Record What + End Sub + + ' Ends a case: reports the failed checks, if any. + Public Sub Done() + Dim text As String + If mFailures > 0 Then text = mFailures & " of " & mChecks & " checks failed: " & mText + Begin + If mCollecting Then + mReachedDone = True + mText = text + ElseIf Len(text) > 0 Then + Assert.Strict.Fail text + End If + End Sub + + ' ---- used by RunAll + + Public Sub StartRun(ByVal Title As String) + mCollecting = True + mPassed = 0 + mFailed = 0 + mSkipped = 0 + Debug.Print Title + End Sub + + Public Sub StartCase() + Begin + mReachedDone = False + End Sub + + ' What the case's call left: Done's report, or the error that ended it before Done. + Public Sub EndCase(ByVal Name As String, ByVal ErrNumber As Long, ByVal ErrDescription As String) + Dim outcome As String + If mReachedDone Then + outcome = mText + ElseIf ErrNumber <> 0 Then + outcome = "run-time error " & ErrNumber & ": " & ErrDescription + Else + outcome = "the case ended before Expect.Done" + End If + If Len(outcome) = 0 Then + mPassed = mPassed + 1 + Debug.Print "PASS " & Name + Else + mFailed = mFailed + 1 + Debug.Print "FAIL " & Name & ": " & outcome + End If + Begin + End Sub + + Public Sub Skip(ByVal Name As String, ByVal Reason As String) + mSkipped = mSkipped + 1 + Debug.Print "SKIP " & Name & ": " & Reason + End Sub + + Public Sub EndRun() + Debug.Print mPassed & " passed, " & mFailed & " failed, " & mSkipped & " skipped" + mCollecting = False + End Sub + + ' ---- inside + + Private Sub Record(ByVal Text As String) + mFailures = mFailures + 1 + If Len(mText) > 0 Then mText = mText & "; " + mText = mText & Text + End Sub + + Private Function Same(ByVal A As Variant, ByVal B As Variant) As Boolean + If IsObject(A) Or IsObject(B) Then + If IsObject(A) And IsObject(B) Then Same = (A Is B) + ElseIf IsArray(A) Or IsArray(B) Then + Same = False + ElseIf IsNull(A) Or IsNull(B) Then + Same = IsNull(A) And IsNull(B) + ElseIf IsEmpty(A) Or IsEmpty(B) Then + Same = IsEmpty(A) And IsEmpty(B) + ElseIf VarType(A) = vbString Or VarType(B) = vbString Then + If VarType(A) = vbString And VarType(B) = vbString Then Same = (StrComp(A, B, vbBinaryCompare) = 0) + Else + Same = (A = B) + End If + End Function + + Private Function Show(ByVal V As Variant) As String + If IsObject(V) Then + If V Is Nothing Then Show = "Nothing" Else Show = "an object of type " & TypeName(V) + ElseIf IsArray(V) Then + Show = "an array of type " & TypeName(V) + ElseIf IsNull(V) Then + Show = "Null" + ElseIf IsEmpty(V) Then + Show = "Empty" + ElseIf VarType(V) = vbString Then + If Len(V) > 120 Then + Show = """" & Left$(V, 100) & """... (" & Len(V) & " characters)" + Else + Show = """" & V & """" + End If + Else + Show = CStr(V) + End If + End Function + +End Module diff --git a/test/package-tests/tbRegExp/Sources/RunAll.twin b/test/package-tests/tbRegExp/Sources/RunAll.twin index 8a724f12..4fe27c9e 100644 --- a/test/package-tests/tbRegExp/Sources/RunAll.twin +++ b/test/package-tests/tbRegExp/Sources/RunAll.twin @@ -1,11 +1,11 @@ ' Generated by scripts/package_tests.mjs from this project's [TestCase] Subs and expect.json: ' do not edit. Run Sub Main: each case prints PASS, FAIL or SKIP in the DEBUG CONSOLE. -' The cases are written for the package tbRegExp, version 1.8.0.0. Each one checks what is +' The cases are written for the package tbRegExp, version 1.9.0.0. Each one checks what is ' correct, so a case that fails names a defect of the package. Module RunAll Public Sub Main() - Expect.StartRun "tbRegExp 1.8.0.0: 31 test cases" + Expect.StartRun "tbRegExp 1.9.0.0: 32 test cases" On Error Resume Next Expect.StartCase Err.Clear @@ -101,28 +101,32 @@ Module RunAll Expect.EndCase "ReplaceTests.Replace_TildeItemIsCopied", Err.Number, Err.Description Expect.StartCase Err.Clear - SearchAbortedTests.SearchAborted_DigitsWithoutXFinishAt2000 - Expect.EndCase "SearchAbortedTests.SearchAborted_DigitsWithoutXFinishAt2000", Err.Number, Err.Description + SearchFinishesTests.Search_DigitsWithoutXFinishAt2000 + Expect.EndCase "SearchFinishesTests.Search_DigitsWithoutXFinishAt2000", Err.Number, Err.Description Expect.StartCase Err.Clear - SearchAbortedTests.SearchAborted_DigitsWithoutXFinishAt5000 - Expect.EndCase "SearchAbortedTests.SearchAborted_DigitsWithoutXFinishAt5000", Err.Number, Err.Description + SearchFinishesTests.Search_DigitsWithoutXFinishAt5000 + Expect.EndCase "SearchFinishesTests.Search_DigitsWithoutXFinishAt5000", Err.Number, Err.Description Expect.StartCase Err.Clear - SearchGivesUpTests.Test_CatastrophicPatternGivesUpWithoutError - Expect.EndCase "SearchGivesUpTests.Test_CatastrophicPatternGivesUpWithoutError", Err.Number, Err.Description + SearchGivesUpTests.Test_CatastrophicPatternRaises + Expect.EndCase "SearchGivesUpTests.Test_CatastrophicPatternRaises", Err.Number, Err.Description Expect.StartCase Err.Clear - SearchGivesUpTests.Execute_CatastrophicPatternKeepsMatchesFound - Expect.EndCase "SearchGivesUpTests.Execute_CatastrophicPatternKeepsMatchesFound", Err.Number, Err.Description + SearchGivesUpTests.Execute_CatastrophicPatternKeepsNothing + Expect.EndCase "SearchGivesUpTests.Execute_CatastrophicPatternKeepsNothing", Err.Number, Err.Description Expect.StartCase Err.Clear - SearchGivesUpTests.Replace_CatastrophicPatternReplacesMatchesFound - Expect.EndCase "SearchGivesUpTests.Replace_CatastrophicPatternReplacesMatchesFound", Err.Number, Err.Description + SearchGivesUpTests.Replace_CatastrophicPatternKeepsNothing + Expect.EndCase "SearchGivesUpTests.Replace_CatastrophicPatternKeepsNothing", Err.Number, Err.Description Expect.StartCase Err.Clear - SearchGivesUpTests.Split_CatastrophicPatternSplitsAtMatchesFound - Expect.EndCase "SearchGivesUpTests.Split_CatastrophicPatternSplitsAtMatchesFound", Err.Number, Err.Description + SearchGivesUpTests.Split_CatastrophicPatternKeepsNothing + Expect.EndCase "SearchGivesUpTests.Split_CatastrophicPatternKeepsNothing", Err.Number, Err.Description + Expect.StartCase + Err.Clear + SearchGivesUpTests.SearchStepLimit_RaisesTheCeiling + Expect.EndCase "SearchGivesUpTests.SearchStepLimit_RaisesTheCeiling", Err.Number, Err.Description Expect.StartCase Err.Clear SearchGivesUpTests.Search_LongMatchOfAStarIsNotCut diff --git a/test/package-tests/tbRegExp/Sources/SearchAbortedTests.twin b/test/package-tests/tbRegExp/Sources/SearchAbortedTests.twin deleted file mode 100644 index b81b661a..00000000 --- a/test/package-tests/tbRegExp/Sources/SearchAbortedTests.twin +++ /dev/null @@ -1,27 +0,0 @@ -' tbRegExp: RegExp.SearchAborted for a pattern that is not catastrophic. A search that only has to -' try each start position finishes, however long the string; SearchAborted is for a search that -' backtracks without end (see SearchGivesUpTests). -[TestFixture] -Module SearchAbortedTests - - [TestCase] - Public Sub SearchAborted_DigitsWithoutXFinishAt2000() - Expect.Begin - Dim re As New RegExp - re.Pattern = "\d+x" - Expect.IsFalse re.Test(String$(2000, "1")), "Test of \d+x on 2,000 digits and no x" - Expect.IsFalse re.SearchAborted, "SearchAborted after Test of \d+x on 2,000 digits and no x" - Expect.Done - End Sub - - [TestCase] - Public Sub SearchAborted_DigitsWithoutXFinishAt5000() - Expect.Begin - Dim re As New RegExp - re.Pattern = "\d+x" - Expect.IsFalse re.Test(String$(5000, "1")), "Test of \d+x on 5,000 digits and no x" - Expect.IsFalse re.SearchAborted, "SearchAborted after Test of \d+x on 5,000 digits and no x" - Expect.Done - End Sub - -End Module diff --git a/test/package-tests/tbRegExp/Sources/SearchFinishesTests.twin b/test/package-tests/tbRegExp/Sources/SearchFinishesTests.twin new file mode 100644 index 00000000..5c3a5b47 --- /dev/null +++ b/test/package-tests/tbRegExp/Sources/SearchFinishesTests.twin @@ -0,0 +1,39 @@ +' tbRegExp: a quadratic search finishes. The step budget scales with the SQUARE of the input +' length, because ordinary backtracking does: '\d+x' over n digits with no x costs about 2.25*n^2 +' steps, since \d+ eats to the end from every start position and backtracks the whole way. A +' budget linear in n loses to that for large enough n whatever constant is chosen, so only the +' crossover moves. Only a pattern whose cost grows FASTER than n^2 is stopped - see +' SearchGivesUpTests. +' +' (Renamed from SearchAbortedTests: RegExp.SearchAborted no longer exists. Running out of budget +' raises, so there is no silently-returned value left for a flag to describe.) +[TestFixture] +Module SearchFinishesTests + + [TestCase] + Public Sub Search_DigitsWithoutXFinishAt2000() + Expect.Begin + Dim re As New RegExp + re.Pattern = "\d+x" + Err.Clear + Expect.IsFalse re.Test(String$(2000, "1")), "Test of \d+x on 2,000 digits and no x" + Expect.AreEqual 0, Err.Number, "Err.Number after Test of \d+x on 2,000 digits" + Expect.Done + End Sub + + ' The case the linear budget failed. It got 5000 * 5000 = 25,000,000 steps against the + ' ~56,000,000 this costs, and gave up. The quadratic budget allows 10 * 5000^2, capped at + ' 100,000,000, so it now completes - and the old 50,000,000 cap was itself below what this + ' needs, which is why raising the per-character constant alone could never have fixed it. + [TestCase] + Public Sub Search_DigitsWithoutXFinishAt5000() + Expect.Begin + Dim re As New RegExp + re.Pattern = "\d+x" + Err.Clear + Expect.IsFalse re.Test(String$(5000, "1")), "Test of \d+x on 5,000 digits and no x" + Expect.AreEqual 0, Err.Number, "Err.Number after Test of \d+x on 5,000 digits" + Expect.Done + End Sub + +End Module diff --git a/test/package-tests/tbRegExp/Sources/SearchGivesUpTests.twin b/test/package-tests/tbRegExp/Sources/SearchGivesUpTests.twin index a255ceac..b93550f5 100644 --- a/test/package-tests/tbRegExp/Sources/SearchGivesUpTests.twin +++ b/test/package-tests/tbRegExp/Sources/SearchGivesUpTests.twin @@ -1,57 +1,125 @@ -' tbRegExp: a search that backtracks without end, such as (a+)+b on thirty a, gives up when its -' step budget runs out instead of hanging, as VBScript's RegExp does. It raises nothing, returns -' what was found before, and RegExp.SearchAborted says so. A long match is not cut short. +' tbRegExp: a search whose cost grows faster than the square of the input - (a+)+b on thirty a - +' is stopped by its step budget instead of hanging, as VBScript's RegExp does. It RAISES. Running +' out of budget means the answer is not known, so nothing is returned and no partial result is +' kept: reporting "no match" would assert something the search never established. The error says +' what to try, and names SearchStepLimit when raising the limit could plausibly help. A long match +' is not cut short. [TestFixture] Module SearchGivesUpTests + ' The engine's error enum is private to the package, so this is spelled out here. + Private Const ERR_SEARCH_ABANDONED As Long = vbObjectError + 3016 + [TestCase] - Public Sub Test_CatastrophicPatternGivesUpWithoutError() + Public Sub Test_CatastrophicPatternRaises() Expect.Begin Dim re As New RegExp + Dim num As Long, desc As String re.Pattern = "(a+)+b" + On Error Resume Next + Err.Clear + re.Test String$(30, "a") + num = Err.Number: desc = Err.Description + On Error GoTo 0 Err.Clear - Expect.IsFalse re.SearchAborted, "SearchAborted of an object that has not searched" - Expect.IsFalse re.Test(String$(30, "a")), "Test of (a+)+b on thirty a" - Expect.AreEqual 0, Err.Number, "Err.Number after the search gave up" - Expect.IsTrue re.SearchAborted, "SearchAborted after Test of (a+)+b on thirty a" + Expect.AreEqual ERR_SEARCH_ABANDONED, num, "Err.Number of Test of (a+)+b on thirty a" + Expect.IsTrue InStr(desc, "abandoned") > 0, "the message says the search was abandoned" + ' At thirty characters the budget is the 1,000,000 floor, which already far exceeds the + ' 2,700 a merely quadratic pattern would need - so this one is worse than quadratic, no + ' limit could cover it, and the message says to simplify rather than naming SearchStepLimit. + Expect.IsTrue InStr(desc, "Simplify the pattern") > 0, "the message says to simplify the pattern" Expect.Done End Sub [TestCase] - Public Sub Execute_CatastrophicPatternKeepsMatchesFound() + Public Sub Execute_CatastrophicPatternKeepsNothing() Expect.Begin Dim re As New RegExp + Dim m As MatchCollection + Dim num As Long re.Global = True re.Pattern = "x|(a+)+b" - Dim m As MatchCollection + On Error Resume Next + Err.Clear Set m = re.Execute("xx" & String$(30, "a")) - Expect.AreEqual 2, m.Count, "matches of x|(a+)+b in xx and thirty a" - Expect.IsTrue re.SearchAborted, "SearchAborted after that Execute" + num = Err.Number + On Error GoTo 0 + Err.Clear + Expect.AreEqual ERR_SEARCH_ABANDONED, num, "Err.Number of Execute of x|(a+)+b in xx and thirty a" + ' The two x matches found before the budget ran out are discarded. Handing them back would + ' be a correct prefix presented as a complete answer, which is the silent wrong result this + ' whole design exists to avoid. + Expect.IsTrue m Is Nothing, "no partial MatchCollection is returned" Expect.Done End Sub [TestCase] - Public Sub Replace_CatastrophicPatternReplacesMatchesFound() + Public Sub Replace_CatastrophicPatternKeepsNothing() Expect.Begin Dim re As New RegExp + Dim s As String + Dim num As Long re.Global = True re.Pattern = "x|(a+)+b" - Expect.AreEqual "--" & String$(30, "a"), re.Replace("xx" & String$(30, "a"), "-"), "Replace of x|(a+)+b in xx and thirty a" - Expect.IsTrue re.SearchAborted, "SearchAborted after that Replace" + On Error Resume Next + Err.Clear + s = re.Replace("xx" & String$(30, "a"), "-") + num = Err.Number + On Error GoTo 0 + Err.Clear + Expect.AreEqual ERR_SEARCH_ABANDONED, num, "Err.Number of Replace of x|(a+)+b in xx and thirty a" + Expect.AreEqual "", s, "no partially replaced string is returned" Expect.Done End Sub [TestCase] - Public Sub Split_CatastrophicPatternSplitsAtMatchesFound() + Public Sub Split_CatastrophicPatternKeepsNothing() Expect.Begin Dim re As New RegExp + Dim c As Collection + Dim num As Long re.Global = True re.Pattern = "x|(?:a+)+b" - Dim c As Collection + On Error Resume Next + Err.Clear Set c = re.Split("xx" & String$(30, "a")) - Expect.AreEqual 3, c.Count, "pieces of xx and thirty a split at x|(?:a+)+b" - Expect.AreEqual String$(30, "a"), c(3), "last piece" - Expect.IsTrue re.SearchAborted, "SearchAborted after that Split" + num = Err.Number + On Error GoTo 0 + Err.Clear + Expect.AreEqual ERR_SEARCH_ABANDONED, num, "Err.Number of Split of xx and thirty a at x|(?:a+)+b" + Expect.IsTrue c Is Nothing, "no partial split is returned" + Expect.Done + End Sub + + ' SearchStepLimit is the recourse when a genuinely quadratic pattern exceeds the cap. Tested + ' here with an explicit low limit rather than by driving the automatic cap, which would mean + ' waiting out two ~25-second searches to prove the same mechanism. + [TestCase] + Public Sub SearchStepLimit_RaisesTheCeiling() + Expect.Begin + Dim re As New RegExp + Dim num As Long, desc As String + re.Pattern = "\d+x" + Expect.AreEqual 0, re.SearchStepLimit, "SearchStepLimit defaults to 0, meaning automatic" + + ' far too few for the ~2,250,000 steps this costs + re.SearchStepLimit = 100000 + On Error Resume Next + Err.Clear + re.Test String$(1000, "1") + num = Err.Number: desc = Err.Description + On Error GoTo 0 + Err.Clear + Expect.AreEqual ERR_SEARCH_ABANDONED, num, "Err.Number with SearchStepLimit set far too low" + ' Here raising the limit genuinely would help, so the message names the property and a + ' figure - the opposite of the (a+)+b case above. + Expect.IsTrue InStr(desc, "SearchStepLimit") > 0, "the message names the property to raise" + + ' raised high enough, the same search completes + re.SearchStepLimit = 3000000 + Err.Clear + Expect.IsFalse re.Test(String$(1000, "1")), "Test of \d+x on 1,000 digits once the limit is raised" + Expect.AreEqual 0, Err.Number, "Err.Number once the limit is raised" Expect.Done End Sub @@ -64,7 +132,6 @@ Module SearchGivesUpTests Set m = re.Execute(String$(1000000, "a")) Expect.AreEqual 1, m.Count, "matches of a* in 1,000,000 a" Expect.AreEqual 1000000, m(0).Length, "length of the match of a* in 1,000,000 a" - Expect.IsFalse re.SearchAborted, "SearchAborted after that Execute" Expect.Done End Sub @@ -77,7 +144,6 @@ Module SearchGivesUpTests Set m = re.Execute(String$(1000000, "a")) Expect.AreEqual 1, m.Count, "matches of ^.*$ in 1,000,000 a" Expect.AreEqual 1000000, m(0).Length, "length of the match of ^.*$ in 1,000,000 a" - Expect.IsFalse re.SearchAborted, "SearchAborted after that Execute" Expect.Done End Sub diff --git a/test/package-tests/tbRegExp/expect.json b/test/package-tests/tbRegExp/expect.json index d796abc1..118b60bc 100644 --- a/test/package-tests/tbRegExp/expect.json +++ b/test/package-tests/tbRegExp/expect.json @@ -1,50 +1,4 @@ { - "version": "1.8.0.0", - "cases": { - "IsValidTests.IsValid_LowCountAboveHigh": { - "fails": "1 of 1 checks failed: IsValid of the pattern a{2,1}, whose lower count is above its upper count: expected False, got True" - }, - "IsValidTests.LastErrorNumber_LowCountAboveHigh": { - "fails": "1 of 1 checks failed: LastErrorNumber of the pattern a{2,1} (invalid quantifier): expected -2147218496, got 0" - }, - "IsValidTests.Test_LowCountAboveHighRaisesCompileError": { - "fails": "2 of 2 checks failed: Err.Number of Test(\"aa\") with the pattern a{2,1}: expected -2147218496, got 3000; Err.Description of Test(\"aa\") with the pattern a{2,1}: expected \"Invalid quantifier: a malformed '{n,m}' count (e.g. '{2,1}', or a number too large).\", got \"The specified print monitor is unknown.\"" - }, - "IsValidTests.Execute_LowCountAboveHighRaisesCompileError": { - "fails": "1 of 1 checks failed: Err.Number of Execute(\"aa\") with the pattern a{2,1}: expected -2147218496, got 3000" - }, - "IsValidTests.Replace_LowCountAboveHighRaisesCompileError": { - "fails": "1 of 1 checks failed: Err.Number of Replace(\"aa\", \"x\") with the pattern a{2,1}: expected -2147218496, got 3000" - }, - "IsValidTests.Split_LowCountAboveHighRaisesCompileError": { - "fails": "1 of 1 checks failed: Err.Number of Split(\"aa\") with the pattern a{2,1}: expected -2147218496, got 3000" - }, - "IsValidTests.IsValid_InvalidPatternLeavesNoError": { - "fails": "1 of 2 checks failed: Err.Number after IsValid compiled the pattern (\\d+: expected 0, got -2147218495" - }, - "IsValidTests.LastError_InvalidPatternLeavesNoError": { - "fails": "1 of 2 checks failed: Err.Number after LastError compiled the pattern (\\d+: expected 0, got -2147218495" - }, - "IsValidTests.LastErrorNumber_InvalidPatternLeavesNoError": { - "fails": "1 of 2 checks failed: Err.Number after LastErrorNumber compiled the pattern (\\d+: expected 0, got -2147218495" - }, - "PatternSyntaxTests.Test_BackreferenceToLaterGroupMatchesNothing": { - "fails": "2 of 2 checks failed: Test of \\1(a) on a (VBScript finds no match): expected False, got True; Test of \\1(a) on aa (VBScript finds no match): expected False, got True" - }, - "PatternSyntaxTests.IsValid_BraceThatStartsNoQuantifier": { - "fails": "3 of 3 checks failed: IsValid of a{ (VBScript reads the brace as text): expected True, got False; IsValid of a{x} (VBScript reads the braces as text): expected True, got False; IsValid of \\p{L} (VBScript reads it as the text p{L}): expected True, got False" - }, - "PatternSyntaxTests.Test_BraceThatStartsNoQuantifierIsText": { - "fails": "2 of 2 checks failed: Err.Number of Test of a{x} on a{x}: expected 0, got -2147218496; Test of a{x} on a{x} (VBScript reads the braces as text): expected True, got False" - }, - "PatternSyntaxTests.Test_CommaAndMaximumOnlyIsText": { - "fails": "2 of 3 checks failed: first match of a{,2} in a{,2} (VBScript reads {,2} as text): expected \"a{,2}\", got \"a\"; Test of a{,2} on aa (VBScript reads {,2} as text): expected False, got True" - }, - "ReplaceTests.Replace_TildeItemIsCopied": { - "fails": "1 of 1 checks failed: Replace(\"abc\", \"x$~y\") for the pattern a (VBScript copies $~): expected \"x$~ybc\", got \"xybc\"" - }, - "SearchAbortedTests.SearchAborted_DigitsWithoutXFinishAt5000": { - "fails": "1 of 2 checks failed: SearchAborted after Test of \\d+x on 5,000 digits and no x: expected False, got True" - } - } + "version": "1.9.0.0", + "cases": {} } diff --git a/test/package-tests/tbRegExp/tbRegExpTests.twinproj b/test/package-tests/tbRegExp/tbRegExpTests.twinproj new file mode 100644 index 00000000..04f9bfde Binary files /dev/null and b/test/package-tests/tbRegExp/tbRegExpTests.twinproj differ