Skip to content

Avoid quadratic backtracking in email and idn-email format regexes - #234

Open
kakkar87d wants to merge 1 commit into
horejsek:masterfrom
kakkar87d:email-format-linear-regex
Open

kakkar87d wants to merge 1 commit into
horejsek:masterfrom
kakkar87d:email-format-linear-regex

Conversation

@kakkar87d

Copy link
Copy Markdown
Contributor

email and idn-email format regexes backtrack quadratically on long invalid strings

>>> v = fastjsonschema.compile({'format': 'idn-email'})
>>> v('a@a' + '.' * 80000 + '@')    # master: 61 s, this branch: 1 ms
>>> v = fastjsonschema.compile({'format': 'email'})
>>> v('a@' + 'a.' * 40000 + '@')    # master: 24 s, this branch: 1 ms
input length email master email here idn-email master idn-email here
10 KB 0.27 s 0.1 ms 0.53 s 0.1 ms
40 KB 4.8 s 0.4 ms 10.1 s 0.5 ms
80 KB 23.8 s 0.8 ms 61.4 s 1.0 ms

After the @, [^@]+\.[^@]+\Z retries every dot once the trailing @ makes \Z fail, and the email lookahead (?!.*\.\..*@) rescans the rest of the string for each ... Format checks run on the data being validated, so a single 80 KB string field in a request body keeps the validator (and the GIL) busy for about a minute.

The local part is now matched as dot-separated runs of [^@.], and the "dot inside the domain" requirement is a single lookahead, (?=[^@][^@.]*\.[^@]), so there are no overlapping choices left to retry. Comparing old and new regexes over every string up to length 10 built from a, ., @, \n (plus 300k random strings), idn-email accepts exactly the same strings. email differs only when a newline sits between a .. in the local part and the @ (e.g. 'a..\nb@c.d'): the old lookahead's . could not see past the newline and accepted it, the new regex rejects it.

tests/test_format.py gets email / idn-email cases and a long-input test that fails on master (about 45 s) and passes here. The full suite passes on Python 3.10 and 3.12, and pylint output is unchanged. CHANGELOG.txt has an entry under UNRELEASED.

Both regexes run on the validated data and backtrack quadratically on
long invalid strings. After the @, [^@]+\.[^@]+\Z retries every dot
when the string ends in a second @, and the email lookahead
(?!.*\.\..*@) rescans the rest of the string for each '..'. Rejecting
'a@' + 'a.' * 40000 + '@' (80 KB) took 24 s for email, and
'a@a' + '.' * 80000 + '@' took 61 s for idn-email.

Match the local part as dot-separated runs of [^@.] and check for a dot
inside the domain with a single lookahead, so both regexes run in linear
time. idn-email accepts the same strings as before. email now also
rejects '..' in the local part when a newline sits between it and the
@, which the old lookahead's '.' could not see.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant