Repository navigation
[lexical-markdown] Bug Fix: Follow CommonMark's Unicode punctuation and whitespace for emphasis flanking - #9430
haripetrov-stack wants to merge 73 commits into
Conversation
…punctuation in emphasis flanking ## Description Markdown import decides whether a `*`, `_` or `__` run can open or close emphasis by looking at the characters on either side of it. `PUNCTUATION` and `PUNCTUATION_OR_SPACE` in `packages/lexical-markdown/src/utils.ts` list only ASCII characters, so an en dash, a curly quote or a currency sign next to a run counts as a letter. Before this change: - `See –**[here](https://example.com)** now.` imports as plain text. The next export escapes the asterisks: ``See –\*\*[here](https://example.com)\*\* now.`` - `See “**[here](https://example.com)**” now.` has the same problem. After this change both import as a bold link. The exporter writes the bold inside the link, so the first one exports as `See –[**here**](https://example.com) now.` CommonMark treats every character in the Unicode P (punctuation) and S (symbol) general categories as punctuation ([Unicode punctuation character](https://spec.commonmark.org/0.31.2/#unicode-punctuation-character), spec 0.31.2). Both constants now use `\p{P}\p{S}` with the `u` flag. - `PUNCTUATION` is read by `isFlanking` and `canEmphasis` in `importTextFormatTransformer.ts`. - `PUNCTUATION_OR_SPACE` is read by the typing shortcuts for underscore formats in `MarkdownShortcuts.ts`, so typing `_here_` after an en dash now formats the same way importing it does. Every ASCII punctuation character is in P or S, so ASCII results do not change, with one exception. `PUNCTUATION` did not contain the backslash (U+005C, category Po) because of the escaped `]` in the class. CommonMark counts it, and the new class includes it. meta#9407 and meta#9414 fix only that character, so this change overlaps with them on the `PUNCTUATION` line. Not changed: characters outside the Basic Multilingual Plane, such as most emoji, are still read one UTF-16 code unit at a time and still count as letters. Reported in sveltia/sveltia-cms#1107. ## Test plan Command: `pnpm exec vitest run --project unit packages/lexical-markdown` ### Before `main` at 4f385cd with only the new tests added: ``` Test Files 2 failed | 6 passed (8) Tests 12 failed | 702 passed (714) ``` The 12 failures are 8 imports, 2 exports and 2 typed shortcuts. Three of them: ``` FAIL LexicalMarkdown.test.ts > emphasis next to Unicode punctuation and symbols > imports "See –**[here](https://example.com)** now." AssertionError: expected [ [ 'See –**', 'plain' ], …(2) ] to deeply equal [ [ 'See –', 'plain' ], …(2) ] FAIL LexicalMarkdown.test.ts > emphasis next to Unicode punctuation and symbols > exports "See –**[here](https://example.com)** now." without escaped asterisks AssertionError: expected 'See –\*\*[here](https://example.com)\…' to be 'See –[**here**](https://example.com) …' FAIL MarkdownTransformers.test.ts > UNDERSCORE_AFTER_UNICODE_PUNCTUATION > typing "See –_here_" formats the word between the underscores AssertionError: expected [ 'See –_here_' ] to deeply equal [ 'See –', 'here' ] ``` The other 8 new tests are controls and pass before and after: ASCII `-` and a space before `**`, `a**b**`, `**a**b`, `snake_case_word`, a Latin and a Cyrillic letter before `**`, and a letter before a typed `_`. ### After ``` Test Files 8 passed (8) Tests 714 passed (714) ``` `prettier --check` and `eslint` are clean on the three changed files. Not run: the e2e suite, Flow and the repo-wide `tsc`.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
lexical
@lexical/a11y
@lexical/clipboard
@lexical/code
@lexical/code-core
@lexical/code-prism
@lexical/code-shiki
@lexical/compiler
@lexical/devtools-core
@lexical/dragon
@lexical/eslint-plugin
@lexical/extension
@lexical/fast-check
@lexical/file
@lexical/hashtag
@lexical/headless
@lexical/history
@lexical/html
@lexical/internal
@lexical/link
@lexical/list
@lexical/mark
@lexical/markdown
@lexical/mdast
@lexical/offset
@lexical/overflow
@lexical/plain-text
@lexical/react
@lexical/rich-text
@lexical/selection
@lexical/table
@lexical/tailwind
@lexical/text
@lexical/utils
@lexical/y
@lexical/yjs
commit: |
…ctuation, and read astral characters whole ## Description With Unicode punctuation (and the backslash) counted as punctuation on import, the exporter's intraword markers no longer round-trip when the formatted text starts or ends with punctuation next to a letter: plain `a` then bold `*` exported as `a**\***`, which CommonMark (and now the importer) reads as literal asterisks. Seven MarkdownEscaping tests from meta#9481 failed after merging main. The exporter now writes the letter beside such a marker as a numeric character reference (`&meta#97;**\***`), whose `&` or `;` is punctuation, so the marker flanks. Flanking checks in import and the typing shortcuts also read the characters around a marker by code point, so an emoji (category So) outside the Basic Multilingual Plane counts as punctuation. ## Changeset `.changeset/markdown-unicode-punctuation-flanking.md` ## Test plan `./node_modules/.bin/vitest run --project unit packages/lexical-markdown` ### Before ``` Test Files 1 failed | 8 passed (9) Tests 7 failed | 844 passed (851) ``` (`MarkdownEscaping.test.ts > preserves existing formatted markers after plain text` and `preserves formatted backslashes next to plain text`, e.g. `expected 'a*****' to be 'a*'`.) ### After ``` Test Files 9 passed (9) Tests 860 passed (860) ``` Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…arkers, and use CommonMark's Unicode whitespace ## Description The previous fix guessed where markers fall from neighbouring text nodes' formats, so it missed a tag closed and reopened around an outer tag's end (`***b****“y”*` lost the italic on “y”), text inside an inline element without a transformer, and inline decorators, and it fired where an unclosable tag wrote nothing. `$exportTextFormat` now writes each marker between sentinel noncharacters (U+FDD0 to U+FDD2, escaped as references when they appear in text), and the export resolves them once on the joined output: for each delimiter run, it looks at the characters a reader will see on either side and writes the word character beside a run that could not otherwise open or close as a numeric reference. Markers of different characters count as separate runs, as in CommonMark. `WHITESPACE` and `PUNCTUATION_OR_SPACE` now use CommonMark's Unicode whitespace (Zs plus tab, line feed, form feed and carriage return) instead of JavaScript's `\s`, which also counts U+FEFF and others. ## Test plan `./node_modules/.bin/vitest run --project unit packages/lexical-markdown` ### Before `***b****“y”*` export case: `expected '***b****“y”*' to be '***&meta#98;****“y”*'` (markdown-it renders the first with literal asterisks around “y”). ### After ``` Test Files 9 passed (9) Tests 865 passed (865) ``` Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…iding flanking ## Description The previous commit marked each format marker with sentinel noncharacters in the exported string. Those leaked into the strings that `exportChildren` and `exportFormat` hand to custom transformers, collided with the same characters in code spans, URLs and decorator text, and let the guard rewrite characters that a transformer wrote. `$exportTextFormat` and the child exporters now return pieces (escaped text, raw output, openers and closers), and `joinPieces` turns them into Markdown wherever a string leaves the exporter: the transformer callbacks and the element exports. Only text this export escaped is written as a reference. When newlines are preserved, a raw backslash before the referenced character goes out as `&meta#92;` so that it does not escape the reference. Unpaired surrogates are left alone, since their references read back as U+FFFD. `escapeLinkPart` shares `toCharacterReference`. ## Test plan `./node_modules/.bin/vitest run --project unit packages/lexical-markdown` ### Before The new preserve-newlines case exported `x\&meta#98;**“y”**`, which re-imports as `x&meta#98;`. ### After ``` Test Files 9 passed (9) Tests 866 passed (866) ``` Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hanges its neighbour ## Description `joinPieces` decided each marker run against the characters before any reference was written. A one-character piece written as a reference for one run reads as punctuation at both ends, which could stop the run on its other side from opening or closing (`x**&meta#97;***“b”*` lost the bold). Runs beside such a piece are now decided again, through a worklist so each piece is revisited at most twice and the join stays linear. Decorator text now counts as text a reference can stand in for, U+0000 is left alone like unpaired surrogates (both read back as U+FFFD), and the changeset no longer implies every emoji counts as punctuation. ## Test plan `./node_modules/.bin/vitest run --project unit packages/lexical-markdown` ### Before `[plain 'x', bold 'a', italic '“b”']` exported as `x**&meta#97;***“b”*`, which re-imports with the bold lost. ### After ``` Test Files 9 passed (9) Tests 869 passed (869) ``` Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e, and note the import change
## Description
Export placed tags inside JavaScript whitespace (`trimStart`/`trimEnd`, `/\S/`) while import now uses CommonMark's Unicode whitespace, so a bold `x` went out as `**x**`. Export now scans for the same whitespace (still linear in long runs). U+0000 can be written as a reference again, since CommonMark reads a raw one as U+FFFD too; the backslash check reuses `isEscaped`; `joinPieces` keeps dense arrays and computes each piece's length check once. The changeset's Verify paragraph now covers Markdown that used to import as bold beside Unicode punctuation.
## Test plan
`./node_modules/.bin/vitest run --project unit packages/lexical-markdown`
### After
```
Test Files 9 passed (9)
Tests 870 passed (870)
```
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d export, and guard raw backslashes before markers ## Description Export now decides whether a reference is needed with the same `isFlankedBy` rule that import's `isFlanking` uses, instead of a copy of it. When newlines are preserved, a raw backslash right before a marker went out as is and escaped it (`a\**b**` lost the bold); it now goes out as `&meta#92;`. `joinPieces` skips its setup when there are no markers, and the trailing-whitespace scan has an explicit bound. ## Test plan `./node_modules/.bin/vitest run --project unit packages/lexical-markdown` ### After ``` Test Files 9 passed (9) Tests 871 passed (871) ``` Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pace, and guard raw backslashes before any markup ## Description The previous commit placed export tags by CommonMark whitespace only, so a tag could land beside U+000B, which markdown-it counts as whitespace (`**\vx\v**` lost the bold there). Export goes back to trimming JavaScript whitespace, which holds CommonMark's and the characters some readers add; import keeps the spec's set. When newlines are preserved, a trailing raw backslash before any following markup (a marker, a reference, a code span, a link) now goes out as `&meta#92;`, in one place, and the reference analysis runs only when there are markers. ## Test plan `./node_modules/.bin/vitest run --project unit packages/lexical-markdown` ### After ``` Test Files 9 passed (9) Tests 871 passed (871) ``` Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ost format for another
## Description
A reference for a one-character piece makes both of its edges punctuation, which the run on its other side may need repaired with another reference, and so on along a chain of one-character pieces. When that chain ended at raw output (a transformer's `@x`) the last run could not be repaired, so the reference only swapped which format was lost. `decideReferences` now walks the chain first and writes nothing when it is blocked; answers are remembered per piece and direction, so each piece is walked at most twice.
U+0000 is left alone again (Lexical's importer keeps a raw one but reads `�` as U+FFFD), the preserve-newlines backslash guard is limited to what `joinPieces` writes (markers and references), and the changeset's Verify paragraph mentions it.
## Test plan
`./node_modules/.bin/vitest run --project unit packages/lexical-markdown`
### Before
```
× writes no reference that would break a run beside raw output
Tests 1 failed | 695 skipped (696)
```
### After
```
Test Files 9 passed (9)
Tests 872 passed (872)
```
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eping
## Description
`decideReferences` keeps one character per delimiter group instead of joined marker strings, decodes each piece's edge characters once, and remembers chain answers in one tri-state table per direction. The comment on that cache now says how an answer can go stale (only toward writing fewer references), and `appendPieces` says why it loops.
## Test plan
`./node_modules/.bin/vitest run --project unit packages/lexical-markdown`
### After
```
Test Files 9 passed (9)
Tests 872 passed (872)
```
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s raw beside markers ## Description A reference for a control character (U+0001 to U+001F, U+007F to U+009F) or a noncharacter reads back as U+FFFD (or `…` for `&meta#133;`) in markdown-it and HTML decoders, so the export no longer writes one for them; they stay raw as on main. `canEmphasis` reads the characters around a delimiter once for both its open and close answers, and `isPunctuationOrSpace` replaces the third hand-written character class. ## Test plan `./node_modules/.bin/vitest run --project unit packages/lexical-markdown` ### After ``` Test Files 9 passed (9) Tests 872 passed (872) ``` Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t, email or mention ## Description A reference for the letter at the edge of `www.example.com`, `me@example.com` or `https://example.com/a` changed what an autolinking reader links to (`www.example.com` links to `www.example.co`). The export now writes none inside such a word, so only the format beside it is lost, as before. `canSpread` models the reference's closing `;` when walking rightward, and the changeset says the raw-backslash guard also covers decorator text. ## Test plan `./node_modules/.bin/vitest run --project unit packages/lexical-markdown` ### After ``` Test Files 9 passed (9) Tests 875 passed (875) ``` Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rences ## Description The export kept every marker and text piece as its own object, decoded edge characters with `String.fromCodePoint`, ran Unicode property regular expressions for ASCII characters and scanned words for autolinks before cheaper checks. Exporting a 40,000-node paragraph took 112 ms (best of 25) against 40 ms on main with heavy formatting beside punctuation, and 39 ms against 28 ms for ordinary prose. Pieces are now two parallel arrays, markers go back to concatenated tag strings, `characterAt`/`characterBefore` slice only for surrogate pairs, flanking classifies ASCII without a regular expression, the autolink check skips pieces with no `://`, `www.` or `@`, and the bookkeeping uses typed arrays. With the same probe: 63 ms (main 40 ms) and 38 ms (main 28 ms). `decideReferences` returns its results, and `isPunctuationOrSpace` shares the classifier. The changeset also lists U+000B, U+2028 and U+2029 with U+FEFF as characters that no longer count as whitespace beside a marker. ## Test plan `./node_modules/.bin/vitest run --project unit packages/lexical-markdown` ### After ``` Test Files 9 passed (9) Tests 875 passed (875) ``` Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t only ## Description The autolink check took the whole run of non-whitespace at a piece's edge, so in spaceless CJK text or after a closing parenthesis an `@` anywhere in it stopped the reference (`联系我@微博你好` before a bold `“重要”` lost the bold), and it rescanned the whole piece on every call. It now scans at most 2,048 URL characters back from the edge and asks whether a URL, `www.` host, email or mention ends there (or starts there, after a closer). `CharacterClass` is plain numeric constants instead of a `const enum`, which Babel does not inline, and the importer's `_` rule asks the shared `isPunctuation` instead of a second regular expression. ## Test plan `./node_modules/.bin/vitest run --project unit packages/lexical-markdown` ### After ``` Test Files 9 passed (9) Tests 877 passed (877) ``` Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…write nested pieces in place ## Description The export's autolink check stopped at the backslash that escapes `_`, `~` or `*`, so a mention like `@john_doe` or a URL path like `/a_b` next to formatted punctuation got a reference written inside it (`@john\_do&meta#101;**(x)**`). The check now steps over escaping backslashes. Nested inline elements also wrote their pieces into a fresh array that the parent copied; they now write into the caller's pieces, in both the full and the selection export. ## Test plan ### Before ``` pnpm vitest run --project unit packages/lexical-markdown/src/__tests__/unit/LexicalMarkdown.test.ts -t autolink × writes no reference inside "https://example.com/a_b", which readers may autolink × writes no reference inside "@john_doe", which readers may autolink Tests 2 failed | 3 passed | 698 skipped (703) ``` ### After ``` pnpm vitest run --project unit packages/lexical-markdown Tests 879 passed (879) ``` Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…elimiter run scan
## Description
The export now works out whether a piece edge is a URL, email or mention once per edge instead of on every reference decision, scans a delimiter run's two ends with one helper, and names the reference check `isReferenceable`. `characterAt` uses `codePointAt`. The changeset's Verify paragraph now also names stored Markdown such as `a**\_x**`, which imports with literal markers now that the backslash counts as punctuation.
## Test plan
### After
```
pnpm vitest run --project unit packages/lexical-markdown
Tests 879 passed (879)
pnpm run check-changesets
Changeset check passed: 32 pending changeset(s) valid.
```
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
## Description Merges main at cfcd3c4, which brings meta#9600 (escape a literal bracket only where it would start a link). It conflicted with this branch in four files: - `MarkdownExport.ts`: text escaping keeps this branch's `shouldPreserveNewLines` backslash handling and `toCharacterReference`, and routes default-mode text through meta#9600's `$escapeLiteralBrackets` instead of writing every bracket as a reference. meta#9600's bracket helpers sit beside this branch's piece joining. - `importTextMatchTransformer.ts`: `matchLink` combines meta#9600's code-span rules (no link opens inside a code span, or has its `](` or `)` in one) with this branch's bounded search (stop at the last `](`, jump over escaped openers, start at the real opener). A search rejected for a code span now jumps to the code span's end or the label's last character, since every `[` before it finds the same rejected link. The code spans come from the line's format scan when the code transformer is present. A link with an empty label after its real opener, as in `x[[](u)`, now stays text as meta#9600 has it, like `[](u)` does, rather than becoming an empty link that dropped `](u)`. - `LexicalMarkdown.test.ts`: `[hello]](https://lexical.dev)[world](...)` exports as `[hello]&meta#93;(https&meta#58;//lexical.dev)[world](https://lexical.dev)`: meta#9600's bracket escape, plus this branch's spoiled scheme so that GFM readers do not run the URL on into the next link. - `MarkdownEscaping.test.ts`: both sides made the same change to the unbalanced-link test; main's version is kept. ## Test plan Command: `pnpm exec vitest run --project unit packages/lexical-markdown packages/lexical-mdast packages/lexical-playground/__tests__/unit scripts/__tests__/unit/treeShakingSource.test.ts` ### After ``` Test Files 46 passed (46) Tests 1599 passed (1599) ``` `matchLink` was compared with main's on 150,000 random lines of brackets, parentheses, escapes and backticks, using the same code spans: no differences. Export and import round trips of 20,000 random paragraphs with links, brackets, formats and bare URLs found none that main reads back correctly and this branch does not, in either `@lexical/markdown` or `@lexical/mdast`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… to the changeset of meta#9600 ## Description The changeset said that a link now starts at the `[` its label really opens with. Main's changeset for meta#9600 says the same, and that change is on main now, so this changeset drops the sentence rather than listing it twice. ## Test plan ### After ``` $ pnpm run check-changesets Changeset check passed: 69 pending changeset(s) valid. ``` Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Thanks for the thorough work here. The change looks good to me: emphasis flanking in @lexical/markdown now follows CommonMark 0.31.2's Unicode punctuation and whitespace, on import and in the typing shortcuts, and the exporter's character references keep formatting readable for CommonMark readers and for @lexical/mdast. The new tests in LexicalMarkdown.test.ts and MarkdownTransformers.test.ts include 92 cases that fail without the source changes and pass with them. Everything below is optional and non-blocking.
Suggestions (optional)
- Formatting right after a custom text match that ends in a letter (
packages/lexical-markdown/src/importTextTransformers.ts:317). With a mention-styleTextMatchTransformer(importRegExp: /@(\w+)/, export writing@bob), mention@bobfollowed by bold(admin)exports as@bob**(admin)**, as before, but now re-imports as literal**(admin)**, and the next export escapes the asterisks. The same happens with italic“a”and bold"x". Onmainthe text after a match was read as the start of a line, so this round-tripped. The text after a match now takes the match's last character as its neighbour, and the exporter can't put a reference inside transformer output, so there is no way to write this shape that the importer accepts. The PR and changeset describe only delimiter runs that a match cuts (*)**a*). AGENTS.md (Backwards Compatibility) asks to keep existing behaviour that callers may depend on, so consider giving the text after (and before) a text match a punctuation or line-edge neighbour when the match doesn't cut a delimiter run, and adding a round-trip test with a mention-style transformer. The built-in and playground transformers aren't affected, since their syntax ends in),$or:. - A referenced U+2028, U+2029, U+FEFF or U+000B after a bare URL (
packages/lexical-markdown/src/MarkdownExport.ts:737, used at:958).findSpoiledLinksends a URL at any JavaScript whitespace, but these four characters are letters for CommonMark, so they get written as references (
), and GFM readers keep those in the link. Plainsee, boldhttp://x.co/, then U+2028 andxexports assee **http://x.co/**
x, which@lexical/mdastreads as one link with the bold lost. Counting referenced positions as URL characters in that scan (isLinkCharacter(text[end]) || isReference[end]) would close it. - Changeset (
.changeset/markdown-unicode-punctuation-flanking.md). CONTRIBUTING.md (Changesets) asks a fix that changes observable behaviour to say what to verify. If item 1 stays as it is, theVerify:paragraph could name it. Splitting the long summary paragraph into its import and export changes would also make the release note easier to scan.
Question for maintainers: Markdown stored by earlier versions with a marker between a letter and Unicode punctuation, such as **“bar”**baz and especially CJK labels like **注意:**请先登录, used to import as bold. It now imports with literal asterisks, which are escaped on the next save. CommonMark requires this, and the changeset's Verify: paragraph documents it, with a CJK-friendly rule left for a follow-up. Is shipping strict flanking for stored documents in a patch release the intended direction?
The CI workflows haven't run on this head yet because they're awaiting approval, so a maintainer still needs to run them before merge.
| scan.rest = rest; | ||
| return { | ||
| after, | ||
| before: characterBeyondRun(text, endIndex, -1, before, delimiters), |
There was a problem hiding this comment.
Optional: here the text after a match takes the match's last character as its neighbour, and the reused line scan reads it the same way. With a mention-style transformer whose output ends in a letter, @bob**(admin)** (Lexical's own export of mention @bob then bold (admin)) now re-imports as literal **(admin)**, where main read the rest as the start of a line. The exporter can't put a reference inside transformer output, so this shape can't be written in a form the importer accepts. Consider a punctuation or line-edge neighbour when the match doesn't cut a delimiter run, plus a round-trip test with such a transformer.
| // Characters a URL or email can hold: GFM ends an autolink at | ||
| // whitespace or `<`, and a URL's host and path may hold any other character. | ||
| // This takes in the backslash that escapes a marker too. | ||
| const LINK_END = /[\s\p{Cc}<]/u; |
There was a problem hiding this comment.
Nit: U+2028, U+2029, U+FEFF and U+000B match \s/\p{Cc} here, so the scan at line 958 ends a URL at them. CommonMark reads them as letters, though, so beside a marker they go out as references (
), and GFM readers keep those in the link: plain see , bold http://x.co/, then U+2028 and x exports as see **http://x.co/**
x, which @lexical/mdast reads as one link with the bold lost. Treating a referenced position as a URL character in that scan (isLinkCharacter(text[end]) || isReference[end]) would fix it.
…de the edge of a line again, and keep a referenced character inside a bare URL ## Description On main, the text after a text match is imported on its own, so a delimiter run at its start reads as one at the start of a line. This branch gave that text the match's last character as its neighbour, and the text before a match the match's first character, so that a delimiter run that a match cuts reads as it did in the whole line. For a custom transformer whose syntax ends or starts in a letter, that broke round trips: mention `user:bob` then bold `(admin)` exports as `user:bob**(admin)**`, as before, but re-imported as literal `**(admin)**`, and the exporter cannot put a reference inside transformer output to avoid it. The text beside a text match now reads the match's edge as the edge of a line again, as on main, unless a delimiter run goes on across the cut. The line's scan is still reused for the text after the match. Such a run can then only differ in whether it opens, since nothing in that text comes before it, and so only in the pair it opens. That pair is found by scanning ever longer prefixes of the text, so finding it costs about as much as the text it holds, instead of a scan of the rest of the line after every match. On a line of 4,000 `user:ab**(x)** ` it imports in about the time the same line with a space before each `**` took before this change. The remaining growth with line length is core `splitText`, as on main. Export ended a bare URL at any JavaScript whitespace when deciding whether to spoil it, but U+000B, U+2028, U+2029 and U+FEFF are letters for CommonMark, so beside a marker they go out as references, which GFM readers keep in the link: plain `see `, bold `http://x.co/`, then U+2028 and `x` exported as `see **http://x.co/**
x`, which `@lexical/mdast` read as one link with the bold lost. A character written as a reference now counts as part of the URL, so this exports as `see **http&meta#58;//x.co/**&meta#8232;x`. The changeset now gives the import fixes and the export changes paragraphs of their own, and its `Verify:` paragraph names the export snapshots. ## Test plan Command: `pnpm exec vitest run --project unit packages/lexical-markdown/src/__tests__/unit/LexicalMarkdown.test.ts -t "whose syntax has a letter"`, and `-t "letter only for CommonMark"` ### Before ``` × reads formatting after a text match whose syntax has a letter beside it 49ms × reads formatting before a text match whose syntax has a letter beside it 4ms × reads formatting in italic after a text match whose syntax has a letter beside it 4ms AssertionError: expected [ Array(2) ] to deep equally contain [ '(admin)', 1 ] AssertionError: expected [ Array(2) ] to deep equally contain [ '(admin)', 1 ] AssertionError: expected [ [ 'bob', +0 ], [ '*“a”*', +0 ] ] to deep equally contain [ '“a”', 2 ] Tests 3 failed | 782 skipped (785) × exports formatting beside a URL before a character that is a letter only for CommonMark, which both importers read back 33ms AssertionError: expected 'see **http://x.co/**
x' to be 'see **http&meta#58;//x.co/**&meta#8232;x' // Object.is equality ``` ### After ``` $ pnpm exec vitest run --project unit packages/lexical-markdown packages/lexical-mdast packages/lexical-playground/__tests__/unit scripts/__tests__/unit/treeShakingSource.test.ts Test Files 46 passed (46) Tests 1603 passed (1603) ``` The prefix search was compared with a full scan of the rest after every match on 47,000 random lines of delimiters, code spans, escapes and a letter-ending text match, with no differences. Round trips of 20,000 random paragraphs with links, bare URLs, emails and U+2028 or U+FEFF found none that main reads back correctly and this branch does not, in either importer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Thanks for the thorough work here. The change looks good to me: emphasis flanking in @lexical/markdown now follows CommonMark 0.31.2's Unicode punctuation and whitespace, on import and in the typing shortcuts, and the exporter's character references keep formatting readable for CommonMark readers and for @lexical/mdast. Everything below is optional and non-blocking.
Suggestion (optional)
-
A delimiter run right after a text match reads two ways (
packages/lexical-markdown/src/importTextTransformers.ts:51-62, used at:358-371). After a link or other text match, the rest of the line reuses the whole line's emphasis scan, unlessopensOnlyBesideEdgefinds that the leading run opens only beside the line edge. That check comparescanOpenalone. But the match's last character also decides whether the run can close, and CommonMark's rule of 3 depends on that: "If one of the delimiters can both open and close emphasis, then the sum of the lengths of the delimiter runs containing the opening and closing delimiters must not be a multiple of 3 unless both lengths are multiples of 3." Text that ends before a later match is scanned fresh with the edge as its neighbour. So the same run reads differently depending on what follows it on the line:Markdown This PR mainCommonMark [a](u)**#*literal **#**+ italic#literal **#*[a](u)**#* [b](v)*+ italic#*+ italic#literal **#*[a](u)**(*s****(+ italics+**+ italic(s*+ italic(sThe description says text beside a text match still reads the match's edge as the edge of a line, as on
main. ComparingcanCloseas well, and sending either difference tofindPairAtEdge, would make the two paths agree. A test that[a](u)**#*reads the same with and without a trailing[b](v)would pin this down. So would one line with more than 64 characters after a match, wherefindPairAtEdgestarts trying prefixes, since no current test reaches the reuse path there. Lexical's own exports don't appear to produce these shapes, so only hand-written Markdown is affected.
Question for maintainers: Markdown stored by earlier versions with a marker between a letter and Unicode punctuation used to import as bold. Examples are **“bar”**baz and, more commonly, CJK labels such as **注意:**请先登录, which earlier versions of the exporter wrote themselves. These now import with literal asterisks, which the next save escapes. CommonMark requires this, the changeset's Verify: paragraph documents it, and a CJK-friendly rule is left for a follow-up. AGENTS.md (Backwards Compatibility) says "when in doubt, assume external code depends on the current behavior". Is shipping strict flanking for stored documents in a patch release the intended direction?
| const after = edge < text.length ? characterAt(text, edge) : outside; | ||
| return ( | ||
| getEmphasisSides(text[0], undefined, after).canOpen && | ||
| !getEmphasisSides(text[0], before, after).canOpen |
There was a problem hiding this comment.
Optional: this compares only canOpen. Beside the match's last character the run may also be able to close, and CommonMark's rule of 3 depends on that. So [a](u)**#* keeps literal **#* while [a](u)**#* [b](v) (deferred and scanned fresh at the edge) italicizes #, and main italicizes both. Comparing canClose too, and sending either difference to findPairAtEdge, would make the two paths agree.
…atever follows it on the line
## Description
After a link or other text match, the text after it reuses the line's emphasis scan unless the delimiter run that starts it reads differently beside the edge of a line. That check compared only whether the run can open. Whether it can close matters too, through CommonMark's rule of 3, so `[a](u)**#*` kept a literal `**#*` while `[a](u)**#* [b](v)`, whose first part is scanned on its own, read `*` and italic `#`, as main reads both. The check now compares both, and either difference goes to the prefix search for the pair the run opens, so both lines read as main reads them.
## Test plan
Command: `pnpm exec vitest run --project unit packages/lexical-markdown/src/__tests__/unit/LexicalMarkdown.test.ts -t "whatever follows it|whose syntax has a letter"`
### Before
```
× reads a run after a link the same whatever follows it on the line, with a short run 24ms
× reads a run after a link the same whatever follows it on the line, with a long run 3ms
AssertionError: expected [ [ 'a', +0 ], [ '**#*', +0 ] ] to deeply equal [ Array(3) ]
AssertionError: expected [ [ 'a', +0 ], [ …(2) ] ] to deeply equal [ [ 'a', +0 ], [ '*', +0 ], [ …(2) ] ]
Tests 2 failed | 4 passed | 783 skipped (789)
```
### After
```
$ pnpm exec vitest run --project unit packages/lexical-markdown packages/lexical-mdast packages/lexical-playground/__tests__/unit scripts/__tests__/unit/treeShakingSource.test.ts
Test Files 46 passed (46)
Tests 1606 passed (1606)
```
The new tests include lines with more than 64 characters after the match, where the prefix search starts, both with the pair inside the first prefix and beyond it, and a mention row whose pair lies past the first prefix. The prefix search was compared with a full scan of the rest after every match on 33,000 random lines of links, letter-ending text matches, delimiters, code spans and escapes, starting from prefixes of 2 and of 64 characters, with no differences.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…atever follows it on the line
## Description
After a link or other text match, the text after it reuses the line's emphasis scan unless the delimiter run that starts it reads differently beside the edge of a line. That check compared only whether the run can open. Whether it can close matters too, through CommonMark's rule of 3, so `[a](u)**#*` kept a literal `**#*` while `[a](u)**#* [b](v)`, whose first part is scanned on its own, read `*` and italic `#`, as main reads both. The check now compares both, and either difference goes to the prefix search for the pair the run opens, so both lines read as main reads them.
## Test plan
Command: `pnpm exec vitest run --project unit packages/lexical-markdown/src/__tests__/unit/LexicalMarkdown.test.ts -t "whatever follows it|whose syntax has a letter"`
### Before
```
× reads a run after a link the same whatever follows it on the line, with a short run 24ms
× reads a run after a link the same whatever follows it on the line, with a long run 3ms
AssertionError: expected [ [ 'a', +0 ], [ '**#*', +0 ] ] to deeply equal [ Array(3) ]
AssertionError: expected [ [ 'a', +0 ], [ …(2) ] ] to deeply equal [ [ 'a', +0 ], [ '*', +0 ], [ …(2) ] ]
Tests 2 failed | 4 passed | 783 skipped (789)
```
### After
```
$ pnpm exec vitest run --project unit packages/lexical-markdown packages/lexical-mdast packages/lexical-playground/__tests__/unit scripts/__tests__/unit/treeShakingSource.test.ts
Test Files 46 passed (46)
Tests 1606 passed (1606)
```
The new tests include lines with more than 64 characters after the match, where the prefix search starts, both with the pair inside the first prefix and beyond it, and a mention row whose pair lies past the first prefix. The prefix search was compared with a full scan of the rest after every match on 33,000 random lines of links, letter-ending text matches, delimiters, code spans and escapes, starting from prefixes of 2 and of 64 characters, with no differences.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ne its named groups where they are
## Description
A look-ahead text match kept for the rest of a line is moved to the text left after an earlier match, but its `indices.groups` still pointed at the ranges in the text it was found in, so a transformer reading a named group's range from a `d`-flag regex got the wrong text (`**a** @bob` with `/@(?<name>\w+)/d` read `''` for `name`). Each named group now takes the moved range of its numbered group.
## Test plan
### Before
```
pnpm exec vitest run --project unit packages/lexical-markdown/src/__tests__/unit/LexicalMarkdown.test.ts -t "named groups"
AssertionError: expected [ '', 'bob' ] to deeply equal [ 'bob', 'bob' ]
Tests 1 failed | 789 skipped (790)
```
### After
```
pnpm exec vitest run --project unit packages/lexical-markdown/src/__tests__/unit/LexicalMarkdown.test.ts
Tests 790 passed (790)
```
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e of its own again
## Description
The content of a format now reads the outer marker beyond its edges, so that `*`, `_` and `~` runs there flank as CommonMark and GFM define. For `==`, which neither defines, this let the rule of 3 refuse a highlight at the content's edge that main applies: `` **==`====** `` imported as bold literal text instead of a bold highlight. A run of a marker that CommonMark and GFM do not define now reads the content's edge as the edge of a line, as on main, while `` **~~`~~~~** `` stays bold literal text as @lexical/mdast reads it.
## Test plan
### Before
```
pnpm exec vitest run --project unit packages/lexical-markdown/src/__tests__/unit/LexicalMarkdown.test.ts -t "highlight inside a format"
× reads a highlight inside a format as a line of its own, and GFM's markers as GFM does, in "**==`====**"
Tests 1 failed | 1 passed | 790 skipped (792)
```
### After
```
pnpm exec vitest run --project unit packages/lexical-markdown/src/__tests__/unit/LexicalMarkdown.test.ts -t "highlight inside a format"
Tests 2 passed | 790 skipped (792)
```
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Thanks, this looks good to me. Flanking now follows CommonMark 0.31.2's Unicode punctuation and whitespace definitions. The exporter's references keep formatting readable for CommonMark readers and @lexical/mdast. The import loop is faster and no longer recursive. Everything below is optional and non-blocking.
A behavior decision for maintainers. With :, 。 and 「 counted as punctuation (packages/lexical-markdown/src/utils.ts:66), CJK Markdown such as **注意:**请先登录, **你好。**世界 and 世界**「你好」** now imports with literal asterisks; main imported all three as bold. Export writes the same nodes as **注意:**请先登录. That is what CommonMark and GFM readers show, and the PR description and changeset say so. Still, it is a visible change for Chinese and Japanese content, where full-width punctuation sits directly beside text. Is the team happy to ship the strict behavior now and add the CJK-friendly rule in the follow-up, or should that rule land in the same release?
Optional nits
[a](u)*`c`)**now imports italic codecand an italic). The link is applied before the code span (packages/lexical-markdown/src/importTextTransformers.ts:279-288), and the run after it reads the link's end as a line edge (:110). CommonMark reads that*as both-flanking, so the rule of 3 keeps it literal, asmaindid.**)*(.`*b!****_loses both italics. The bold's content is re-read with the closing run as context (importTextTransformers.ts:406-415). That makes the leftover**both-flanking, and the rule of 3 then counts its length as 2 instead of 4.mainkeeps the italics. Both inputs are contrived and close to the rule-of-3 limitation the PR already lists (#4895).- AGENTS.md ("Commits and Pull Requests") asks the Test plan's
### Beforeto show the failing output before the change. The current Before showsmain's escaping tests failing against an earlier head of this PR. A run of the new tests againstmainwould show what they catch.
| // U+FEFF. Delimiter flanking depends on both. | ||
| // https://spec.commonmark.org/0.31.2/#unicode-punctuation-character | ||
| // https://spec.commonmark.org/0.31.2/#unicode-whitespace-character | ||
| const PUNCTUATION = /[\p{P}\p{S}]/u; |
There was a problem hiding this comment.
Optional, for maintainers: counting full-width CJK punctuation (:, 。, 「) as punctuation makes **注意:**请先登录 and **你好。**世界 import with literal asterisks, where main imported them as bold. That matches CommonMark 0.31.2, but it is a visible change for Chinese and Japanese content. Is shipping the strict rule now, with the CJK-friendly rule as a follow-up, the intended order?
| foundTextMatch.endIndex >= foundTextFormat.endIndex | ||
| (startIndex <= foundTextFormat.startIndex && | ||
| endIndex >= foundTextFormat.endIndex) || | ||
| endIndex <= foundTextFormat.startIndex |
There was a problem hiding this comment.
Optional nit: applying a link that ends before a code span first means the run after it reads the link's end as a line edge, so [a](u)*`c`)** now imports italic code and an italic ). CommonMark (and main) keep it literal, because that * is both-flanking and the rule of 3 stops it from pairing with **. It is rare, so feel free to leave it for the rule-of-3 follow-up (#4895).
…ext format, and read a run a format's tag cuts beside a line edge
## Description
Two inputs imported otherwise than on main and @lexical/mdast:
- `` [a](u)*`c`)** `` imported italic code and an italic `)`. The run after a link, read beside the edge of a line, was paired across the code span that the line's scan had found next. It now pairs only before that format, which is imported first as it would be with the link before it.
- `` **)*(.`*b!****_ `` lost both italics. The bold's content read the `**` that its closing tag cut beside what lies after the whole run, so it could open, and the rule of 3 then measured it without the part the tag took. A run that the tag cuts now reads beside the edge of a line, as on main; one it does not cut still reads beside the tag when its marker is one CommonMark or GFM define.
## Test plan
### Before
```
pnpm exec vitest run --project unit packages/lexical-markdown/src/__tests__/unit/LexicalMarkdown.test.ts -t "beside another marker"
× reads a run beside another marker as CommonMark does in "[a](u)*`c`)**"
× reads a run beside another marker as CommonMark does in "**)*(.`*b!****_"
Tests 2 failed | 4 passed | 788 skipped (794)
```
### After
```
pnpm exec vitest run --project unit packages/lexical-markdown/src/__tests__/unit/LexicalMarkdown.test.ts -t "beside another marker"
Tests 6 passed | 788 skipped (794)
```
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t an ASCII table
## Description
`classify` filled a lazy 128-entry table so that ASCII characters needed no regular expression. It is called once per delimiter run and per marker, not per character, so the table saved nothing measurable, and it cost 186 bytes minified (73 gzipped). It now tests the two category expressions directly.
## Test plan
### Before
```
$ esbuild packages/lexical-markdown/src/index.ts --bundle --minify --external:lexical --external:'@lexical/*' | wc -c; ... | gzip -9 | wc -c
43701
16621
```
### After
```
$ pnpm exec vitest run --project unit packages/lexical-markdown
Test Files 9 passed (9)
Tests 1063 passed (1063)
$ esbuild packages/lexical-markdown/src/index.ts --bundle --minify --external:lexical --external:'@lexical/*' | wc -c; ... | gzip -9 | wc -c
43515
16548
```
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nk search in less code
## Description
The same output in less code, from 43,515 to 42,305 bytes minified (16,548 to 16,152 gzipped) for the package bundle:
- Export marks the code units that go out as references on one array over the joined text, which the URL spoiling marks too, and writes them in one pass, rather than keeping per-piece lengths and a list of spoiled positions. A marker run is a tuple, and one function decides whether a reference helps a run open or close.
- A link whose label holds its real opener is found by searching on from that opener, which finds the same link, so `moveMatch` no longer cuts a match and a found link needs no separate position.
- `characterAt` and `characterBefore` use `codePointAt`, and the cut-run walk reads the scan's delimiter set instead of a cached string.
## Test plan
### Before
```
$ esbuild packages/lexical-markdown/src/index.ts --bundle --minify --external:lexical --external:'@lexical/*' | wc -c; ... | gzip -9 | wc -c
43515
16548
```
### After
```
$ pnpm exec vitest run --project unit packages/lexical-markdown packages/lexical-mdast
Test Files 17 passed (17)
Tests 1348 passed (1348)
$ esbuild packages/lexical-markdown/src/index.ts --bundle --minify --external:lexical --external:'@lexical/*' | wc -c; ... | gzip -9 | wc -c
42305
16152
```
The export and import output was also compared with the previous commit on 210,000 random paragraphs and 830,000 random lines (formats, links, line breaks, raw transformer output, URLs, emails, references, code spans and brackets), with no difference.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Thanks for this. The change looks good to me: classifying punctuation and whitespace with the Unicode sets follows CommonMark 0.31.2, the new tests fail on main and pass here, and the exported references read back through both @lexical/markdown and @lexical/mdast. A few suggestions, all optional and non-blocking:
- Stored CJK Markdown (a call for maintainers). Markdown that
mainexported and read back as bold, such as**注意:**请先登录, now imports with literal asterisks, and saving it again writes\*\*注意:\*\*请先登录. That is what CommonMark specifies, and the changeset'sVerify:already says so. Bold labels that end in full-width punctuation right before CJK text are common in Chinese and Japanese writing, so it may be worth shipping the CJK-friendly follow-up in the same release. CONTRIBUTING.md files a behaviour-changing fix as[fix]withVerify:, as you did. AGENTS.md (Backwards Compatibility) describes[breaking]withMigration:/Detect:for changes callers depend on, if maintainers prefer that category here. - Edge case right after a link (
packages/lexical-markdown/src/importTextFormatTransformer.ts,findPairAtEdge, lines 161–195).[x](u)**:*a.**imports onmainasx,*, italic:a., reading the run after the link beside the edge of a line as the description intends, but here asx,**:, italica.,*. The search for the pair that the run after the link opens stops before the scan's next emphasis (end), but that pair can contain it. Returningfalse(so the rest is scanned on its own) whenendcomes from the next emphasis or code span would keep the edge reading. It is a narrow shape. - Export cost (
packages/lexical-markdown/src/MarkdownExport.ts,findSpoiledLinks, line 897). Any paragraph whose text containswww.,://or@builds a masked copy and three typed arrays, even when no reference or marker touches a link. For a 300-paragraph document with a bare URL in each paragraph,$convertToMarkdownStringtakes about 30 ms against about 7.5 ms onmain(12.6 ms without the URLs). An early exit when no link run sits next to a marker or reference would help apps that export on every change. - Test plan. The
### Beforeblock shows the original head failing theMarkdownEscaping.test.tsround trips. AGENTS.md (Commits and Pull Requests) asks for the real failing output before the change, so pastingmain's run of the new tests would show what they catch.
| }; | ||
| return true; | ||
| } | ||
| if (length === end) { |
There was a problem hiding this comment.
Optional: when no pair is found in prefixes that end before end, this keeps the whole-line scan, but the pair the edge run opens can contain the next emphasis. [x](u)**:*a.** imports on main as x, *, italic :a. (the run after the link read beside the edge of a line, as the description intends) and here as x, **:, italic a., *. Returning false when end came from the next emphasis or code span, so the rest is scanned on its own, would keep the edge reading.
| ) => void, | ||
| ): number[] { | ||
| // Every URL and email starts with one of these. | ||
| if (!values.some((value, i) => kinds[i] === TEXT && LINKISH.test(value))) { |
There was a problem hiding this comment.
Optional: any text piece with www., :// or @ sends the whole paragraph through the masked copy and typed arrays below, even when no reference or marker touches a link. A 300-paragraph document with one bare URL per paragraph exports in about 30 ms against about 7.5 ms on main. An early exit when no link run is next to a marker or reference would recover most of that.
… no URL can need spoiling ## Description Export scanned every paragraph whose text holds `www.`, `://` or `@` for bare URLs to spoil, building a masked copy of the paragraph and running counts over it, even where nothing could need it. A URL is spoiled only for a reference that is written, or for one that runs on to the end of its text into markers or raw output, so the scan now runs only for a paragraph that writes a reference or has such a URL. Output is unchanged. ## Test plan ### Before A 300-paragraph document, each paragraph with a bare URL and one bold word, median of 15 runs of `$convertToMarkdownString`: ``` url=true cur 7.9ms url=true main 3.0ms ``` ### After ``` url=true cur 4.2ms url=true main 3.0ms $ pnpm exec vitest run --project unit packages/lexical-markdown Test Files 9 passed (9) Tests 1063 passed (1063) ``` The export was also compared with the previous commit on 45,000 random paragraphs with URLs, emails, references, markers and raw output, with no difference. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Thanks for this! The change looks good to me: classifying punctuation and whitespace with the Unicode sets follows CommonMark 0.31.2, the new tests fail on main and pass here, and the exported references read back through both @lexical/markdown and @lexical/mdast. A few suggestions, all optional and non-blocking:
-
Quadratic escape with
shouldPreserveNewLines(packages/lexical-markdown/src/MarkdownExport.ts, line 566)./\\+(?=[*_`~])|[*_`~]/gretries\\+from every backslash of a run that no marker follows, so exporting a text node of 16,000 backslashes takes about 570 ms, against 0.1 ms onmain. Matching the whole run and checking the character after it gives the same output in linear time:output = output.replace( /\\+|[*_`~]/g, (escaped, offset: number, text: string) => escaped[0] !== '\\' ? `\\${escaped}` : /[*_`~]/.test(text[offset + escaped.length] ?? '') ? escaped + escaped : escaped, );
-
Stored CJK Markdown (a call for maintainers). Markdown that
mainexported and read back as bold, such as**注意:**请先登录, now imports with literal asterisks, and saving it again writes\*\*注意:\*\*请先登录. That is what CommonMark specifies, and the changeset'sVerify:says so. Bold labels that end in full-width punctuation right before CJK text are common in Chinese and Japanese writing, so it may be worth shipping the CJK-friendly follow-up in the same release. -
Export cost beside bare URLs (
MarkdownExport.ts,spoilLinks, line 872). Any paragraph whose text containswww.,://or@builds the masked copy and the prefix sums even when no reference or marker touches a link; 300 such paragraphs export in about 11 ms against about 2 ms onmain. An early exit for that case would help apps that export on every change. -
A run right after a link (
packages/lexical-markdown/src/importTextFormatTransformer.ts,findPairAtEdge, lines 153–197).[x](u)**:*a.**imports onmainasx,*, italic:a., reading the run after the link beside the edge of a line as the description says, but here asx,**:, italica.,*. Returningfalsewhenendcomes from the next emphasis or code span, so that the rest is scanned on its own, would keep the edge reading. It is a narrow shape. -
Test plan. The
### Beforeblock shows the original head merged withmainfailing theMarkdownEscaping.test.tsround trips. AGENTS.md (Commits and Pull Requests) asks for the real failing output of the command before the change, so pastingmain's run of the new tests would show what they catch.
| output = output.replace(/\\+(?=[*_`~])|[*_`~]/g, escaped => | ||
| escaped[0] === '\\' ? escaped + escaped : `\\${escaped}`, | ||
| ); |
There was a problem hiding this comment.
Optional: \\+(?=[*_~])is retried from every backslash of a run that no marker follows and backtracks through the rest of it each time, so this is quadratic in the run (16,000 backslashes: about 570 ms, against 0.1 ms onmain`). Matching the whole run and checking the next character gives the same output in linear time:
| output = output.replace(/\\+(?=[*_`~])|[*_`~]/g, escaped => | |
| escaped[0] === '\\' ? escaped + escaped : `\\${escaped}`, | |
| ); | |
| output = output.replace( | |
| /\\+|[*_`~]/g, | |
| (escaped, offset: number, text: string) => | |
| escaped[0] !== '\\' | |
| ? `\\${escaped}` | |
| : /[*_`~]/.test(text[offset + escaped.length] ?? '') | |
| ? escaped + escaped | |
| : escaped, | |
| ); |
| }; | ||
| return true; | ||
| } | ||
| if (length === end) { |
There was a problem hiding this comment.
Nit: when no pair is found before end and end comes from the next emphasis or code span, this keeps the whole-line scan, which read the edge run beside the text match. So [x](u)**:*a.** imports as x, **:, italic a., * instead of main's x, *, italic :a.. Returning false here in that case (so the rest is scanned on its own) would keep the edge reading the description promises.
There was a problem hiding this comment.
Thanks for this! The change looks good to me: classifying punctuation and whitespace with CommonMark 0.31.2's Unicode sets is what the spec asks for, the new tests fail on main and pass here, and the exported references read back through both @lexical/markdown and @lexical/mdast. A few suggestions, all optional and non-blocking:
-
Quadratic escape with
shouldPreserveNewLines(packages/lexical-markdown/src/MarkdownExport.ts, line 566)./\\+(?=[*_`~])|[*_`~]/gretries\\+from every backslash of a run that no marker follows, so exporting a text node of 16,000 backslashes takes about half a second, against well under a millisecond onmain. Matching the whole run and checking the character after it gives the same output in linear time:output = output.replace( /\\+|[*_`~]/g, (escaped, offset: number, text: string) => escaped[0] !== '\\' ? `\\${escaped}` : /[*_`~]/.test(text[offset + escaped.length] ?? '') ? escaped + escaped : escaped, );
-
Stored CJK Markdown (a call for maintainers). Markdown that
mainexported and read back as bold, such as**注意:**请先登录, now imports with literal asterisks, and saving it again writes\*\*注意:\*\*请先登录. That is what CommonMark specifies and how@lexical/mdastalready reads it, and the changeset'sVerify:paragraph says so. Bold labels that end in full-width punctuation right before CJK text are common in Chinese and Japanese writing, so it may be worth shipping the CJK-friendly follow-up in the same release. -
Test plan. The
### Beforeblock shows the original head merged withmainfailing theMarkdownEscaping.test.tsround trips. AGENTS.md (Commits and Pull Requests) asks for the real failing output of the command before the change, so pastingmain's run of the new tests would show what they catch.
The CI workflows have not run on this head yet; they are waiting for a maintainer to approve them for the fork.
| output = output.replace(/\\+(?=[*_`~])|[*_`~]/g, escaped => | ||
| escaped[0] === '\\' ? escaped + escaped : `\\${escaped}`, | ||
| ); |
There was a problem hiding this comment.
Optional, non-blocking: \\+(?=[*_~])is retried from every backslash of a run that no marker follows, so this is quadratic in the run (16,000 backslashes take about half a second to export, against well under a millisecond onmain`). Matching the whole run and checking the character after it gives the same output in linear time:
output = output.replace(
/\\+|[*_`~]/g,
(escaped, offset: number, text: string) =>
escaped[0] !== '\\'
? `\\${escaped}`
: /[*_`~]/.test(text[offset + escaped.length] ?? '')
? escaped + escaped
: escaped,
);
Description
Markdown import decides whether a
*,_,~~or==run can open or close formatting by looking at the characters on either side of it. The punctuation and whitespace classes listed only ASCII characters, so an en dash, a curly quote, a currency sign or an emoji next to a run counted as a letter.See –**[here](https://example.com)** now.imported as plain text, and its next export escaped the asterisks.CommonMark 0.31.2 counts every character in the Unicode P and S categories as punctuation, and Zs plus tab, LF, FF and CR as whitespace. Import and the typing shortcuts now use those sets, reading whole code points so astral characters such as 😀 are classified correctly. The backslash, missing from the old ASCII list, now counts as punctuation too.
Export has to follow: Lexical wrote shapes like
foo**“bar”**baz, whose markers no longer flank, and 7 of theMarkdownEscaping.test.tsround trips from #9481 failed. The exporter now writes the letter beside such a marker as a numeric character reference (foo**“bar”**baz), which every CommonMark reader decodes to the same text. It writes none inside code or transformer output. A raw backslash right before a marker, or before a character that export escapes withshouldPreserveNewLines, is escaped too. The references are chosen so that micromark, which@lexical/mdastimports with and which reads an emoji as a letter, formats the result the same way (😀**“b”**).A bare URL,
www.host or email that GFM readers would link needs care: a reference inside it changes the link, and GFM readers take markers and code spans after it into the link. When either would happen, export writes the.ofwww.or the:after the scheme as a reference, so that no reader links it and the text and formatting come through (**www.x.co**a,see https://x.com/a**“x”**). An email that holds a reference has its@written as@, so that micromark links the whole decoded address.Import fixes that the new flanking exposed:
**a.**x**`c`**).[or code spans.*)**a*). Text beside a text match that does not cut a run still reads the match's edge as the edge of a line, as on main, so a custom syntax that ends in a letter (user:bob**(admin)**) keeps the formatting after it.x\``bb`x), which export writes for a plain backtick before inline code.Import now finds a line's code spans, emphasis and links, and the matches of other text-match transformers whose patterns do not look behind, once rather than again after each match, and imports the text after each match in a loop, in the same order as before. Long lines import faster than on main, for example 4 s instead of 9 s for
'a **b** [c](d) `e` _f_ 'repeated 1,600 times, and very long paragraphs no longer overflow the stack.Markdown such as
**你好。**世界, which main read as bold, now imports with literal asterisks, as CommonMark, GFM, micromark and markdown-it show it; a CJK-friendly rule for both importers is left for a follow-up.Known limitations:
x~~[_](u)[~~](u), as on main.==in text is not escaped, so it can pair with a highlight marker that both opens and closes. With Unicode punctuation that now includes a highlight next to an emoji or symbol, not only next to ASCII punctuation as on main.\_, become part of the link in GFM readers, as on main.\vbeside a marker cannot be written so that both CommonMark and micromark readers agree.Reported in sveltia/sveltia-cms#1107.
Closes #9407
Closes #9410
Closes #9443
Changeset
.changeset/markdown-unicode-punctuation-flanking.md, a[fix]patch for@lexical/markdown. ItsVerify:paragraph covers stored Markdown such as**“bar”**bazanda**\_x**, which now imports with literal asterisks as CommonMark readers show it, and the new character references in export.Test plan
Command:
pnpm exec vitest run --project unit packages/lexical-markdownBefore
With main merged into the original PR head (5debdb3):
After
The new tests cover Unicode and astral punctuation, spec whitespace, character references read back by both
@lexical/markdownand@lexical/mdast, formatting beside bare URLs and emails, code-span and link precedence, cut delimiter runs and custom text-match transformers. Import was also compared with markdown-it on random inputs, where it agrees on many more of them than main does, and export round trips were checked through both importers and cmark-gfm on random paragraphs, with none that main got right and this gets wrong.