[lexical-markdown] Bug Fix: Follow CommonMark's Unicode punctuation and whitespace for emphasis flanking

.changeset/markdown-unicode-punctuation-flanking.md

@@ -0,0 +1,13 @@
+---
+'@lexical/markdown': patch
+---
+
+[fix] Follow CommonMark's Unicode punctuation and whitespace when deciding whether `*`, `_`, `~~` and `==` markers open or close formatting
+
+Markdown import and the typing shortcuts now count every character in the Unicode punctuation (P) and symbol (S) categories as punctuation, so an en dash, a curly quote, a currency sign or a symbol such as ✓ or 😀 next to a marker no longer reads as a letter, and `See –**[here](https://example.com)**` imports as a bold link. On import, the backslash, which was missing from its ASCII list, counts too. Whitespace is the Zs category plus tab, line feed, form feed and carriage return, so U+000B, U+2028, U+2029 and U+FEFF no longer count as whitespace beside a marker, on import or for the typing shortcuts.
+
+Import also applies emphasis that closes before a later code span ahead of that span, as in `` **a.**x**`c`** ``, so its markers no longer split apart and stay as literal text. A link or other text match that ends before a code span is applied ahead of it, so `[a](b) and [`c`](d)` keeps both links, and one no longer takes part of any code span in the text, not only the first. A delimiter run that an earlier match cuts through now reads as it did in the whole text instead of as one at the edge of a line, so `*)**a*` keeps its literal asterisks. An escaped backtick right before more backticks no longer keeps the rest of that run from opening a code span, so ```x\``bb`x``` imports with code `bb`, as CommonMark reads it. Long lines import faster, since their code spans, emphasis, links and the matches of other text-match transformers whose patterns do not look behind are found once per line rather than again after each match.
+
+Export of formatting that starts or ends with punctuation right beside a letter, such as plain `foo` then bold `“bar”`, now writes that letter as a numeric character reference (`foo**“bar”**baz`), so that the formatting survives a round trip through a CommonMark reader. Those references are chosen so that micromark-based readers such as `@lexical/mdast`, which read an emoji or another character outside the Basic Multilingual Plane as a letter, format it the same way, so plain `😀` then bold `“b”` exports as `😀**“b”**`. Characters that are neither punctuation nor whitespace count as letters here, including the variation selector that ends an emoji such as ❤️. A raw backslash that ends text right before a marker or such a reference (text exported with `shouldPreserveNewLines`, or decorator text) now goes out as `\`, so that it does not escape it. With `shouldPreserveNewLines`, backslashes right before a `*`, `_`, `` ` `` or `~` of the text are escaped too, so that they no longer cancel that character's escape. A bare URL, `www.` host or email that GFM readers would link, and that would hold such a reference or take in the markers or code span after it, now has a character of its prefix written as a reference too, so that no reader links it and the formatting and text come through: bold `www.x.co` then `a` exports as `**www.x.co**a`, and plain `see https://x.com/a` then bold `“x”` as `see https://x.com/a**“x”**`, where a GFM reader used to take the markers into the link. `@lexical/mdast` still links the decoded text.
+
+Verify: Markdown with a marker between a letter and Unicode punctuation, such as `**“bar”**baz`, `**你好。**世界` or `**注意：**请先登录`, now imports with literal asterisks, as CommonMark readers show it, where it used to import as bold. The same goes for formatting that starts or ends with an escaped ASCII character right beside a letter, such as `a**\_x**`, since the backslash is punctuation. Lexical exported these shapes for such formatting before, so check stored Markdown that has them. Exported Markdown for such formatting, and for formatting beside a bare URL, now holds the character references described above, so check export snapshots too.

packages/lexical-markdown/README.md

@@ -99,6 +99,18 @@ reference definition or a check list item, so text such as `[1]` and `array[0]`
 exports as typed. Escaped brackets and guarded `<`/`&` use numeric references so
 escaping does not change adjacent emphasis flanking.
 
+Import follows CommonMark's Unicode punctuation and whitespace when deciding
+whether `*`, `_`, `~~` and `==` open or close formatting. When formatted text
+starts or ends with punctuation right beside a letter, export writes that
+letter as a numeric reference (`fo&#111;**“bar”**`) so that the markers still
+open and close, for micromark-based readers such as `@lexical/mdast` too. This
+holds with `shouldPreserveNewLines` as well, where a raw backslash right before
+such a marker or reference goes out as `&#92;`. Where a bare URL, `www.` host
+or email that GFM readers would link holds such a reference, or would take in
+the markers or code span after it, export writes a character of its prefix as
+a reference too (`www&#46;x.co*a*b`), so that no reader links it and the
+formatting comes through.
+
 There're three types of transformers:
 
 - **Element transformer** handles top level elements (lists, headings, quotes, tables or code blocks)