Skip to content

unicode/norm: do not compose across a ccc=0 character that combines backward - #69

Open
tannevaled wants to merge 1 commit into
golang:masterfrom
tannevaled:norm-blocking-ccc0-combines-backward
Open

unicode/norm: do not compose across a ccc=0 character that combines backward#69
tannevaled wants to merge 1 commit into
golang:masterfrom
tannevaled:norm-blocking-ccc0-combines-backward

Conversation

@tannevaled

@tannevaled tannevaled commented Aug 23, 2026

Copy link
Copy Markdown

UAX #15, D115: a character C is blocked from starter A when some B between
them has ccc(B) = 0 or ccc(B) >= ccc(C). reorderBuffer.compose tracks the
current starter in s, but only ever updates it inside the combinesBackward
branch. That is enough for a ccc = 0 character that does not combine
backward, because such a character ends the segment and never shares a
buffer with an earlier starter. One that does combine backward has to stay
in the buffer, since it may compose with what precedes it; when it does not,
it falls through with s still pointing at the earlier starter, and the next
mark composes across it.

norm.NFC.String("iা̖̀")
  got  "ìা̖"       -- i + U+0300 composed across U+09BE
  want "iা̖̀"

Set s when the character just written is itself a starter.

The bug is not new and not tied to a particular Unicode version. Of the 35
characters in Unicode 17 that have ccc = 0 and are the second element of a
canonical composition, NFC(<i, S, U+0300, U+0316>) fails to keep the "i" for
24 of them under the Unicode 15 tables and 33 under Unicode 17 -- among them
U+09BE Bengali, U+0BBE Tamil, U+0D3E Malayalam, U+1B35 Balinese and U+102E
Myanmar. The nine that differ between the two table sets are simply those
Unicode 15 had not assigned yet. The two that never fail, U+0FB5 and U+0FB7,
are composition exclusions.

The added test uses U+09BE, which is assigned with ccc = 0 in both table
sets, so it does not depend on which toolchain selects which tables, and it
is paired with U+0903 -- ccc = 0, does not combine backward -- so the two
cases sit either side of the distinction.

Verified beyond the package's own tests: NFC, NFKC and NFD of 3112063 inputs
were compared before and after, and every one of the 2778 (Unicode 15) and
4153 (Unicode 17) differing results was judged by a separate UAX #15
implementation built from UnicodeData.txt, itself checked against
NormalizationTest.txt at 305184 and 320544 assertions with no failures. All
of the differences are corrections; there are no regressions.

Fixes golang/go#81001

@google-cla

google-cla Bot commented Aug 23, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@gopherbot

Copy link
Copy Markdown
Contributor

This PR (HEAD: eab86a8) has been imported to Gerrit for code review.

Please visit Gerrit at https://go-review.googlesource.com/c/text/+/821020.

Important tips:

  • Don't comment on this PR. All discussion takes place in Gerrit.
  • You need a Gmail or other Google account to log in to Gerrit.
  • To change your code in response to feedback:
    • Push a new commit to the branch used by your GitHub PR.
    • A new "patch set" will then appear in Gerrit.
    • Respond to each comment by marking as Done in Gerrit if implemented as suggested. You can alternatively write a reply.
    • Critical: you must click the blue Reply button near the top to publish your Gerrit responses.
    • Multiple commits in the PR will be squashed by GerritBot.
  • The title and description of the GitHub PR are used to construct the final commit message.
    • Edit these as needed via the GitHub web interface (not via Gerrit or git).
    • You should word wrap the PR description at ~76 characters unless you need longer lines (e.g., for tables or URLs).
  • See the Sending a change via GitHub and Reviews sections of the Contribution Guide as well as the FAQ for details.

@gopherbot

Copy link
Copy Markdown
Contributor

Message from Gopher Robot:

Patch Set 1:

(1 comment)


Please don’t reply on this GitHub thread. Visit golang.org/cl/821020.
After addressing review feedback, remember to publish your drafts!

@gopherbot

Copy link
Copy Markdown
Contributor

Message from Gopher Robot:

Patch Set 1:

Congratulations on opening your first change. Thank you for your contribution!

Next steps:
A maintainer will review your change and provide feedback. See
https://go.dev/doc/contribute#review for more info and tips to get your
patch through code review.

Most changes in the Go project go through a few rounds of revision. This can be
surprising to people new to the project. The careful, iterative review process
is our way of helping mentor contributors and ensuring that their contributions
have a lasting impact.

During May-July and Nov-Jan the Go project is in a code freeze, during which
little code gets reviewed or merged. If a reviewer responds with a comment like
R=go1.11 or adds a tag like "wait-release", it means that this CL will be
reviewed as part of the next development cycle. See https://go.dev/s/release
for more details.


Please don’t reply on this GitHub thread. Visit golang.org/cl/821020.
After addressing review feedback, remember to publish your drafts!

…ackward

UAX golang#15, D115: a character C is blocked from starter A when some B between
them has ccc(B) = 0 or ccc(B) >= ccc(C). reorderBuffer.compose tracks the
current starter in s, but only ever updates it inside the combinesBackward
branch. That is enough for a ccc = 0 character that does not combine
backward, because such a character ends the segment and never shares a
buffer with an earlier starter. One that does combine backward has to stay
in the buffer, since it may compose with what precedes it; when it does not,
it falls through with s still pointing at the earlier starter, and the next
mark composes across it.

    norm.NFC.String("iা̖̀")
      got  "ìা̖"       -- i + U+0300 composed across U+09BE
      want "iা̖̀"

Set s when the character just written is itself a starter.

The bug is not new and not tied to a particular Unicode version. Of the 35
characters in Unicode 17 that have ccc = 0 and are the second element of a
canonical composition, NFC(<i, S, U+0300, U+0316>) fails to keep the "i" for
24 of them under the Unicode 15 tables and 33 under Unicode 17 -- among them
U+09BE Bengali, U+0BBE Tamil, U+0D3E Malayalam, U+1B35 Balinese and U+102E
Myanmar. The nine that differ between the two table sets are simply those
Unicode 15 had not assigned yet. The two that never fail, U+0FB5 and U+0FB7,
are composition exclusions.

The added test uses U+09BE, which is assigned with ccc = 0 in both table
sets, so it does not depend on which toolchain selects which tables, and it
is paired with U+0903 -- ccc = 0, does not combine backward -- so the two
cases sit either side of the distinction.

Verified beyond the package's own tests: NFC, NFKC and NFD of 3112063 inputs
were compared before and after, and every one of the 2778 (Unicode 15) and
4153 (Unicode 17) differing results was judged by a separate UAX golang#15
implementation built from UnicodeData.txt, itself checked against
NormalizationTest.txt at 305184 and 320544 assertions with no failures. All
of the differences are corrections; there are no regressions.

Fixes golang/go#81001
@tannevaled
tannevaled force-pushed the norm-blocking-ccc0-combines-backward branch from eab86a8 to aa7e629 Compare August 26, 2026 12:31
@gopherbot

Copy link
Copy Markdown
Contributor

This PR (HEAD: aa7e629) has been imported to Gerrit for code review.

Please visit Gerrit at https://go-review.googlesource.com/c/text/+/821020.

Important tips:

  • Don't comment on this PR. All discussion takes place in Gerrit.
  • You need a Gmail or other Google account to log in to Gerrit.
  • To change your code in response to feedback:
    • Push a new commit to the branch used by your GitHub PR.
    • A new "patch set" will then appear in Gerrit.
    • Respond to each comment by marking as Done in Gerrit if implemented as suggested. You can alternatively write a reply.
    • Critical: you must click the blue Reply button near the top to publish your Gerrit responses.
    • Multiple commits in the PR will be squashed by GerritBot.
  • The title and description of the GitHub PR are used to construct the final commit message.
    • Edit these as needed via the GitHub web interface (not via Gerrit or git).
    • You should word wrap the PR description at ~76 characters unless you need longer lines (e.g., for tables or URLs).
  • See the Sending a change via GitHub and Reviews sections of the Contribution Guide as well as the FAQ for details.

@gopherbot

Copy link
Copy Markdown
Contributor

Message from t hepudds:

Patch Set 2:

(1 comment)


Please don’t reply on this GitHub thread. Visit golang.org/cl/821020.
After addressing review feedback, remember to publish your drafts!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

x/text/unicode/norm: NFC/NFKC composes across a ccc=0 character that combines backward (24-33 of 35 affected, not go1.27-specific)

2 participants