Allow a wider range of unicode characters in text messages - #1443
Draft
quis wants to merge 6 commits into
Draft
Conversation
quis
force-pushed
the
allow-unicode-sms
branch
4 times, most recently
from
August 10, 2026 14:29
9083039 to
be8a6f8
Compare
quis
force-pushed
the
count-unicode-characters-in-sms
branch
3 times, most recently
from
August 10, 2026 15:51
acb5ec4 to
65f3ef4
Compare
This is a proof of concept, need to think more about how we’d roll this out
We can treat it like any other unicode character now, for SMS. When sanitising text to ASCII I think it’s still nice to treat this as a special case, otherwise it will get downgraded to ?
We don’t treat these any differently to other non-GSM characters now.
It’s just as alias for `SanitiseSMS.encode` which adds another lay of indeirection, and it’s not used by any of the apps.
The zero width joiner character is used to combine multiple emoji into new, novel emoji. Therefore stripping it out can cause these newer emjoi to decompose into their constituent parts, which looks odd. We don’t want to introduce a breaking change by changing the behaviour of `formatters.normalise_whitespace` etc, which are used all over the place. So instead we add an optional flag which the SMS template classes can pass in.
quis
force-pushed
the
allow-unicode-sms
branch
2 times, most recently
from
August 10, 2026 16:24
f89a52d to
cdbb4ad
Compare
The order of operations should always be: - remove whitespace before punctuation - encode the resulting content Otherwise things which are downgraded to full stops (for example `…` to `...`) will have the whitespace removal applied, which might not be appropriate. `SMSMessageTemplate` was working this way already, but `SMSPreviewTemplate` and `SMSBodyPreviewTemplate` were doing it the other way. This commit ensures they are all consistent. --- We need to replace lower quotation marks with ASCII ones because at least one of our providers does this for us. So if we don’t do it too we could end up with a mismatch between what we send (straight quotes) and what we charge our users for (low quotes).
quis
force-pushed
the
allow-unicode-sms
branch
from
August 10, 2026 16:53
cdbb4ad to
c798a63
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a proof of concept, need to think more about how we’d roll this out to the apps.