gh-154682: Fix corruption of locale-encoded text on FreeBSD, NetBSD, DragonFly BSD and macOS

[serhiy-storchaka]

On FreeBSD, NetBSD and DragonFly BSD, wchar_t values are not Unicode code points in non-UTF-8 locales: mbstowcs() and wcstombs() keep the raw locale values (e.g. the KOI8-U byte 0xD0 becomes 0x00D0, and the EUC-JP byte pair 0xB7 0xEE becomes 0xB7EE), and the collation tables and the curses library work with such values. This corrupted every interface which passes locale-encoded text through wchar_t:

• time.strftime() (and therefore datetime and time.strptime()) and locale.nl_langinfo() returned mojibake (e.g. 'ÐÏÎÅĦÌÏË' instead of 'понеділок' in the uk_UA.KOI8-U locale);
• locale.localeconv() and format(n, 'n') returned mojibake monetary and numeric strings;
• localized OSError, ctypes and other error messages were mojibake;
• sys.argv and other startup data could not round-trip through os.fsencode();
• locale.strcoll() and locale.strxfrm() produced wrong collation (the collation tables are keyed by the raw locale values, and characters missing from the table are compared by their numeric value);
• the wide-character curses API could not display or read non-ASCII characters at all (Cyrillic text in a Cyrillic locale was rendered as blanks).

The fix converts between the locale encoding and Unicode with iconv() in _Py_DecodeLocaleEx() and _Py_EncodeLocaleEx() -- unlike the codec machinery, iconv() is usable in early interpreter startup, and the C library's iconv() supports all encodings used by its own locales. time.strftime() uses strftime() instead of wcsftime() (whose output is non-Unicode wchar_t), the locale module uses strcoll() and strxfrm() on the locale-encoded strings, and the curses module converts between Unicode and the native wchar_t form on the library boundary. If iconv() is not functional for the locale encoding, the current behavior is kept.

An alternative -- converting globally in PyUnicode_FromWideChar() and PyUnicode_AsWideChar() like on Oracle Solaris (HAVE_NON_UNICODE_WCHAR_T_REPRESENTATION) -- was prototyped and rejected: wchar_t at that boundary is also used as a plain Unicode container (e.g. os.path.normpath() is implemented on wchar_t), and on these platforms the raw locale values overlap with Unicode code points, so no boundary policy can distinguish the two uses. It broke 46-126 test files, either with hard errors or with silent corruption (os.path.normpath('café') returned 'cafФ' in the KOI8-U locale).

Verified on FreeBSD 15.1, NetBSD 10.1 and DragonFly BSD 6.4 with Ukrainian, Greek, German, Turkish, Catalan and Japanese locales in eight encodings: the full test suite has no regressions, previously failing tests (test_cmd_line.test_invalid_utf8_arg) now pass, and test_curses passes in non-UTF-8 locales with the ncursesw backend. The full suite in the uk_UA.KOI8-U locale on FreeBSD now passes with zero failures.

Note: test_logging.test_listen_server_error in non-UTF-8 locales additionally needs the gh-93251 fix (#154683).

• Issue: Locale-encoded text is corrupted on FreeBSD, NetBSD, DragonFly BSD and macOS in non-UTF-8 locales #154682