Skip to content

iconv codecs corrupt stateful encodings when decoding incrementally #154859

Description

@fedonman

Bug description:

The iconv-backed codecs start a fresh conversion on every call, so an incremental decode of
a stateful encoding loses the shift state and silently returns wrong text. iso-2022-cn has
no built-in codec, so the plain name reaches the iconv codec:

import codecs
blob = 'ABC中文DEF'.encode('iso-2022-cn')
print('one-shot:   ', blob.decode('iso-2022-cn'))
d = codecs.getincrementaldecoder('iso-2022-cn')()
print('incremental:', ''.join(d.decode(bytes([b])) for b in blob) + d.decode(b'', True))
one-shot:    ABC中文DEF
incremental: ABCVPNDDEF

codecs.iterdecode() and StreamReader.read(1) are wrong in the same way. A stateful
encoding forced with the iconv: prefix is also affected, even when it has a built-in
codec.

Either the codec objects need to keep one conversion across calls, or the iconv search
function should refuse stateful encodings, which would make iso-2022-cn a LookupError.

The iconv codecs are new in 3.16 (gh-152997), so no released version is affected.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.16new features, bugs and security fixesstdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

    Projects

    Status
    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions