Bug report

Bug description:

Current behavior Python 3.16.0a0:

By the time that an incremental UTF-8 parser reads ed aX or ed bX it should know that the resulting codepoint will be in the surrogate range. But Python's codecs.getincrementaldecoder("UTF-8")() decoder delays raising an exception or emitting U+FFFD ('�') characters until it reads the third byte.

Low surrogates: U+D800 (ed a0 80) to U+DBFF (ed af bf)

High surrogates: U+DC00 (ed b0 80) to U+DFFF (ed bf bf)

"strict" error handling:

import codecs

decoder = codecs.getincrementaldecoder("UTF-8")()

s = decoder.decode(b"\xed\xa0")
print(repr(s))
# ''
s = decoder.decode(b"\x80")
print(repr(s))
# UnicodeDecodeError: 'utf-8' codec can't decode byte 0xed
#                     in position 0: invalid continuation byte

"replace" error handling:

import codecs

decoder = codecs.getincrementaldecoder("UTF-8")("replace")

s = decoder.decode(b"\xed\xa0")
print(repr(s))
# ''
s = decoder.decode(b"\x80")
print(repr(s))
# '���'

Whereas with normal, non-incremental decoding, it seems that the exception below acknowledges that eg ed a0 can be immediately deduced as invalid. The error message is a bit cryptic, as #67802 points out in general, but ignoring the 'position 0' part, it seems the error is not caused by what would be an 'end of data' error, as can be seen with eg just the ed on its own in the second exception below, but by being in the surrogate range.

b"\xed\xa0".decode()
# UnicodeDecodeError: 'utf-8' codec can't decode byte 0xed
#                     in position 0: invalid continuation byte

b"\xed".decode()
# UnicodeDecodeError: 'utf-8' codec can't decode byte 0xed
#                     in position 0: unexpected end of data

The number of U+FFFD ('�'), one per byte in the surrogate bytes, seems consistent with how Python implements 'maximal subpart' decoding, part 3.9.6 of the Unicode (17) standard, but I think the incremental decoder should start emitting them at the second, not third, byte of a surrogate sequence, right?

CPython versions tested on:

3.16, 3.14

Operating systems tested on:

Linux