Repository navigation
iconv_codec NG #38255
Description
| BPO | 713820 |
|---|---|
| Nosy | @loewis, @hyeshik |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2003-04-03.05:07:58.000>
created_at = <Date 2003-04-02.11:10:16.000>
labels = ['library']
title = 'iconv_codec NG'
updated_at = <Date 2003-04-03.05:07:58.000>
user = 'https://github.com/hyeshik'bugs.python.org fields:
activity = <Date 2003-04-03.05:07:58.000>
actor = 'loewis'
assignee = 'none'
closed = True
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2003-04-02.11:10:16.000>
creator = 'hyeshik.chang'
dependencies = []
files = ['5152']
hgrepos = []
issue_num = 713820
keywords = ['patch']
message_count = 2.0
messages = ['43268', '43269']
nosy_count = 2.0
nosy_names = ['loewis', 'hyeshik.chang']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue713820'
versions = ['Python 2.3']
Activity
hyeshik commentedon Apr 2, 2003
This new implementation of iconv_codec resolves
problems of current
implementations below:
Having a reentrant context vulnerable point:
encoder and/or decoder
can be called multiple level in a same time when PEP-293
codec error
callback can call another iconv encoder session, too.
So, all encode/
decode session must open their own iconv session but
the current
implementation shares the iconv session in the whole
codec life time.
StreamReader can't work correctly: Because iconv
keeps their context
private, StreamReader can't work smart only with
encode/decode function.
Also, handling EINVAL and giving pending characters
from previous data
to error callback is very weak in the current
implementation.
Putting a replacement character as just '?' is not
safe for many
encodings: On stateful encodings and non-byte stream
encodings, we need
to encode with iconv even for the replacement character.
Can't use encoding names including - and
uppercases: Because codec
subsystem changes - to _ and uppercases to lowercases,
we can't pass
them to the iconv_codec module without loss. For
example, we need the
next aliases to use CJK encodings on Sun iconv:
# simplified chinese
"euc_cn": "zh_CN.euc",
"iso_2022_zh": "zh_CN.iso2022-CN",
"gbk": "zh_CN.gbk",
"cp935": "zh_CN-cp935",
# traditional chinese
"euc_tw": "zh_TW.euc",
"iso_2022_tw": "zh_TW.iso2022-7",
"big5": "zh_TW.big5",
"cp937": "zh_TW.cp937",
# japanese
"iso_2022_jp": "ISO-2022-JP",
"euc_jp": "eucJP",
"shift_jis": "PCK",
# korean
"euc_kr": "ko_KR.euc",
"iso_2022_kr": "ISO-2022-KR",
"johab": "ko_KR.johap",
"cp932": "ko_KR.cp932",
"cp949": "ko_KR.cp949",
some iconv
implementations like of HP-UX or Solaris, UCS2 ->
ISO-8859-1 is
available but UCS2 -> euc-kr isn't avaiable and only
UTF-8 -> euc-kr is.
And, many multibyte codecs such as CJK or iconv might
have duplicated
code for processing error callbacks and handling
Streams. So, I splitted
them out to another source. CJK and iconv codecs can
share them just in
source level by putting multibytecodec.c to Modules/
and linking the
file to each of the codecs. Alternatively, if
multibytecodec.c goes
to Python/ and is linked to main python library, the
codecs can be
compiled and loaded by themselves.
multibytecodec.c, the common multibyte codec framework
can be used by
any usual multibyte encodings. By using it, some codec
writer can create
a codec for his/her multibyte encodings without any
care for handling
error callbacks or implementing StreamReader structure.
I wrote CJK codecs using it. and
will submit a patch in an individual patch report.
loewis commentedon Apr 3, 2003
Logged In: YES
user_id=21627
Because of complaints about its brokenness, I had to revert
the current iconv codec. It is extremely unlikely that such
code will be added for Python 2.3. So please update this
submission for the current CVS. There is no need to hurry,
though.