Universal Unicode Codec for POSIX iconv

@malemburg

BPO	670715
Nosy	@malemburg, @loewis, @doerwalter, @hyeshik
Files	iconvcodec-1.txt: Patch iconvcodec-2.txt: Revised patch (r2) iconvcodec-3.txt diff-char.txt: Use '0' as test char. diff-debug.txt: Add debugging output to see what iconv() generated.

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/loewis'
closed_at = <Date 2003-04-03.05:04:51.000>
created_at = <Date 2003-01-19.16:51:35.000>
labels = ['library']
title = 'Universal Unicode Codec for POSIX iconv'
updated_at = <Date 2003-04-03.05:04:51.000>
user = 'https://github.com/hyeshik'

bugs.python.org fields:

activity = <Date 2003-04-03.05:04:51.000>
actor = 'loewis'
assignee = 'loewis'
closed = True
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2003-01-19.16:51:35.000>
creator = 'hyeshik.chang'
dependencies = []
files = ['4938', '4939', '4940', '4941', '4942']
hgrepos = []
issue_num = 670715
keywords = ['patch']
message_count = 14.0
messages = ['42479', '42480', '42481', '42482', '42483', '42484', '42485', '42486', '42487', '42488', '42489', '42490', '42491', '42492']
nosy_count = 5.0
nosy_names = ['lemburg', 'loewis', 'doerwalter', 'tzot', 'hyeshik.chang']
pr_nums = []
priority = 'normal'
resolution = 'accepted'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue670715'
versions = ['Python 2.3']

Here's the unicode codec using POSIX iconv(3) library.

Tested on these platforms and seems to work:
FreeBSD/i386, FreeBSD/alpha, FreeBSD/ia64,
FreeBSD/sparc64, MacOS X/ppc, HP-UX/pa-risc2

This codec implementation supports PEP-293, also.

Logged In: YES
user_id=38388

The patch looks good, but you'll need to add some form
of testing to underline the "seems to work" :-)

Some docs on how to use the codec would also be needed.

Martin von Loewis has written a similar codec some months ago.
Perhaps you two could get in touch and sort out the details ?!

Logged In: YES
user_id=55188

Thank you for comments. :->
I uploaded a new revised patch with unittest and some code
style fixes.

I saw Martin v. Loewis's iconvcodecs about a years ago.
His implementation is very neat, but it had a limit on error
handling due to recursive call.

Logged In: YES
user_id=21627

I'm quite happy with this patch, and will apply it shortly.
However, I am concerned that it is specific for GNU iconv.
IMO, there should be machinery to find out the "internal"
encoding, in case native the native iconv implementation is
used instead of GNU iconv.

Logged In: YES
user_id=55188

iconv implementations on commercial UNIXes are very scary.

Solaris implementation:
no support for CJK <-> UCS conversion.
They support UCS[24] only for iso-8859 and UTFs.

HP-UX implementation:
They have useless iconv. HP-UX iconv has no unicode support.

BSD implementation (Konstantin Chuguev's):
compatible with this patch (provides ucs-[24]-internal)

GLIBC implementation:
provides ucs-[24] and they are same with GNU iconv's
ucs-[24]-internal.
Because ucs-[24] of GNU/BSD implentation is big endian
always. We can't use ucs-[24] for every platform.

In conclusion, we must use 3rd party iconv on Solaris or HP-UX.
And, we need to detect whether the linked iconv is GNU/BSD
iconv or GLIBC iconv. (how?)

I'll investigate how to detect them, but ... :)

Logged In: YES
user_id=21627

Hmm. I see that Solaris does support conversion of CJK to
UTF-8. So even though we cannot convert into the internal
encoding, we could still convert through UTF-8.

Looking at /usr/lib/nls/iconv/config.iconv of HP-UX 11.11, I
see conversions from and to ucs2, for iso-8859, eucJP, sjis,
eucTW, big5, roc15, kore5, hp15CN, and many IBM code pages.

So I think the iconv codec should convert into the Python
internal representation if possible. If no encoding name for
that is known, it should convert to ucs2 (be) if possible,
or else to UTF-8; in all cases, it will then construct a
Unicode object from the resulting string.

Logged In: YES
user_id=21627

I have committed it now with minimal changes so that it
works on Linux, as

setup.py 1.138
__init__.py 1.15
iconv_codec.py 1.1
regrtest.py 1.117
NEWS 1.627

I will make further changes; please watch the CVS. If you
would like to make further changes, please submit patches
against the CVS.

Logged In: YES
user_id=55188

Thank you very much for your works.
I'm working on UCS endian detection and UCS transparent
wrapper for UTF-{8,16}. I'll submit new patch in a week.
Please feel free to change my codes because I'm not familiar
with python code convention and culture. :-)

Logged In: YES
user_id=89016

iconvcodec-3.txt does byteswapping under the following
conditions:
    #ifndef WORDS_BIGENDIAN
    #ifdef __GNU_LIBRARY__
Byteswapping is done before encoding to the whole input and
to every piece returned from iconv() for decoding.

Detecting whether byteswapping is neccessary might not work
reliably with the above tests. If this is the case, a test
call to iconv() should probably be done in
Modules/_conv_codec.c::init_iconv_codec() to determine
whether to byte swap or not.

Another possibility might be to use utf-16/utf-32 instead of
ucs-2/ucs-4.

One test still fails: test_sane(), because it uses the
internal encoding in Python, where the real endianness is of
course unknown. The test also assumes a narrow Python build.

Logged In: YES
user_id=89016

I checked in a version of iconvcodec-3.txt that does a byte
swapping check in the init function as
Modules/_iconv_codec.c 1.5

Logged In: YES
user_id=539787

I am afraid that SGI Irix' iconv must be added to the list of
scary commercial implementations... at first the module did
not compile due to two missing typecasts (see patch
680146), but even after that, the module fails (and python
dumps core):

Fatal Python error: can't initialize the _iconv_codec module:
iconv_open() failed
Abort (core dumped)

(message at line 674 of the module)

This is because Irix iconv knows nothing about ASCII
encoding...

I changed the "ASCII" argument to something
existing, "ISO8859_7" which is my country's encoding, and
then python dumps core with:

Fatal Python error: can't initialize the _iconv_codec module:
mixed endianess
Abort (core dumped)

MIPS processors are big endian.

python works fine in all my programs where there is no use of
str.decode and unicode.encode .
To make sure that the problem exists only in this module, I
need to compile without the _iconv_codec . Do I do that by
changing setup.py? This seems the way, but I haven't
succeeded yet.

Logged In: YES
user_id=89016

"ISO8859_7" is not known on Linux, can you try "ISO8859-7"
or better "ISO8859-1" or "LATIN1"?

Also I wonder whether it is a good thing to test iconv()
with the character '\x01'. Can you try diff-char.txt and see
what happens?

If all this fails, try diff-debug.txt and report what the
output is.

Logged In: YES
user_id=539787

It was a misspelling; ISO8859-7 is what I used in the module,
the underscore came from my using it with str.decode
("iso8859_7").

o2k 348# ls -l python
-rwxr-xr-x    1 root     sys       1976320 Feb  4 21:21 python
o2k 349# ./python
Python 2.3a1 (#12, Feb  4 2003, 21:14:08) [C] on irix646
Type "help", "copyright", "credits" or "license" for more 
information.
>>> "a".decode("ascii")
Fatal Python error: can't initialize the _iconv_codec module: 
iconv_open() failed
Abort (core dumped)

I then changed it to:
iconv_t hdl = iconv_open("UCS-2", "LATIN1");
(sys.maxunicode is 65535)
and it still fails at the same line (I ran "dbx python core"). I
made sure that I use the freshly built _iconv_codec.so (ie no
other exists on the system).
I added some code to show the errno, and it's 22 (EINVAL).
I ran iconv -l at the prompt, and used UCS-2 and ISO8859-1
(LATIN1 didn't show up in the list). Now, the code runs fine
for str.decode and unicode.encode, and test.test_codecs and
test.test_263 pass fine.

(BTW, if you're used to vi keystrokes, never press ESC while
typing with IE in the "Add a comment:" text box... :( )

I then applied the diff-debug patch, and got:
>>> "a".decode("ascii")
init_iconv_codec: 0x0030
u'a'

Please note that on Irix, UCS-2-INTERNAL is not available,
only UCS-2, so that had to be changed too.

It's an initialisation thing; I could provide a special case of
defines for Irix, but how can you know which codecs are
available on a platform during the configure process?
Perhaps running iconv -l and trying to decode the output?
iconv -l on GNU/linux shows a comma separated list, while
on Irix it shows pairs of available conversions, one on each
line...

Thanks for the direction about encodings/init.py, Walter.

Thanks guys, now I must find if pymalloc has changed and
occasionally dumps core in dictresize or listextend_internal...

Logged In: YES
user_id=21627

I have reverted this patch because of problems with

setup.py 1.159
__init__.py 1.17
iconv_codec.py delete
regrtest.py 1.137
test_iconv_codecs.py delete
NEWS 1.712
Setup.dist 1.39
_iconv_codec.c delete

Navigation Menu

Repository navigation

Universal Unicode Codec for POSIX iconv #37800

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Universal Unicode Codec for POSIX iconv #37800

Description

Activity

hyeshik commented on Jan 19, 2003

malemburg commented on Jan 19, 2003

hyeshik commented on Jan 20, 2003

loewis commented on Jan 21, 2003

hyeshik commented on Jan 22, 2003

loewis commented on Jan 22, 2003

loewis commented on Jan 26, 2003

hyeshik commented on Jan 26, 2003

doerwalter commented on Jan 30, 2003

doerwalter commented on Jan 30, 2003

tzot commented on Feb 4, 2003

doerwalter commented on Feb 4, 2003

tzot commented on Feb 4, 2003

loewis commented on Apr 3, 2003

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions