Skip to content

Commit b03c2f7

Browse files
ncoghlanncoghlan
committed
Close #17828: better handling of codec errors
- output type errors now redirect users to the type-neutral convenience functions in the codecs module - stateless errors that occur during encoding and decoding will now be automatically wrapped in exceptions that give the name of the codec involved
1 parent 954ac09 commit b03c2f7

7 files changed

Lines changed: 414 additions & 46 deletions

File tree

‎Doc/whatsnew/3.4.rst‎

Lines changed: 65 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -102,6 +102,7 @@ New expected features for Python implementations:
102102
* :ref:`PEP 446: Make newly created file descriptors non-inheritable <pep-446>`.
103103
* command line option for :ref:`isolated mode <using-on-misc-options>`,
104104
(:issue:`16499`).
105+
* improvements to handling of non-Unicode codecs
105106

106107
Significantly Improved Library Modules:
107108

@@ -170,6 +171,70 @@ PEP 446: Make newly created file descriptors non-inheritable
170171
PEP written and implemented by Victor Stinner.
171172

172173

174+
Improvements to handling of non-Unicode codecs
175+
==============================================
176+
177+
Since it was first introduced, the :mod:`codecs` module has always been
178+
intended to operate as a type-neutral dynamic encoding and decoding
179+
system. However, its close coupling with the Python text model, especially
180+
the type restricted convenience methods on the builtin :class:`str`,
181+
:class:`bytes` and :class:`bytearray` types, has historically obscured that
182+
fact.
183+
184+
As a key step in clarifying the situation, the :meth:`codecs.encode` and
185+
:meth:`codecs.decode` convenience functions are now properly documented in
186+
Python 2.7, 3.3 and 3.4. These functions have existed in the :mod:`codecs`
187+
module and have been covered by the regression test suite since Python 2.4,
188+
but were previously only discoverable through runtime introspection.
189+
190+
Unlike the convenience methods on :class:`str`, :class:`bytes` and
191+
:class:`bytearray`, these convenience functions support arbitrary codecs
192+
in both Python 2 and Python 3, rather than being limited to Unicode text
193+
encodings (in Python 3) or ``basestring`` <-> ``basestring`` conversions
194+
(in Python 2).
195+
196+
In Python 3.4, the errors raised by the convenience methods when a codec
197+
produces the incorrect output type have also been updated to direct users
198+
towards these general purpose convenience functions::
199+
200+
>>> import codecs
201+
202+
>>> codecs.encode(b"hello", "bz2_codec").decode("bz2_codec")
203+
Traceback (most recent call last):
204+
File "<stdin>", line 1, in <module>
205+
TypeError: 'bz2_codec' decoder returned 'bytes' instead of 'str'; use codecs.decode() to decode to arbitrary types
206+
207+
>>> "hello".encode("rot_13")
208+
Traceback (most recent call last):
209+
File "<stdin>", line 1, in <module>
210+
TypeError: 'rot_13' encoder returned 'str' instead of 'bytes'; use codecs.encode() to encode to arbitrary types
211+
212+
In a related change, whenever it is feasible without breaking backwards
213+
compatibility, exceptions raised during encoding and decoding operations
214+
will be wrapped in a chained exception of the same type that mentions the
215+
name of the codec responsible for producing the error::
216+
217+
>>> b"hello".decode("uu_codec")
218+
ValueError: Missing "begin" line in input data
219+
220+
The above exception was the direct cause of the following exception:
221+
222+
Traceback (most recent call last):
223+
File "<stdin>", line 1, in <module>
224+
ValueError: decoding with 'uu_codec' codec failed (ValueError: Missing "begin" line in input data)
225+
226+
>>> "hello".encode("bz2_codec")
227+
TypeError: 'str' does not support the buffer interface
228+
229+
The above exception was the direct cause of the following exception:
230+
231+
Traceback (most recent call last):
232+
File "<stdin>", line 1, in <module>
233+
TypeError: encoding with 'bz2_codec' codec failed (TypeError: 'str' does not support the buffer interface)
234+
235+
(Contributed by Nick Coghlan in :issue:`17827` and :issue:`17828`)
236+
237+
173238
Other Language Changes
174239
======================
175240

@@ -262,19 +327,6 @@ audioop
262327
Added support for 24-bit samples (:issue:`12866`).
263328

264329

265-
codecs
266-
------
267-
268-
The :meth:`codecs.encode` and :meth:`codecs.decode` convenience functions are
269-
now properly documented. These functions have existed in the :mod:`codecs`
270-
module since ~2004, but were previously only discoverable through runtime
271-
introspection.
272-
273-
Unlike the convenience methods on :class:`str`, :class:`bytes` and
274-
:class:`bytearray`, these convenience functions support arbitrary codecs,
275-
rather than being limited to Unicode text encodings.
276-
277-
278330
colorsys
279331
--------
280332

‎Include/pyerrors.h‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -285,6 +285,28 @@ PyAPI_FUNC(PyObject *) PyErr_NewExceptionWithDoc(
285285
const char *name, const char *doc, PyObject *base, PyObject *dict);
286286
PyAPI_FUNC(void) PyErr_WriteUnraisable(PyObject *);
287287

288+
/* In exceptions.c */
289+
#ifndef Py_LIMITED_API
290+
/* Helper that attempts to replace the current exception with one of the
291+
* same type but with a prefix added to the exception text. The resulting
292+
* exception description looks like:
293+
*
294+
* prefix (exc_type: original_exc_str)
295+
*
296+
* Only some exceptions can be safely replaced. If the function determines
297+
* it isn't safe to perform the replacement, it will leave the original
298+
* unmodified exception in place.
299+
*
300+
* Returns a borrowed reference to the new exception (if any), NULL if the
301+
* existing exception was left in place.
302+
*/
303+
PyAPI_FUNC(PyObject *) _PyErr_TrySetFromCause(
304+
const char *prefix_format, /* ASCII-encoded string */
305+
...
306+
);
307+
#endif
308+
309+
288310
/* In sigcheck.c or signalmodule.c */
289311
PyAPI_FUNC(int) PyErr_CheckSignals(void);
290312
PyAPI_FUNC(void) PyErr_SetInterrupt(void);

‎Lib/test/test_codecs.py‎

Lines changed: 169 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
import _testcapi
22
import codecs
3+
import contextlib
34
import io
45
import locale
56
import sys
@@ -2292,46 +2293,190 @@ class TransformCodecTest(unittest.TestCase):
22922293
def test_basics(self):
22932294
binput = bytes(range(256))
22942295
for encoding in bytes_transform_encodings:
2295-
# generic codecs interface
2296-
(o, size) = codecs.getencoder(encoding)(binput)
2297-
self.assertEqual(size, len(binput))
2298-
(i, size) = codecs.getdecoder(encoding)(o)
2299-
self.assertEqual(size, len(o))
2300-
self.assertEqual(i, binput)
2296+
with self.subTest(encoding=encoding):
2297+
# generic codecs interface
2298+
(o, size) = codecs.getencoder(encoding)(binput)
2299+
self.assertEqual(size, len(binput))
2300+
(i, size) = codecs.getdecoder(encoding)(o)
2301+
self.assertEqual(size, len(o))
2302+
self.assertEqual(i, binput)
23012303

23022304
def test_read(self):
23032305
for encoding in bytes_transform_encodings:
2304-
sin = codecs.encode(b"\x80", encoding)
2305-
reader = codecs.getreader(encoding)(io.BytesIO(sin))
2306-
sout = reader.read()
2307-
self.assertEqual(sout, b"\x80")
2306+
with self.subTest(encoding=encoding):
2307+
sin = codecs.encode(b"\x80", encoding)
2308+
reader = codecs.getreader(encoding)(io.BytesIO(sin))
2309+
sout = reader.read()
2310+
self.assertEqual(sout, b"\x80")
23082311

23092312
def test_readline(self):
23102313
for encoding in bytes_transform_encodings:
23112314
if encoding in ['uu_codec', 'zlib_codec']:
23122315
continue
2313-
sin = codecs.encode(b"\x80", encoding)
2314-
reader = codecs.getreader(encoding)(io.BytesIO(sin))
2315-
sout = reader.readline()
2316-
self.assertEqual(sout, b"\x80")
2316+
with self.subTest(encoding=encoding):
2317+
sin = codecs.encode(b"\x80", encoding)
2318+
reader = codecs.getreader(encoding)(io.BytesIO(sin))
2319+
sout = reader.readline()
2320+
self.assertEqual(sout, b"\x80")
23172321

23182322
def test_buffer_api_usage(self):
23192323
# We check all the transform codecs accept memoryview input
23202324
# for encoding and decoding
23212325
# and also that they roundtrip correctly
23222326
original = b"12345\x80"
23232327
for encoding in bytes_transform_encodings:
2324-
data = original
2325-
view = memoryview(data)
2326-
data = codecs.encode(data, encoding)
2327-
view_encoded = codecs.encode(view, encoding)
2328-
self.assertEqual(view_encoded, data)
2329-
view = memoryview(data)
2330-
data = codecs.decode(data, encoding)
2331-
self.assertEqual(data, original)
2332-
view_decoded = codecs.decode(view, encoding)
2333-
self.assertEqual(view_decoded, data)
2328+
with self.subTest(encoding=encoding):
2329+
data = original
2330+
view = memoryview(data)
2331+
data = codecs.encode(data, encoding)
2332+
view_encoded = codecs.encode(view, encoding)
2333+
self.assertEqual(view_encoded, data)
2334+
view = memoryview(data)
2335+
data = codecs.decode(data, encoding)
2336+
self.assertEqual(data, original)
2337+
view_decoded = codecs.decode(view, encoding)
2338+
self.assertEqual(view_decoded, data)
2339+
2340+
def test_type_error_for_text_input(self):
2341+
# Check binary -> binary codecs give a good error for str input
2342+
bad_input = "bad input type"
2343+
for encoding in bytes_transform_encodings:
2344+
with self.subTest(encoding=encoding):
2345+
msg = "^encoding with '{}' codec failed".format(encoding)
2346+
with self.assertRaisesRegex(TypeError, msg) as failure:
2347+
bad_input.encode(encoding)
2348+
self.assertTrue(isinstance(failure.exception.__cause__,
2349+
TypeError))
2350+
2351+
def test_type_error_for_binary_input(self):
2352+
# Check str -> str codec gives a good error for binary input
2353+
for bad_input in (b"immutable", bytearray(b"mutable")):
2354+
with self.subTest(bad_input=bad_input):
2355+
msg = "^decoding with 'rot_13' codec failed"
2356+
with self.assertRaisesRegex(AttributeError, msg) as failure:
2357+
bad_input.decode("rot_13")
2358+
self.assertTrue(isinstance(failure.exception.__cause__,
2359+
AttributeError))
2360+
2361+
def test_bad_decoding_output_type(self):
2362+
# Check bytes.decode and bytearray.decode give a good error
2363+
# message for binary -> binary codecs
2364+
data = b"encode first to ensure we meet any format restrictions"
2365+
for encoding in bytes_transform_encodings:
2366+
with self.subTest(encoding=encoding):
2367+
encoded_data = codecs.encode(data, encoding)
2368+
fmt = ("'{}' decoder returned 'bytes' instead of 'str'; "
2369+
"use codecs.decode\(\) to decode to arbitrary types")
2370+
msg = fmt.format(encoding)
2371+
with self.assertRaisesRegex(TypeError, msg):
2372+
encoded_data.decode(encoding)
2373+
with self.assertRaisesRegex(TypeError, msg):
2374+
bytearray(encoded_data).decode(encoding)
2375+
2376+
def test_bad_encoding_output_type(self):
2377+
# Check str.encode gives a good error message for str -> str codecs
2378+
msg = ("'rot_13' encoder returned 'str' instead of 'bytes'; "
2379+
"use codecs.encode\(\) to encode to arbitrary types")
2380+
with self.assertRaisesRegex(TypeError, msg):
2381+
"just an example message".encode("rot_13")
2382+
2383+
2384+
# The codec system tries to wrap exceptions in order to ensure the error
2385+
# mentions the operation being performed and the codec involved. We
2386+
# currently *only* want this to happen for relatively stateless
2387+
# exceptions, where the only significant information they contain is their
2388+
# type and a single str argument.
2389+
class ExceptionChainingTest(unittest.TestCase):
23342390

2391+
def setUp(self):
2392+
# There's no way to unregister a codec search function, so we just
2393+
# ensure we render this one fairly harmless after the test
2394+
# case finishes by using the test case repr as the codec name
2395+
# The codecs module normalizes codec names, although this doesn't
2396+
# appear to be formally documented...
2397+
self.codec_name = repr(self).lower().replace(" ", "-")
2398+
self.codec_info = None
2399+
codecs.register(self.get_codec)
2400+
2401+
def get_codec(self, codec_name):
2402+
if codec_name != self.codec_name:
2403+
return None
2404+
return self.codec_info
2405+
2406+
def set_codec(self, obj_to_raise):
2407+
def raise_obj(*args, **kwds):
2408+
raise obj_to_raise
2409+
self.codec_info = codecs.CodecInfo(raise_obj, raise_obj,
2410+
name=self.codec_name)
2411+
2412+
@contextlib.contextmanager
2413+
def assertWrapped(self, operation, exc_type, msg):
2414+
full_msg = "{} with '{}' codec failed \({}: {}\)".format(
2415+
operation, self.codec_name, exc_type.__name__, msg)
2416+
with self.assertRaisesRegex(exc_type, full_msg) as caught:
2417+
yield caught
2418+
2419+
def check_wrapped(self, obj_to_raise, msg):
2420+
self.set_codec(obj_to_raise)
2421+
with self.assertWrapped("encoding", RuntimeError, msg):
2422+
"str_input".encode(self.codec_name)
2423+
with self.assertWrapped("encoding", RuntimeError, msg):
2424+
codecs.encode("str_input", self.codec_name)
2425+
with self.assertWrapped("decoding", RuntimeError, msg):
2426+
b"bytes input".decode(self.codec_name)
2427+
with self.assertWrapped("decoding", RuntimeError, msg):
2428+
codecs.decode(b"bytes input", self.codec_name)
2429+
2430+
def test_raise_by_type(self):
2431+
self.check_wrapped(RuntimeError, "")
2432+
2433+
def test_raise_by_value(self):
2434+
msg = "This should be wrapped"
2435+
self.check_wrapped(RuntimeError(msg), msg)
2436+
2437+
@contextlib.contextmanager
2438+
def assertNotWrapped(self, operation, exc_type, msg):
2439+
with self.assertRaisesRegex(exc_type, msg) as caught:
2440+
yield caught
2441+
actual_msg = str(caught.exception)
2442+
self.assertNotIn(operation, actual_msg)
2443+
self.assertNotIn(self.codec_name, actual_msg)
2444+
2445+
def check_not_wrapped(self, obj_to_raise, msg):
2446+
self.set_codec(obj_to_raise)
2447+
with self.assertNotWrapped("encoding", RuntimeError, msg):
2448+
"str input".encode(self.codec_name)
2449+
with self.assertNotWrapped("encoding", RuntimeError, msg):
2450+
codecs.encode("str input", self.codec_name)
2451+
with self.assertNotWrapped("decoding", RuntimeError, msg):
2452+
b"bytes input".decode(self.codec_name)
2453+
with self.assertNotWrapped("decoding", RuntimeError, msg):
2454+
codecs.decode(b"bytes input", self.codec_name)
2455+
2456+
def test_init_override_is_not_wrapped(self):
2457+
class CustomInit(RuntimeError):
2458+
def __init__(self):
2459+
pass
2460+
self.check_not_wrapped(CustomInit, "")
2461+
2462+
def test_new_override_is_not_wrapped(self):
2463+
class CustomNew(RuntimeError):
2464+
def __new__(cls):
2465+
return super().__new__(cls)
2466+
self.check_not_wrapped(CustomNew, "")
2467+
2468+
def test_instance_attribute_is_not_wrapped(self):
2469+
msg = "This should NOT be wrapped"
2470+
exc = RuntimeError(msg)
2471+
exc.attr = 1
2472+
self.check_not_wrapped(exc, msg)
2473+
2474+
def test_non_str_arg_is_not_wrapped(self):
2475+
self.check_not_wrapped(RuntimeError(1), "1")
2476+
2477+
def test_multiple_args_is_not_wrapped(self):
2478+
msg = "\('a', 'b', 'c'\)"
2479+
self.check_not_wrapped(RuntimeError('a', 'b', 'c'), msg)
23352480

23362481

23372482
@unittest.skipUnless(sys.platform == 'win32',

‎Misc/NEWS‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,15 @@ Projected release date: 2013-11-24
1010
Core and Builtins
1111
-----------------
1212

13+
- Issue #17828: Output type errors in str.encode(), bytes.decode() and
14+
bytearray.decode() now direct users to codecs.encode() or codecs.decode()
15+
as appropriate.
16+
17+
- Issue #17828: The interpreter now attempts to chain errors that occur in
18+
codec processing with a replacement exception of the same type that
19+
includes the codec name in the error message. It ensures it only does this
20+
when the creation of the replacement exception won't lose any information.
21+
1322
- Issue #19466: Clear the frames of daemon threads earlier during the
1423
Python shutdown to call objects destructors. So "unclosed file" resource
1524
warnings are now corretly emitted for daemon threads.

0 commit comments

Comments
 (0)