****************************

What's New In Python 3.4

****************************

.. Rules for maintenance:

* Anyone can add text to this document, but the maintainer reserves the

right to rewrite any additions. In particular, for obscure or esoteric

features, the maintainer may reduce any addition to a simple reference to

the new documentation rather than explaining the feature inline.

* While the maintainer will periodically go through Misc/NEWS

and add changes, it's best not to rely on this. We know from experience

that any changes that aren't in the What's New documentation around the

time of the original release will remain largely unknown to the community

for years, even if they're added later. We also know from experience that

other priorities can arise, and the maintainer will run out of time to do

notifications that at least give a hint about new features to

investigate.

* This is not a complete list of every single change; completeness

is the purpose of Misc/NEWS. The What's New should focus on changes that

are visible to Python *users* and that *require* a feature release (i.e.

most bug fixes should only be recorded in Misc/NEWS)

* PEPs should not be marked Final until they have an entry in What's New.

A placeholder entry that is just a section header and a link to the PEP

(e.g ":pep:`397` has been implemented") is acceptable. If a PEP has been

implemented and noted in What's New, don't forget to mark it as Final!

* If you want to draw your new text to the attention of the

maintainer, add 'XXX' to the beginning of the paragraph or

section.

* It's OK to add just a very brief note about a change. For

example: "The :ref:`~socket.transmogrify()` function was added to the

:mod:`socket` module." The maintainer will research the change and

write the necessary text (if appropriate). The advantage of doing this

is that even if no more descriptive text is ever added, readers will at

least have a notification that the new feature exists and a link to the

relevant documentation.

* You can comment out your additions if you like, but it's not

necessary (especially when a final release is some months away).

* Credit the author of a patch or bugfix. Just the name is

sufficient; the e-mail address isn't necessary.

* It's helpful to add the bug/patch number as a comment:

This saves the maintainer the effort of going through the Mercurial log

when researching a change.

* Cross referencing tip: :ref:`mod.attr` will display as ``mod.attr``,

while :ref:`~mod.attr` will display as ``attr``.

.. seealso::

=============================

Brevity is key.

New syntax features:

* :ref:`pip should always be available <whatsnew-pep-453>` (:pep:`453`).

* :ref:`Newly created file descriptors are non-inheritable <whatsnew-pep-446>`

* :ref:`improvements in the handling of codecs <codec-handling-improvements>`

that are not text encodings (multiple issues).

* The :mod:`marshal` format has been made :ref:`more compact and efficient

<whatsnew-marshal-3>` (:issue:`16475`).

New library modules:

* :mod:`asyncio`: :ref:`New provisional API for asynchronous IO

<whatsnew-asyncio>` (:pep:`3156`).

* :mod:`ensurepip`: :ref:`Bootstrapping the pip installer <whatsnew-ensurepip>`

(:pep:`453`).

* :mod:`enum`: :ref:`Support for enumeration types <whatsnew-enum>`

(:pep:`435`).

* :mod:`pathlib`: :ref:`Object-oriented filesystem paths <whatsnew-pathlib>`

(:pep:`428`).

* :mod:`selectors`: :ref:`High-level and efficient I/O multiplexing

<whatsnew-selectors>`, built upon the :mod:`select` module primitives (part

of :pep:`3156`).

* :mod:`statistics`: A basic :ref:`numerically stable statistics library

<whatsnew-statistics>` (:pep:`450`).

* :mod:`tracemalloc`: :ref:`Trace Python memory allocations

<whatsnew-tracemalloc>` (:pep:`454`).

:mod:`functools` (:pep:`443`).

* New :mod:`pickle` :ref:`protocol 4 <whatsnew-protocol-4>` (:pep:`3154`).

* :mod:`multiprocessing` now has :ref:`an option to avoid using os.fork

on Unix <whatsnew-multiprocessing-no-fork>` (:issue:`8713`).

* :mod:`email` has a new submodule, :mod:`~email.contentmanager`, and

a new :mod:`~email.message.Message` subclass

* The :mod:`inspect` and :mod:`pydoc` modules are now capable of

correct introspection of a much wider variety of callable objects,

which improves the output of the Python :func:`help` system.

Security improvements:

* :ref:`Secure and interchangeable hash algorithm <whatsnew-pep-456>`

(:pep:`456`).

* :ref:`Make newly created file descriptors non-inheritable <whatsnew-pep-446>`

(:pep:`446`) to avoid leaking file descriptors to child processes.

* New command line option for :ref:`isolated mode <whatsnew-isolated-mode>`,

(:issue:`16499`).

* :mod:`multiprocessing` now has :ref:`an option to avoid using os.fork

on Unix <whatsnew-multiprocessing-no-fork>`. *spawn* and *forkserver* are

more secure because they avoid sharing data with child processes.

* :mod:`multiprocessing` child processes on Windows no longer inherit

all of the parent's inheritable handles, only the necessary ones.

* A new :func:`hashlib.pbkdf2_hmac` function provides

the `PKCS#5 password-based key derivation function 2

* :ref:`TLSv1.1 and TLSv1.2 support <whatsnew-tls-11-12>` for :mod:`ssl`.

* :ref:`Retrieving certificates from the Windows system cert store support

<whatsnew34-win-cert-store>` for :mod:`ssl`.

* :ref:`Server-side SNI (Server Name Indication) support

<whatsnew34-sni>` for :mod:`ssl`.

* The :class:`ssl.SSLContext` class has a :ref:`lot of improvements

<whatsnew34-sslcontext>`.

* All modules in the standard library that support SSL now support server

certificate verification, including hostname matching

* :ref:`Configurable memory allocators <whatsnew-pep-445>` (:pep:`445`).

* :ref:`Argument Clinic <whatsnew-pep-436>` (:pep:`436`).

Please read on for a comprehensive list of user-facing changes, including many

other smaller improvements, CPython optimizations, deprecations, and potential

porting issues.

New Features

============

.. _whatsnew-pep-453:

Bootstrapping pip By Default

~~~~~~~~~~~~~~~~~~~~~~~~~~~~

installations and virtual environments. The version of ``pip`` included

with Python 3.4.0 is ``pip`` 1.5.4, and future 3.4.x maintenance releases

will update the bundled version to the latest version of ``pip`` that is

available at the time of creating the release candidate.

By default, the commands ``pipX`` and ``pipX.Y`` will be installed on all

platforms (where X.Y stands for the version of the Python installation),

along with the ``pip`` Python package and its dependencies. On Windows and

in virtual environments on all platforms, the unversioned ``pip`` command

will also be installed. On other platforms, the system wide unversioned

``pip`` command typically refers to the separately installed Python 2

version.

module make use of the :mod:`ensurepip` module to make ``pip`` readily

available in virtual environments. When using the command line utility,

``pip`` is installed by default, while when using the :mod:`venv` module

:ref:`venv-api` installation of ``pip`` must be requested explicitly.

For CPython :ref:`source builds on POSIX systems <building-python-on-unix>`,

the ``make install`` and ``make altinstall`` commands bootstrap ``pip`` by

default. This behaviour can be controlled through configure options, and

overridden through Makefile options.

On Windows and Mac OS X, the CPython installers now default to installing

``pip`` along with CPython itself (users may opt out of installing it

during the installation process). Window users will need to opt in to the

automatic ``PATH`` modifications to have ``pip`` available from the command

line by default, otherwise it can still be accessed through the Python

launcher for Windows as ``py -m pip``.

As :pep:`discussed in the PEP <0453#recommendations-for-downstream-distributors>`

platform packagers may choose not to install

these commands by default, as long as, when invoked, they provide clear and

simple directions on how to install them on that platform (usually using

the system package manager).

To avoid conflicts between parallel Python 2 and Python 3 installations,

only the versioned ``pip3`` and ``pip3.4`` commands are bootstrapped by

default when ``ensurepip`` is invoked directly - the ``--default-pip``

option is needed to also request the unversioned ``pip`` command.

``pyvenv`` and the Windows installer ensure that the unqualified ``pip``

command is made available in those environments, and ``pip`` can always be

invoked via the ``-m`` switch rather than directly to avoid ambiguity on

systems with multiple Python installations.

Documentation Changes

~~~~~~~~~~~~~~~~~~~~~

As part of this change, the :ref:`installing-index` and

:ref:`distributing-index` sections of the documentation have been

completely redesigned as short getting started and FAQ documents. Most

packaging documentation has now been moved out to the Python Packaging

Authority maintained `Python Packaging User Guide

projects.

However, as this migration is currently still incomplete, the legacy

versions of those guides remaining available as :ref:`install-index`

.. seealso::

PEP 446: Newly Created File Descriptors Are Non-Inheritable

-----------------------------------------------------------

<fd_inheritance>`. In general, this is the behavior an application will

want: when launching a new process, having currently open files also

open in the new process can lead to all sorts of hard to find bugs,

and potentially to security issues.

However, there are occasions when inheritance is desired. To support

these cases, the following new functions and methods are available:

* :func:`os.get_inheritable`, :func:`os.set_inheritable`

* :func:`os.get_handle_inheritable`, :func:`os.set_handle_inheritable`

* :meth:`socket.socket.get_inheritable`, :meth:`socket.socket.set_inheritable`

.. seealso::

.. _codec-handling-improvements:

Since it was first introduced, the :mod:`codecs` module has always been

intended to operate as a type-neutral dynamic encoding and decoding

system. However, its close coupling with the Python text model, especially

the type restricted convenience methods on the builtin :class:`str`,

:class:`bytes` and :class:`bytearray` types, has historically obscured that

fact.

As a key step in clarifying the situation, the :meth:`codecs.encode` and

:meth:`codecs.decode` convenience functions are now properly documented in

Python 2.7, 3.3 and 3.4. These functions have existed in the :mod:`codecs`

but were previously only discoverable through runtime introspection.

Unlike the convenience methods on :class:`str`, :class:`bytes` and

:class:`bytearray`, the :mod:`codecs` convenience functions support arbitrary

codecs in both Python 2 and Python 3, rather than being limited to Unicode text

encodings (in Python 3) or ``basestring`` <-> ``basestring`` conversions (in

Python 2).

In Python 3.4, the interpreter is able to identify the known non-text

encodings provided in the standard library and direct users towards these

general purpose convenience functions when appropriate::

Traceback (most recent call last):

File "<stdin>", line 1, in <module>

Traceback (most recent call last):

File "<stdin>", line 1, in <module>

>>> open("foo.txt", encoding="hex")

Traceback (most recent call last):

File "<stdin>", line 1, in <module>

LookupError: 'hex' is not a text encoding; use codecs.open() to handle arbitrary codecs

In a related change, whenever it is feasible without breaking backwards

compatibility, exceptions raised during encoding and decoding operations

name of the codec responsible for producing the error::

>>> import codecs

>>> codecs.decode(b"abcdefgh", "hex")

Traceback (most recent call last):

File "/usr/lib/python3.4/encodings/hex_codec.py", line 20, in hex_decode

return (binascii.a2b_hex(input), len(input))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

File "<stdin>", line 1, in <module>

Traceback (most recent call last):

File "/usr/lib/python3.4/encodings/bz2_codec.py", line 17, in bz2_encode

return (bz2.compress(input), len(input))

File "/usr/lib/python3.4/bz2.py", line 498, in compress

return comp.compress(data) + comp.flush()

TypeError: 'str' does not support the buffer interface

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

File "<stdin>", line 1, in <module>

TypeError: encoding with 'bz2' codec failed (TypeError: 'str' does not support the buffer interface)

Finally, as the examples above show, these improvements have permitted

the restoration of the convenience aliases for the non-Unicode codecs that

were themselves restored in Python 3.2. This means that encoding binary data

to and from its hexadecimal representation (for example) can now be written

as::

>>> from codecs import encode, decode

>>> encode(b"hello", "hex")

b'68656c6c6f'

>>> decode(b"68656c6c6f", "hex")

b'hello'

The binary and text transforms provided in the standard library are detailed

in :ref:`binary-transforms` and :ref:`text-transforms`.

PEP 451: A ModuleSpec Type for the Import System

:pep:`451` provides an encapsulation of the information about a module that the

import machinery will use to load it (that is, a module specification). This

helps simplify both the import implementation and several import-related APIs.

The change is also a stepping stone for `several future import-related

improvements`__.

The public-facing changes from the PEP are entirely backward-compatible.

Furthermore, they should be transparent to everyone but importer authors. Key

finder and loader methods have been deprecated, but they will continue working.

New importers should use the new methods described in the PEP. Existing

importers should be updated to implement the new methods. See the

:ref:`deprecated-3.4` section for a list of methods that should be replaced and

their replacements.

Some smaller changes made to the core Python language are:

* :func:`min` and :func:`max` now accept a *default* keyword-only argument that

can be used to specify the value they return if the iterable they are

evaluating has no elements. (Contributed by Julian Berman in

:issue:`18111`.)

* Module ``__file__`` attributes (and related values) should now always

contain absolute paths by default, with the sole exception of

``__main__.__file__`` when a script has been executed directly using

with the exception of the UTF-16 decoder (which accepts valid surrogate pairs)

and the UTF-16 encoder (which produces them while encoding non-BMP characters).

(Contributed by Victor Stinner, Kang-Hao (Kenny) Lu and Serhiy Storchaka in

:issue:`12892`.)

* New German EBCDIC :ref:`codec <standard-encodings>` ``cp273``. (Contributed

by Michael Bierenfeld and Andrew Kuchling in :issue:`1097797`.)

* New Ukrainian :ref:`codec <standard-encodings>` ``cp1125``. (Contributed by

Serhiy Storchaka in :issue:`19668`.)

* :class:`bytes`.join() and :class:`bytearray`.join() now accept arbitrary

buffer objects as arguments. (Contributed by Antoine Pitrou in

:issue:`15958`.)

* The :class:`int` constructor now accepts any object that has an ``__index__``

method for its *base* argument. (Contributed by Mark Dickinson in

:issue:`16772`.)

* Frame objects now have a :func:`~frame.clear` method that clears all

references to local variables from the frame. (Contributed by Antoine Pitrou

in :issue:`17934`.)

* :class:`memoryview` is now registered as a :class:`Sequence <collections.abc>`,

and supports the :func:`reversed` builtin. (Contributed by Nick Coghlan

and Claudiu Popa in :issue:`18690` and :issue:`19078`.)

* Signatures reported by :func:`help` have been modified and improved in

several cases as a result of the introduction of Argument Clinic and other

changes to the :mod:`inspect` and :mod:`pydoc` modules.

* :meth:`~object.__length_hint__` is now part of the formal language

specification (see :pep:`424`). (Contributed by Armin Ronacher in

:issue:`16148`.)

New Modules

===========

.. _whatsnew-asyncio:

asyncio

-------

The new :mod:`asyncio` module (defined in :pep:`3156`) provides a standard

pluggable event loop model for Python, providing solid asynchronous IO

support in the standard library, and making it easier for other event loop

implementations to interoperate with the standard library and each other.

For Python 3.4, this module is considered a :term:`provisional API`.

.. seealso::

PEP written and implementation led by Guido van Rossum.

ensurepip

---------

The new :mod:`ensurepip` module is the primary infrastructure for the

:pep:`453` implementation. In the normal course of events end users will not

need to interact with this module, but it can be used to manually bootstrap

``pip`` if the automated bootstrapping into an installation or virtual

environment was declined.

:mod:`ensurepip` includes a bundled copy of ``pip``, up-to-date as of the first

release candidate of the release of CPython with which it ships (this applies

to both maintenance releases and feature releases). ``ensurepip`` does not