Skip to content

Commit 4bd6574

Browse files
warsawwarsaw
committed
PEP 3147
1 parent 66506cc commit 4bd6574

39 files changed

Lines changed: 1204 additions & 288 deletions

‎.bzrignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,3 +33,4 @@ Parser/pgen
3333
Lib/test/data/*
3434
Lib/lib2to3/Grammar*.pickle
3535
Lib/lib2to3/PatternGrammar*.pickle
36+
__pycache__

‎.hgignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,3 +54,4 @@ PCbuild/*.o
5454
PCbuild/*.ncb
5555
PCbuild/*.bsc
5656
PCbuild/Win32-temp-*
57+
__pycache__

‎Doc/c-api/import.rst‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -124,12 +124,24 @@ Importing Modules
124124
If *name* points to a dotted name of the form ``package.module``, any package
125125
structures not already created will still not be created.
126126

127+
See also :func:`PyImport_ExecCodeModuleEx` and
128+
:func:`PyImport_ExecCodeModuleWithPathnames`.
129+
127130

128131
.. cfunction:: PyObject* PyImport_ExecCodeModuleEx(char *name, PyObject *co, char *pathname)
129132

130133
Like :cfunc:`PyImport_ExecCodeModule`, but the :attr:`__file__` attribute of
131134
the module object is set to *pathname* if it is non-``NULL``.
132135

136+
See also :func:`PyImport_ExecCodeModuleWithPathnames`.
137+
138+
139+
.. cfunction:: PyObject* PyImport_ExecCodeModuleWithPathnames(char *name, PyObject *co, char *pathname, char *cpathname)
140+
141+
Like :cfunc:`PyImport_ExecCodeModuleEx`, but the :attr:`__cached__`
142+
attribute of the module object is set to *cpathname* if it is
143+
non-``NULL``. Of the three functions, this is the preferred one to use.
144+
133145

134146
.. cfunction:: long PyImport_GetMagicNumber()
135147

@@ -138,6 +150,11 @@ Importing Modules
138150
of the bytecode file, in little-endian byte order.
139151

140152

153+
.. cfunction:: const char * PyImport_GetMagicTag()
154+
155+
Return the magic tag string for :pep:`3147` format Python bytecode file
156+
names.
157+
141158
.. cfunction:: PyObject* PyImport_GetModuleDict()
142159

143160
Return the dictionary used for the module administration (a.k.a.

‎Doc/library/compileall.rst‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -17,9 +17,11 @@ line. If no arguments are given, the invocation is equivalent to ``-l
1717
sys.path``. Printing lists of the files compiled can be disabled with the
1818
:option:`-q` flag. In addition, the :option:`-x` option takes a regular
1919
expression argument. All files that match the expression will be skipped.
20+
The :option:`-b` flag may be given to write legacy ``.pyc`` file path names,
21+
otherwise :pep:`3147` style byte-compiled path names are written.
2022

2123

22-
.. function:: compile_dir(dir, maxlevels=10, ddir=None, force=False, rx=None, quiet=False)
24+
.. function:: compile_dir(dir, maxlevels=10, ddir=None, force=False, rx=None, quiet=False, legacy=False)
2325

2426
Recursively descend the directory tree named by *dir*, compiling all :file:`.py`
2527
files along the way. The *maxlevels* parameter is used to limit the depth of
@@ -34,12 +36,16 @@ expression argument. All files that match the expression will be skipped.
3436
If *quiet* is true, nothing is printed to the standard output in normal
3537
operation.
3638

39+
If *legacy* is true, old-style ``.pyc`` file path names are written,
40+
otherwise (the default), :pep:`3147` style path names are written.
3741

38-
.. function:: compile_path(skip_curdir=True, maxlevels=0, force=False)
42+
43+
.. function:: compile_path(skip_curdir=True, maxlevels=0, force=False, legacy=False)
3944

4045
Byte-compile all the :file:`.py` files found along ``sys.path``. If
4146
*skip_curdir* is true (the default), the current directory is not included in
42-
the search. The *maxlevels* and *force* parameters default to ``0`` and are
47+
the search. The *maxlevels* parameter defaults to ``0``, and the *force*
48+
and *legacy* parameters default to ``False``. All are
4349
passed to the :func:`compile_dir` function.
4450

4551
To force a recompile of all the :file:`.py` files in the :file:`Lib/`

‎Doc/library/imp.rst‎

Lines changed: 35 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -204,8 +204,41 @@ This module provides an interface to the mechanisms used to implement the
204204
function does nothing.
205205

206206

207-
The following constants with integer values, defined in this module, are used to
208-
indicate the search result of :func:`find_module`.
207+
The following functions and data provide conveniences for handling :pep:`3147`
208+
byte-compiled file paths.
209+
210+
.. versionadded:: 3.2
211+
212+
.. function:: cache_from_source(path, debug_override=None)
213+
214+
Return the PEP 3147 path to the byte-compiled file associated with the
215+
source *path*. For example, if *path* is ``/foo/bar/baz.py`` the return
216+
value would be ``/foo/bar/__pycache__/baz.cpython-32.pyc`` for Python 3.2.
217+
The ``cpython-32`` string comes from the current magic tag (see
218+
:func:`get_tag`). The returned path will end in ``.pyc`` when
219+
``__debug__`` is True or ``.pyo`` for an optimized Python
220+
(i.e. ``__debug__`` is False). By passing in True or False for
221+
*debug_override* you can override the system's value for ``__debug__`` for
222+
extension selection.
223+
224+
*path* need not exist.
225+
226+
.. function:: source_from_cache(path)
227+
228+
Given the *path* to a PEP 3147 file name, return the associated source code
229+
file path. For example, if *path* is
230+
``/foo/bar/__pycache__/baz.cpython-32.pyc`` the returned path would be
231+
``/foo/bar/baz.py``. *path* need not exist, however if it does not conform
232+
to PEP 3147 format, a ``ValueError`` is raised.
233+
234+
.. function:: get_tag()
235+
236+
Return the PEP 3147 magic tag string matching this version of Python's
237+
magic number, as returned by :func:`get_magic`.
238+
239+
240+
The following constants with integer values, defined in this module, are used
241+
to indicate the search result of :func:`find_module`.
209242

210243

211244
.. data:: PY_SOURCE

‎Doc/library/py_compile.rst‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -26,12 +26,16 @@ byte-code cache files in the directory containing the source code.
2626

2727
Compile a source file to byte-code and write out the byte-code cache file. The
2828
source code is loaded from the file name *file*. The byte-code is written to
29-
*cfile*, which defaults to *file* ``+`` ``'c'`` (``'o'`` if optimization is
30-
enabled in the current interpreter). If *dfile* is specified, it is used as the
29+
*cfile*, which defaults to the :PEP:`3147` path, ending in ``.pyc``
30+
(``'.pyo`` if optimization is enabled in the current interpreter). For
31+
example, if *file* is ``/foo/bar/baz.py`` *cfile* will default to
32+
``/foo/bar/__pycache__/baz.cpython-32.pyc`` for Python 3.2. If *dfile* is specified, it is used as the
3133
name of the source file in error messages instead of *file*. If *doraise* is
3234
true, a :exc:`PyCompileError` is raised when an error is encountered while
3335
compiling *file*. If *doraise* is false (the default), an error string is
34-
written to ``sys.stderr``, but no exception is raised.
36+
written to ``sys.stderr``, but no exception is raised. This function
37+
returns the path to byte-compiled file, i.e. whatever *cfile* value was
38+
used.
3539

3640

3741
.. function:: main(args=None)

‎Doc/library/runpy.rst‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,8 @@ The :mod:`runpy` module provides two functions:
3232
below are defined in the supplied dictionary, those definitions are
3333
overridden by :func:`run_module`.
3434

35-
The special global variables ``__name__``, ``__file__``, ``__loader__``
35+
The special global variables ``__name__``, ``__file__``, ``__cached__``,
36+
``__loader__``
3637
and ``__package__`` are set in the globals dictionary before the module
3738
code is executed (Note that this is a minimal set of variables - other
3839
variables may be set implicitly as an interpreter implementation detail).
@@ -45,6 +46,8 @@ The :mod:`runpy` module provides two functions:
4546
loader does not make filename information available, this variable is set
4647
to :const:`None`.
4748

49+
``__cached__`` will be set to ``None``.
50+
4851
``__loader__`` is set to the PEP 302 module loader used to retrieve the
4952
code for the module (This loader may be a wrapper around the standard
5053
import mechanism).

‎Include/import.h‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,9 +8,12 @@ extern "C" {
88
#endif
99

1010
PyAPI_FUNC(long) PyImport_GetMagicNumber(void);
11+
PyAPI_FUNC(const char *) PyImport_GetMagicTag(void);
1112
PyAPI_FUNC(PyObject *) PyImport_ExecCodeModule(char *name, PyObject *co);
1213
PyAPI_FUNC(PyObject *) PyImport_ExecCodeModuleEx(
1314
char *name, PyObject *co, char *pathname);
15+
PyAPI_FUNC(PyObject *) PyImport_ExecCodeModuleWithPathnames(
16+
char *name, PyObject *co, char *pathname, char *cpathname);
1417
PyAPI_FUNC(PyObject *) PyImport_GetModuleDict(void);
1518
PyAPI_FUNC(PyObject *) PyImport_AddModule(const char *name);
1619
PyAPI_FUNC(PyObject *) PyImport_ImportModule(const char *name);

‎Lib/compileall.py‎

Lines changed: 48 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212
1313
"""
1414
import os
15+
import errno
1516
import sys
1617
import py_compile
1718
import struct
@@ -20,7 +21,7 @@
2021
__all__ = ["compile_dir","compile_file","compile_path"]
2122

2223
def compile_dir(dir, maxlevels=10, ddir=None,
23-
force=0, rx=None, quiet=0):
24+
force=False, rx=None, quiet=False, legacy=False):
2425
"""Byte-compile all modules in the given directory tree.
2526
2627
Arguments (only dir is required):
@@ -29,8 +30,9 @@ def compile_dir(dir, maxlevels=10, ddir=None,
2930
maxlevels: maximum recursion level (default 10)
3031
ddir: if given, purported directory name (this is the
3132
directory name that will show up in error messages)
32-
force: if 1, force compilation, even if timestamps are up-to-date
33-
quiet: if 1, be quiet during compilation
33+
force: if True, force compilation, even if timestamps are up-to-date
34+
quiet: if True, be quiet during compilation
35+
legacy: if True, produce legacy pyc paths instead of PEP 3147 paths
3436
3537
"""
3638
if not quiet:
@@ -49,24 +51,26 @@ def compile_dir(dir, maxlevels=10, ddir=None,
4951
else:
5052
dfile = None
5153
if not os.path.isdir(fullname):
52-
if not compile_file(fullname, ddir, force, rx, quiet):
54+
if not compile_file(fullname, ddir, force, rx, quiet, legacy):
5355
success = 0
5456
elif maxlevels > 0 and \
5557
name != os.curdir and name != os.pardir and \
5658
os.path.isdir(fullname) and \
5759
not os.path.islink(fullname):
5860
if not compile_dir(fullname, maxlevels - 1, dfile, force, rx,
59-
quiet):
61+
quiet, legacy):
6062
success = 0
6163
return success
6264

63-
def compile_file(fullname, ddir=None, force=0, rx=None, quiet=0):
65+
def compile_file(fullname, ddir=None, force=0, rx=None, quiet=False,
66+
legacy=False):
6467
"""Byte-compile file.
65-
file: the file to byte-compile
68+
fullname: the file to byte-compile
6669
ddir: if given, purported directory name (this is the
6770
directory name that will show up in error messages)
68-
force: if 1, force compilation, even if timestamps are up-to-date
69-
quiet: if 1, be quiet during compilation
71+
force: if True, force compilation, even if timestamps are up-to-date
72+
quiet: if True, be quiet during compilation
73+
legacy: if True, produce legacy pyc paths instead of PEP 3147 paths
7074
7175
"""
7276
success = 1
@@ -80,13 +84,22 @@ def compile_file(fullname, ddir=None, force=0, rx=None, quiet=0):
8084
if mo:
8185
return success
8286
if os.path.isfile(fullname):
87+
if legacy:
88+
cfile = fullname + ('c' if __debug__ else 'o')
89+
else:
90+
cfile = imp.cache_from_source(fullname)
91+
cache_dir = os.path.dirname(cfile)
92+
try:
93+
os.mkdir(cache_dir)
94+
except OSError as error:
95+
if error.errno != errno.EEXIST:
96+
raise
8397
head, tail = name[:-3], name[-3:]
8498
if tail == '.py':
8599
if not force:
86100
try:
87101
mtime = int(os.stat(fullname).st_mtime)
88102
expect = struct.pack('<4sl', imp.get_magic(), mtime)
89-
cfile = fullname + (__debug__ and 'c' or 'o')
90103
with open(cfile, 'rb') as chandle:
91104
actual = chandle.read(8)
92105
if expect == actual:
@@ -96,14 +109,15 @@ def compile_file(fullname, ddir=None, force=0, rx=None, quiet=0):
96109
if not quiet:
97110
print('Compiling', fullname, '...')
98111
try:
99-
ok = py_compile.compile(fullname, None, dfile, True)
112+
ok = py_compile.compile(fullname, cfile, dfile, True)
100113
except py_compile.PyCompileError as err:
101114
if quiet:
102115
print('*** Error compiling', fullname, '...')
103116
else:
104117
print('*** ', end='')
105118
# escape non-printable characters in msg
106-
msg = err.msg.encode(sys.stdout.encoding, errors='backslashreplace')
119+
msg = err.msg.encode(sys.stdout.encoding,
120+
errors='backslashreplace')
107121
msg = msg.decode(sys.stdout.encoding)
108122
print(msg)
109123
success = 0
@@ -119,15 +133,17 @@ def compile_file(fullname, ddir=None, force=0, rx=None, quiet=0):
119133
success = 0
120134
return success
121135

122-
def compile_path(skip_curdir=1, maxlevels=0, force=0, quiet=0):
136+
def compile_path(skip_curdir=1, maxlevels=0, force=False, quiet=False,
137+
legacy=False):
123138
"""Byte-compile all module on sys.path.
124139
125140
Arguments (all optional):
126141
127142
skip_curdir: if true, skip current directory (default true)
128143
maxlevels: max recursion level (default 0)
129-
force: as for compile_dir() (default 0)
130-
quiet: as for compile_dir() (default 0)
144+
force: as for compile_dir() (default False)
145+
quiet: as for compile_dir() (default False)
146+
legacy: as for compile_dir() (default False)
131147
132148
"""
133149
success = 1
@@ -136,7 +152,8 @@ def compile_path(skip_curdir=1, maxlevels=0, force=0, quiet=0):
136152
print('Skipping current directory')
137153
else:
138154
success = success and compile_dir(dir, maxlevels, None,
139-
force, quiet=quiet)
155+
force, quiet=quiet,
156+
legacy=legacy)
140157
return success
141158

142159
def expand_args(args, flist):
@@ -162,10 +179,10 @@ def main():
162179
"""Script main program."""
163180
import getopt
164181
try:
165-
opts, args = getopt.getopt(sys.argv[1:], 'lfqd:x:i:')
182+
opts, args = getopt.getopt(sys.argv[1:], 'lfqd:x:i:b')
166183
except getopt.error as msg:
167184
print(msg)
168-
print("usage: python compileall.py [-l] [-f] [-q] [-d destdir] " \
185+
print("usage: python compileall.py [-l] [-f] [-q] [-d destdir] "
169186
"[-x regexp] [-i list] [directory|file ...]")
170187
print("-l: don't recurse down")
171188
print("-f: force rebuild even if timestamps are up-to-date")
@@ -174,23 +191,27 @@ def main():
174191
print(" if no directory arguments, -l sys.path is assumed")
175192
print("-x regexp: skip files matching the regular expression regexp")
176193
print(" the regexp is searched for in the full path of the file")
177-
print("-i list: expand list with its content (file and directory names)")
194+
print("-i list: expand list with its content "
195+
"(file and directory names)")
196+
print("-b: Produce legacy byte-compile file paths")
178197
sys.exit(2)
179198
maxlevels = 10
180199
ddir = None
181-
force = 0
182-
quiet = 0
200+
force = False
201+
quiet = False
183202
rx = None
184203
flist = None
204+
legacy = False
185205
for o, a in opts:
186206
if o == '-l': maxlevels = 0
187207
if o == '-d': ddir = a
188-
if o == '-f': force = 1
189-
if o == '-q': quiet = 1
208+
if o == '-f': force = True
209+
if o == '-q': quiet = True
190210
if o == '-x':
191211
import re
192212
rx = re.compile(a)
193213
if o == '-i': flist = a
214+
if o == '-b': legacy = True
194215
if ddir:
195216
if len(args) != 1 and not os.path.isdir(args[0]):
196217
print("-d destdir require exactly one directory argument")
@@ -207,13 +228,14 @@ def main():
207228
for arg in args:
208229
if os.path.isdir(arg):
209230
if not compile_dir(arg, maxlevels, ddir,
210-
force, rx, quiet):
231+
force, rx, quiet, legacy):
211232
success = 0
212233
else:
213-
if not compile_file(arg, ddir, force, rx, quiet):
234+
if not compile_file(arg, ddir, force, rx,
235+
quiet, legacy):
214236
success = 0
215237
else:
216-
success = compile_path()
238+
success = compile_path(legacy=legacy)
217239
except KeyboardInterrupt:
218240
print("\n[interrupt]")
219241
success = 0

0 commit comments

Comments
 (0)