Skip to content

Commit 495da29

Browse files
gvanrossummiss-islington
gvanrossum
authored and
miss-islington
committed
bpo-35975: Support parsing earlier minor versions of Python 3 (GH-12086)
This adds a `feature_version` flag to `ast.parse()` (documented) and `compile()` (hidden) that allow tweaking the parser to support older versions of the grammar. In particular if `feature_version` is 5 or 6, the hacks for the `async` and `await` keyword from PEP 492 are reinstated. (For 7 or higher, these are unconditionally treated as keywords, but they are still special tokens rather than `NAME` tokens that the parser driver recognizes.) https://bugs.python.org/issue35975
1 parent bf94cc7 commit 495da29

29 files changed

Lines changed: 473 additions & 198 deletions

‎Doc/library/ast.rst‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -126,7 +126,7 @@ The abstract grammar is currently defined as follows:
126126
Apart from the node classes, the :mod:`ast` module defines these utility functions
127127
and classes for traversing abstract syntax trees:
128128

129-
.. function:: parse(source, filename='<unknown>', mode='exec', *, type_comments=False)
129+
.. function:: parse(source, filename='<unknown>', mode='exec', *, type_comments=False, feature_version=-1)
130130

131131
Parse the source into an AST node. Equivalent to ``compile(source,
132132
filename, mode, ast.PyCF_ONLY_AST)``.
@@ -145,13 +145,19 @@ and classes for traversing abstract syntax trees:
145145
modified to correspond to :pep:`484` "signature type comments",
146146
e.g. ``(str, int) -> List[str]``.
147147

148+
Also, setting ``feature_version`` to the minor version of an
149+
earlier Python 3 version will attempt to parse using that version's
150+
grammar. For example, setting ``feature_version=4`` will allow
151+
the use of ``async`` and ``await`` as variable names. The lowest
152+
supported value is 4; the highest is ``sys.version_info[1]``.
153+
148154
.. warning::
149155
It is possible to crash the Python interpreter with a
150156
sufficiently large/complex string due to stack depth limitations
151157
in Python's AST compiler.
152158

153159
.. versionchanged:: 3.8
154-
Added ``type_comments=True`` and ``mode='func_type'``.
160+
Added ``type_comments``, ``mode='func_type'`` and ``feature_version``.
155161

156162

157163
.. function:: literal_eval(node_or_string)

‎Doc/library/token-list.inc‎

Lines changed: 4 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Doc/library/token.rst‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -88,3 +88,6 @@ the :mod:`tokenize` module.
8888

8989
.. versionchanged:: 3.8
9090
Added :data:`TYPE_COMMENT`.
91+
Added :data:`AWAIT` and :data:`ASYNC` tokens back (they're needed
92+
to support parsing older Python versions for :func:`ast.parse` with
93+
``feature_version`` set to 6 or lower).

‎Grammar/Grammar‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ decorator: '@' dotted_name [ '(' [arglist] ')' ] NEWLINE
1818
decorators: decorator+
1919
decorated: decorators (classdef | funcdef | async_funcdef)
2020

21-
async_funcdef: 'async' funcdef
21+
async_funcdef: ASYNC funcdef
2222
funcdef: 'def' NAME parameters ['->' test] ':' [TYPE_COMMENT] func_body_suite
2323

2424
parameters: '(' [typedargslist] ')'
@@ -70,7 +70,7 @@ nonlocal_stmt: 'nonlocal' NAME (',' NAME)*
7070
assert_stmt: 'assert' test [',' test]
7171

7272
compound_stmt: if_stmt | while_stmt | for_stmt | try_stmt | with_stmt | funcdef | classdef | decorated | async_stmt
73-
async_stmt: 'async' (funcdef | with_stmt | for_stmt)
73+
async_stmt: ASYNC (funcdef | with_stmt | for_stmt)
7474
if_stmt: 'if' namedexpr_test ':' suite ('elif' namedexpr_test ':' suite)* ['else' ':' suite]
7575
while_stmt: 'while' namedexpr_test ':' suite ['else' ':' suite]
7676
for_stmt: 'for' exprlist 'in' testlist ':' [TYPE_COMMENT] suite ['else' ':' suite]
@@ -106,7 +106,7 @@ arith_expr: term (('+'|'-') term)*
106106
term: factor (('*'|'@'|'/'|'%'|'//') factor)*
107107
factor: ('+'|'-'|'~') factor | power
108108
power: atom_expr ['**' factor]
109-
atom_expr: ['await'] atom trailer*
109+
atom_expr: [AWAIT] atom trailer*
110110
atom: ('(' [yield_expr|testlist_comp] ')' |
111111
'[' [testlist_comp] ']' |
112112
'{' [dictorsetmaker] '}' |
@@ -144,7 +144,7 @@ argument: ( test [comp_for] |
144144

145145
comp_iter: comp_for | comp_if
146146
sync_comp_for: 'for' exprlist 'in' or_test [comp_iter]
147-
comp_for: ['async'] sync_comp_for
147+
comp_for: [ASYNC] sync_comp_for
148148
comp_if: 'if' test_nocond [comp_iter]
149149

150150
# not used in grammar, but may appear in "node" passed from Parser to Compiler

‎Grammar/Tokens‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,8 @@ ELLIPSIS '...'
5555
COLONEQUAL ':='
5656

5757
OP
58+
AWAIT
59+
ASYNC
5860
TYPE_IGNORE
5961
TYPE_COMMENT
6062
ERRORTOKEN

‎Include/Python-ast.h‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Include/compile.h‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@ PyAPI_FUNC(PyCodeObject *) PyNode_Compile(struct _node *, const char *);
2727
#ifndef Py_LIMITED_API
2828
typedef struct {
2929
int cf_flags; /* bitmask of CO_xxx flags relevant to future */
30+
int cf_feature_version; /* minor Python version (PyCF_ONLY_AST) */
3031
} PyCompilerFlags;
3132
#endif
3233

‎Include/parsetok.h‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@ typedef struct {
3535
#define PyPARSE_IGNORE_COOKIE 0x0010
3636
#define PyPARSE_BARRY_AS_BDFL 0x0020
3737
#define PyPARSE_TYPE_COMMENTS 0x0040
38+
#define PyPARSE_ASYNC_HACKS 0x0080
3839

3940
PyAPI_FUNC(node *) PyParser_ParseString(const char *, grammar *, int,
4041
perrdetail *);

‎Include/token.h‎

Lines changed: 6 additions & 4 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Lib/ast.py‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,8 @@
2727
from _ast import *
2828

2929

30-
def parse(source, filename='<unknown>', mode='exec', *, type_comments=False):
30+
def parse(source, filename='<unknown>', mode='exec', *,
31+
type_comments=False, feature_version=-1):
3132
"""
3233
Parse the source into an AST node.
3334
Equivalent to compile(source, filename, mode, PyCF_ONLY_AST).
@@ -36,7 +37,8 @@ def parse(source, filename='<unknown>', mode='exec', *, type_comments=False):
3637
flags = PyCF_ONLY_AST
3738
if type_comments:
3839
flags |= PyCF_TYPE_COMMENTS
39-
return compile(source, filename, mode, flags)
40+
return compile(source, filename, mode, flags,
41+
feature_version=feature_version)
4042

4143

4244
def literal_eval(node_or_string):

0 commit comments

Comments
 (0)