Skip to content

Commit dcfcd14

Browse files
gvanrossumambv
gvanrossum
authored and
ambv
committed
bpo-35766: Merge typed_ast back into CPython (pythonGH-11645)
1 parent d97daeb commit dcfcd14

30 files changed

+2039
-651
lines changed

‎Doc/library/ast.rst‎

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -126,16 +126,33 @@ The abstract grammar is currently defined as follows:
126126
Apart from the node classes, the :mod:`ast` module defines these utility functions
127127
and classes for traversing abstract syntax trees:
128128

129-
.. function:: parse(source, filename='<unknown>', mode='exec')
129+
.. function:: parse(source, filename='<unknown>', mode='exec', *, type_comments=False)
130130

131131
Parse the source into an AST node. Equivalent to ``compile(source,
132132
filename, mode, ast.PyCF_ONLY_AST)``.
133133

134+
If ``type_comments=True`` is given, the parser is modified to check
135+
and return type comments as specified by :pep:`484` and :pep:`526`.
136+
This is equivalent to adding :data:`ast.PyCF_TYPE_COMMENTS` to the
137+
flags passed to :func:`compile()`. This will report syntax errors
138+
for misplaced type comments. Without this flag, type comments will
139+
be ignored, and the ``type_comment`` field on selected AST nodes
140+
will always be ``None``. In addition, the locations of ``# type:
141+
ignore`` comments will be returned as the ``type_ignores``
142+
attribute of :class:`Module` (otherwise it is always an empty list).
143+
144+
In addition, if ``mode`` is ``'func_type'``, the input syntax is
145+
modified to correspond to :pep:`484` "signature type comments",
146+
e.g. ``(str, int) -> List[str]``.
147+
134148
.. warning::
135149
It is possible to crash the Python interpreter with a
136150
sufficiently large/complex string due to stack depth limitations
137151
in Python's AST compiler.
138152

153+
.. versionchanged:: 3.8
154+
Added ``type_comments=True`` and ``mode='func_type'``.
155+
139156

140157
.. function:: literal_eval(node_or_string)
141158

‎Doc/library/token-list.inc‎

Lines changed: 4 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Doc/library/token.rst‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,13 @@ the :mod:`tokenize` module.
6969
always be an ``ENCODING`` token.
7070

7171

72+
.. data:: TYPE_COMMENT
73+
74+
Token value indicating that a type comment was recognized. Such
75+
tokens are only produced when :func:`ast.parse()` is invoked with
76+
``type_comments=True``.
77+
78+
7279
.. versionchanged:: 3.5
7380
Added :data:`AWAIT` and :data:`ASYNC` tokens.
7481

@@ -78,3 +85,6 @@ the :mod:`tokenize` module.
7885
.. versionchanged:: 3.7
7986
Removed :data:`AWAIT` and :data:`ASYNC` tokens. "async" and "await" are
8087
now tokenized as :data:`NAME` tokens.
88+
89+
.. versionchanged:: 3.8
90+
Added :data:`TYPE_COMMENT`.

‎Grammar/Grammar‎

Lines changed: 22 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,9 @@
77
# single_input is a single interactive statement;
88
# file_input is a module or sequence of commands read from an input file;
99
# eval_input is the input for the eval() functions.
10+
# func_type_input is a PEP 484 Python 2 function type comment
1011
# NB: compound_stmt in single_input is followed by extra NEWLINE!
12+
# NB: due to the way TYPE_COMMENT is tokenized it will always be followed by a NEWLINE
1113
single_input: NEWLINE | simple_stmt | compound_stmt NEWLINE
1214
file_input: (NEWLINE | stmt)* ENDMARKER
1315
eval_input: testlist NEWLINE* ENDMARKER
@@ -17,14 +19,14 @@ decorators: decorator+
1719
decorated: decorators (classdef | funcdef | async_funcdef)
1820

1921
async_funcdef: 'async' funcdef
20-
funcdef: 'def' NAME parameters ['->' test] ':' suite
22+
funcdef: 'def' NAME parameters ['->' test] ':' [TYPE_COMMENT] func_body_suite
2123

2224
parameters: '(' [typedargslist] ')'
23-
typedargslist: (tfpdef ['=' test] (',' tfpdef ['=' test])* [',' [
24-
'*' [tfpdef] (',' tfpdef ['=' test])* [',' ['**' tfpdef [',']]]
25-
| '**' tfpdef [',']]]
26-
| '*' [tfpdef] (',' tfpdef ['=' test])* [',' ['**' tfpdef [',']]]
27-
| '**' tfpdef [','])
25+
typedargslist: (tfpdef ['=' test] (',' [TYPE_COMMENT] tfpdef ['=' test])* (TYPE_COMMENT | [',' [TYPE_COMMENT] [
26+
'*' [tfpdef] (',' [TYPE_COMMENT] tfpdef ['=' test])* (TYPE_COMMENT | [',' [TYPE_COMMENT] ['**' tfpdef [','] [TYPE_COMMENT]]])
27+
| '**' tfpdef [','] [TYPE_COMMENT]]])
28+
| '*' [tfpdef] (',' [TYPE_COMMENT] tfpdef ['=' test])* (TYPE_COMMENT | [',' [TYPE_COMMENT] ['**' tfpdef [','] [TYPE_COMMENT]]])
29+
| '**' tfpdef [','] [TYPE_COMMENT])
2830
tfpdef: NAME [':' test]
2931
varargslist: (vfpdef ['=' test] (',' vfpdef ['=' test])* [',' [
3032
'*' [vfpdef] (',' vfpdef ['=' test])* [',' ['**' vfpdef [',']]]
@@ -39,7 +41,7 @@ simple_stmt: small_stmt (';' small_stmt)* [';'] NEWLINE
3941
small_stmt: (expr_stmt | del_stmt | pass_stmt | flow_stmt |
4042
import_stmt | global_stmt | nonlocal_stmt | assert_stmt)
4143
expr_stmt: testlist_star_expr (annassign | augassign (yield_expr|testlist) |
42-
('=' (yield_expr|testlist_star_expr))*)
44+
[('=' (yield_expr|testlist_star_expr))+ [TYPE_COMMENT]] )
4345
annassign: ':' test ['=' (yield_expr|testlist)]
4446
testlist_star_expr: (test|star_expr) (',' (test|star_expr))* [',']
4547
augassign: ('+=' | '-=' | '*=' | '@=' | '/=' | '%=' | '&=' | '|=' | '^=' |
@@ -71,13 +73,13 @@ compound_stmt: if_stmt | while_stmt | for_stmt | try_stmt | with_stmt | funcdef
7173
async_stmt: 'async' (funcdef | with_stmt | for_stmt)
7274
if_stmt: 'if' namedexpr_test ':' suite ('elif' namedexpr_test ':' suite)* ['else' ':' suite]
7375
while_stmt: 'while' test ':' suite ['else' ':' suite]
74-
for_stmt: 'for' exprlist 'in' testlist ':' suite ['else' ':' suite]
76+
for_stmt: 'for' exprlist 'in' testlist ':' [TYPE_COMMENT] suite ['else' ':' suite]
7577
try_stmt: ('try' ':' suite
7678
((except_clause ':' suite)+
7779
['else' ':' suite]
7880
['finally' ':' suite] |
7981
'finally' ':' suite))
80-
with_stmt: 'with' with_item (',' with_item)* ':' suite
82+
with_stmt: 'with' with_item (',' with_item)* ':' [TYPE_COMMENT] suite
8183
with_item: test ['as' expr]
8284
# NB compile.c makes sure that the default except clause is last
8385
except_clause: 'except' [test ['as' NAME]]
@@ -150,3 +152,14 @@ encoding_decl: NAME
150152

151153
yield_expr: 'yield' [yield_arg]
152154
yield_arg: 'from' test | testlist_star_expr
155+
156+
# the TYPE_COMMENT in suites is only parsed for funcdefs,
157+
# but can't go elsewhere due to ambiguity
158+
func_body_suite: simple_stmt | NEWLINE [TYPE_COMMENT NEWLINE] INDENT stmt+ DEDENT
159+
160+
func_type_input: func_type NEWLINE* ENDMARKER
161+
func_type: '(' [typelist] ')' '->' test
162+
# typelist is a modified typedargslist (see above)
163+
typelist: (test (',' test)* [','
164+
['*' [test] (',' test)* [',' '**' test] | '**' test]]
165+
| '*' [test] (',' test)* [',' '**' test] | '**' test)

‎Grammar/Tokens‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,8 @@ ELLIPSIS '...'
5555
COLONEQUAL ':='
5656

5757
OP
58+
TYPE_IGNORE
59+
TYPE_COMMENT
5860
ERRORTOKEN
5961

6062
# These aren't used by the C tokenizer but are needed for tokenize.py

‎Include/Python-ast.h‎

Lines changed: 64 additions & 30 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Include/compile.h‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ PyAPI_FUNC(PyCodeObject *) PyNode_Compile(struct _node *, const char *);
2222
#define PyCF_DONT_IMPLY_DEDENT 0x0200
2323
#define PyCF_ONLY_AST 0x0400
2424
#define PyCF_IGNORE_COOKIE 0x0800
25+
#define PyCF_TYPE_COMMENTS 0x1000
2526

2627
#ifndef Py_LIMITED_API
2728
typedef struct {
@@ -85,10 +86,10 @@ PyAPI_FUNC(int) _PyAST_Optimize(struct _mod *, PyArena *arena, int optimize);
8586

8687
#endif /* !Py_LIMITED_API */
8788

88-
/* These definitions must match corresponding definitions in graminit.h.
89-
There's code in compile.c that checks that they are the same. */
89+
/* These definitions must match corresponding definitions in graminit.h. */
9090
#define Py_single_input 256
9191
#define Py_file_input 257
9292
#define Py_eval_input 258
93+
#define Py_func_type_input 345
9394

9495
#endif /* !Py_COMPILE_H */

‎Include/graminit.h‎

Lines changed: 4 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)