Skip to content

Commit 00676d1

Browse files
author
Victor Stinner
committed
Issue python#9738: Document encodings of AST, compiler, parser and PyRun functions
1 parent dc2081f commit 00676d1

6 files changed

Lines changed: 111 additions & 49 deletions

File tree

‎Doc/c-api/veryhigh.rst‎

Lines changed: 20 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -66,8 +66,9 @@ the same library that the Python runtime is using.
6666
If *fp* refers to a file associated with an interactive device (console or
6767
terminal input or Unix pseudo-terminal), return the value of
6868
:c:func:`PyRun_InteractiveLoop`, otherwise return the result of
69-
:c:func:`PyRun_SimpleFile`. If *filename* is *NULL*, this function uses
70-
``"???"`` as the filename.
69+
:c:func:`PyRun_SimpleFile`. *filename* is decoded from the filesystem
70+
encoding (:func:`sys.getfilesystemencoding`). If *filename* is *NULL*, this
71+
function uses ``"???"`` as the filename.
7172
7273
7374
.. c:function:: int PyRun_SimpleString(const char *command)
@@ -110,9 +111,10 @@ the same library that the Python runtime is using.
110111
.. c:function:: int PyRun_SimpleFileExFlags(FILE *fp, const char *filename, int closeit, PyCompilerFlags *flags)
111112
112113
Similar to :c:func:`PyRun_SimpleStringFlags`, but the Python source code is read
113-
from *fp* instead of an in-memory string. *filename* should be the name of the
114-
file. If *closeit* is true, the file is closed before PyRun_SimpleFileExFlags
115-
returns.
114+
from *fp* instead of an in-memory string. *filename* should be the name of
115+
the file, it is decoded from the filesystem encoding
116+
(:func:`sys.getfilesystemencoding`). If *closeit* is true, the file is
117+
closed before PyRun_SimpleFileExFlags returns.
116118
117119
118120
.. c:function:: int PyRun_InteractiveOne(FILE *fp, const char *filename)
@@ -125,7 +127,10 @@ the same library that the Python runtime is using.
125127
126128
Read and execute a single statement from a file associated with an
127129
interactive device according to the *flags* argument. The user will be
128-
prompted using ``sys.ps1`` and ``sys.ps2``. Returns ``0`` when the input was
130+
prompted using ``sys.ps1`` and ``sys.ps2``. *filename* is decoded from the
131+
filesystem encoding (:func:`sys.getfilesystemencoding`).
132+
133+
Returns ``0`` when the input was
129134
executed successfully, ``-1`` if there was an exception, or an error code
130135
from the :file:`errcode.h` include file distributed as part of Python if
131136
there was a parse error. (Note that :file:`errcode.h` is not included by
@@ -142,7 +147,8 @@ the same library that the Python runtime is using.
142147
143148
Read and execute statements from a file associated with an interactive device
144149
until EOF is reached. The user will be prompted using ``sys.ps1`` and
145-
``sys.ps2``. Returns ``0`` at EOF.
150+
``sys.ps2``. *filename* is decoded from the filesystem encoding
151+
(:func:`sys.getfilesystemencoding`). Returns ``0`` at EOF.
146152
147153
148154
.. c:function:: struct _node* PyParser_SimpleParseString(const char *str, int start)
@@ -164,7 +170,8 @@ the same library that the Python runtime is using.
164170
Parse Python source code from *str* using the start token *start* according to
165171
the *flags* argument. The result can be used to create a code object which can
166172
be evaluated efficiently. This is useful if a code fragment must be evaluated
167-
many times.
173+
many times. *filename* is decoded from the filesystem encoding
174+
(:func:`sys.getfilesystemencoding`).
168175
169176
170177
.. c:function:: struct _node* PyParser_SimpleParseFile(FILE *fp, const char *filename, int start)
@@ -217,7 +224,8 @@ the same library that the Python runtime is using.
217224
.. c:function:: PyObject* PyRun_FileExFlags(FILE *fp, const char *filename, int start, PyObject *globals, PyObject *locals, int closeit, PyCompilerFlags *flags)
218225
219226
Similar to :c:func:`PyRun_StringFlags`, but the Python source code is read from
220-
*fp* instead of an in-memory string. *filename* should be the name of the file.
227+
*fp* instead of an in-memory string. *filename* should be the name of the file,
228+
it is decoded from the filesystem encoding (:func:`sys.getfilesystemencoding`).
221229
If *closeit* is true, the file is closed before :c:func:`PyRun_FileExFlags`
222230
returns.
223231
@@ -241,8 +249,9 @@ the same library that the Python runtime is using.
241249
code which can be compiled and should be :const:`Py_eval_input`,
242250
:const:`Py_file_input`, or :const:`Py_single_input`. The filename specified by
243251
*filename* is used to construct the code object and may appear in tracebacks or
244-
:exc:`SyntaxError` exception messages. This returns *NULL* if the code cannot
245-
be parsed or compiled.
252+
:exc:`SyntaxError` exception messages, it is decoded from the filesystem
253+
encoding (:func:`sys.getfilesystemencoding`). This returns *NULL* if the
254+
code cannot be parsed or compiled.
246255
247256
The integer *optimize* specifies the optimization level of the compiler; a
248257
value of ``-1`` selects the optimization level of the interpreter as given by

‎Include/ast.h‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,11 @@
44
extern "C" {
55
#endif
66

7-
PyAPI_FUNC(mod_ty) PyAST_FromNode(const node *, PyCompilerFlags *flags,
8-
const char *, PyArena *);
7+
PyAPI_FUNC(mod_ty) PyAST_FromNode(
8+
const node *n,
9+
PyCompilerFlags *flags,
10+
const char *filename, /* decoded from the filesystem encoding */
11+
PyArena *arena);
912

1013
#ifdef __cplusplus
1114
}

‎Include/compile.h‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -30,8 +30,12 @@ typedef struct {
3030

3131
struct _mod; /* Declare the existence of this type */
3232
#define PyAST_Compile(mod, s, f, ar) PyAST_CompileEx(mod, s, f, -1, ar)
33-
PyAPI_FUNC(PyCodeObject *) PyAST_CompileEx(struct _mod *, const char *,
34-
PyCompilerFlags *, int, PyArena *);
33+
PyAPI_FUNC(PyCodeObject *) PyAST_CompileEx(
34+
mod_ty mod,
35+
const char *filename, /* decoded from the filesystem encoding */
36+
PyCompilerFlags *flags,
37+
int optimize,
38+
PyArena *arena);
3539
PyAPI_FUNC(PyFutureFeatures *) PyFuture_FromAST(struct _mod *, const char *);
3640

3741

‎Include/parsetok.h‎

Lines changed: 20 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -9,10 +9,10 @@ extern "C" {
99

1010
typedef struct {
1111
int error;
12-
const char *filename;
12+
const char *filename; /* decoded from the filesystem encoding */
1313
int lineno;
1414
int offset;
15-
char *text;
15+
char *text; /* UTF-8-encoded string */
1616
int token;
1717
int expected;
1818
} perrdetail;
@@ -39,23 +39,32 @@ PyAPI_FUNC(node *) PyParser_ParseFile (FILE *, const char *, grammar *, int,
3939

4040
PyAPI_FUNC(node *) PyParser_ParseStringFlags(const char *, grammar *, int,
4141
perrdetail *, int);
42-
PyAPI_FUNC(node *) PyParser_ParseFileFlags(FILE *, const char *,
42+
PyAPI_FUNC(node *) PyParser_ParseFileFlags(FILE *, const char *,
4343
const char*, grammar *,
4444
int, char *, char *,
4545
perrdetail *, int);
46-
PyAPI_FUNC(node *) PyParser_ParseFileFlagsEx(FILE *, const char *,
47-
const char*, grammar *,
48-
int, char *, char *,
49-
perrdetail *, int *);
46+
PyAPI_FUNC(node *) PyParser_ParseFileFlagsEx(
47+
FILE *fp,
48+
const char *filename, /* decoded from the filesystem encoding */
49+
const char *enc,
50+
grammar *g,
51+
int start,
52+
char *ps1,
53+
char *ps2,
54+
perrdetail *err_ret,
55+
int *flags);
5056

5157
PyAPI_FUNC(node *) PyParser_ParseStringFlagsFilename(const char *,
5258
const char *,
5359
grammar *, int,
5460
perrdetail *, int);
55-
PyAPI_FUNC(node *) PyParser_ParseStringFlagsFilenameEx(const char *,
56-
const char *,
57-
grammar *, int,
58-
perrdetail *, int *);
61+
PyAPI_FUNC(node *) PyParser_ParseStringFlagsFilenameEx(
62+
const char *s,
63+
const char *filename, /* decoded from the filesystem encoding */
64+
grammar *g,
65+
int start,
66+
perrdetail *err_ret,
67+
int *flags);
5968

6069
/* Note that he following function is defined in pythonrun.c not parsetok.c. */
6170
PyAPI_FUNC(void) PyParser_SetError(perrdetail *);

‎Include/pythonrun.h‎

Lines changed: 54 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -38,19 +38,41 @@ PyAPI_FUNC(void) Py_EndInterpreter(PyThreadState *);
3838
#ifndef Py_LIMITED_API
3939
PyAPI_FUNC(int) PyRun_SimpleStringFlags(const char *, PyCompilerFlags *);
4040
PyAPI_FUNC(int) PyRun_AnyFileFlags(FILE *, const char *, PyCompilerFlags *);
41-
PyAPI_FUNC(int) PyRun_AnyFileExFlags(FILE *, const char *, int, PyCompilerFlags *);
42-
PyAPI_FUNC(int) PyRun_SimpleFileExFlags(FILE *, const char *, int, PyCompilerFlags *);
43-
PyAPI_FUNC(int) PyRun_InteractiveOneFlags(FILE *, const char *, PyCompilerFlags *);
44-
PyAPI_FUNC(int) PyRun_InteractiveLoopFlags(FILE *, const char *, PyCompilerFlags *);
45-
46-
PyAPI_FUNC(struct _mod *) PyParser_ASTFromString(const char *, const char *,
47-
int, PyCompilerFlags *flags,
48-
PyArena *);
49-
PyAPI_FUNC(struct _mod *) PyParser_ASTFromFile(FILE *, const char *,
50-
const char*, int,
51-
char *, char *,
52-
PyCompilerFlags *, int *,
53-
PyArena *);
41+
PyAPI_FUNC(int) PyRun_AnyFileExFlags(
42+
FILE *fp,
43+
const char *filename, /* decoded from the filesystem encoding */
44+
int closeit,
45+
PyCompilerFlags *flags);
46+
PyAPI_FUNC(int) PyRun_SimpleFileExFlags(
47+
FILE *fp,
48+
const char *filename, /* decoded from the filesystem encoding */
49+
int closeit,
50+
PyCompilerFlags *flags);
51+
PyAPI_FUNC(int) PyRun_InteractiveOneFlags(
52+
FILE *fp,
53+
const char *filename, /* decoded from the filesystem encoding */
54+
PyCompilerFlags *flags);
55+
PyAPI_FUNC(int) PyRun_InteractiveLoopFlags(
56+
FILE *fp,
57+
const char *filename, /* decoded from the filesystem encoding */
58+
PyCompilerFlags *flags);
59+
60+
PyAPI_FUNC(struct _mod *) PyParser_ASTFromString(
61+
const char *s,
62+
const char *filename, /* decoded from the filesystem encoding */
63+
int start,
64+
PyCompilerFlags *flags,
65+
PyArena *arena);
66+
PyAPI_FUNC(struct _mod *) PyParser_ASTFromFile(
67+
FILE *fp,
68+
const char *filename, /* decoded from the filesystem encoding */
69+
const char* enc,
70+
int start,
71+
char *ps1,
72+
char *ps2,
73+
PyCompilerFlags *flags,
74+
int *errcode,
75+
PyArena *arena);
5476
#endif
5577

5678
#ifndef PyParser_SimpleParseString
@@ -68,20 +90,32 @@ PyAPI_FUNC(struct _node *) PyParser_SimpleParseFileFlags(FILE *, const char *,
6890
PyAPI_FUNC(PyObject *) PyRun_StringFlags(const char *, int, PyObject *,
6991
PyObject *, PyCompilerFlags *);
7092

71-
PyAPI_FUNC(PyObject *) PyRun_FileExFlags(FILE *, const char *, int,
72-
PyObject *, PyObject *, int,
73-
PyCompilerFlags *);
93+
PyAPI_FUNC(PyObject *) PyRun_FileExFlags(
94+
FILE *fp,
95+
const char *filename, /* decoded from the filesystem encoding */
96+
int start,
97+
PyObject *globals,
98+
PyObject *locals,
99+
int closeit,
100+
PyCompilerFlags *flags);
74101
#endif
75102

76103
#ifdef Py_LIMITED_API
77104
PyAPI_FUNC(PyObject *) Py_CompileString(const char *, const char *, int);
78105
#else
79106
#define Py_CompileString(str, p, s) Py_CompileStringExFlags(str, p, s, NULL, -1)
80107
#define Py_CompileStringFlags(str, p, s, f) Py_CompileStringExFlags(str, p, s, f, -1)
81-
PyAPI_FUNC(PyObject *) Py_CompileStringExFlags(const char *, const char *, int,
82-
PyCompilerFlags *, int);
83-
#endif
84-
PyAPI_FUNC(struct symtable *) Py_SymtableString(const char *, const char *, int);
108+
PyAPI_FUNC(PyObject *) Py_CompileStringExFlags(
109+
const char *str,
110+
const char *filename, /* decoded from the filesystem encoding */
111+
int start,
112+
PyCompilerFlags *flags,
113+
int optimize);
114+
#endif
115+
PyAPI_FUNC(struct symtable *) Py_SymtableString(
116+
const char *str,
117+
const char *filename, /* decoded from the filesystem encoding */
118+
int start);
85119

86120
PyAPI_FUNC(void) PyErr_Print(void);
87121
PyAPI_FUNC(void) PyErr_PrintEx(int);

‎Include/symtable.h‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,8 @@ typedef enum _block_type { FunctionBlock, ClassBlock, ModuleBlock }
1616
struct _symtable_entry;
1717

1818
struct symtable {
19-
const char *st_filename; /* name of file being compiled */
19+
const char *st_filename; /* name of file being compiled,
20+
decoded from the filesystem encoding */
2021
struct _symtable_entry *st_cur; /* current symbol table entry */
2122
struct _symtable_entry *st_top; /* symbol table entry for module */
2223
PyObject *st_blocks; /* dict: map AST node addresses
@@ -60,8 +61,10 @@ PyAPI_DATA(PyTypeObject) PySTEntry_Type;
6061

6162
PyAPI_FUNC(int) PyST_GetScope(PySTEntryObject *, PyObject *);
6263

63-
PyAPI_FUNC(struct symtable *) PySymtable_Build(mod_ty, const char *,
64-
PyFutureFeatures *);
64+
PyAPI_FUNC(struct symtable *) PySymtable_Build(
65+
mod_ty mod,
66+
const char *filename, /* decoded from the filesystem encoding */
67+
PyFutureFeatures *future);
6568
PyAPI_FUNC(PySTEntryObject *) PySymtable_Lookup(struct symtable *, void *);
6669

6770
PyAPI_FUNC(void) PySymtable_Free(struct symtable *);

0 commit comments

Comments
 (0)