Repository navigation
"python3 -m pegen python" generates a Python parser that contains invalid code #133560
Closed as not planned
Description
Bug report
Bug description:
I am currently experimenting with the cpython parser/tokenizer, following the parser documentation:
$ python3.14 -m pegen python ../../Grammar/python.gramThis generates a parser.py file, that contains remnants of C code:
# line 34
return _PyAST_FunctionType ( a , b , p -> arena )
# end of file
void *
_PyPegen_parse(Parser *p)
{
// Initialize keywords
p->keywords = reserved_keywords;
p->n_keyword_lists = n_keyword_lists;
p->soft_keywords = soft_keywords;
// Run parser
CPython versions tested on:
CPython main branch
Operating systems tested on:
Linux
Activity
sobolevn commentedon May 7, 2025
This looks like a possible duplicate of #96424
alexprengere commentedon May 7, 2025
OK I get it now, this is definitely a duplicate.
I think perhaps the instructions in parser.md are a bit misleading, as they suggest that when you are "adding some new rules to the grammar" (e.g. updating the
python.gram), "it is a good idea to experiment first by generating a Python parser", when in fact you cannot usepython -m pegen pythonon the modifiedpython.gramyou may be working on.I will see if I can improve the docs once I get a better understanding on how to leverage pegen to generate a Python parser.
lysnikolaou commentedon May 7, 2025
You can do so using the pure Python version of the grammar. I don't think it's up-to-date with the latest updates to the CPython grammar though. I need to find some time to update it.
I think we can close this as won't fix for now. If you wanna update the docs in the future, please open a new issue for that.
sobolevn commentedon May 7, 2025
However, since this is not the first time people ask this - I propose adding a clear error message that this can't be done.
gh-133560: Clarified `parser.md` doc for pegen parser issue in using …