Skip to content

Commit e645761

Browse files
youknowoneShaharNaveh
youknowone
and
ShaharNaveh
authored
Implement _tokenize and update tokenize from v3.14.3 (#7392)
* Base implementation of _tokenize module Port from PR #6240 by ShaharNaveh, adapted to current codebase. Uses ruff_python_parser for tokenization via TokenizerIter. * Update tokenize from v3.14.3 * Rewrite _tokenize with 2-phase model Replace per-line reparsing with single-pass tokenization: - Read all lines via readline, parse once, yield tokens - Fix token type values (COMMENT=65, NL=66, OP=55) - Fix NEWLINE/NL end positions and implicit newline handling - Fix DEDENT positions via look-ahead to next non-DEDENT token - Handle FSTRING_MIDDLE brace unescaping ({{ → {, }} → }) - Emit implicit NL before ENDMARKER when source lacks trailing newline - Raise IndentationError from lexer errors - Remove 13 expectedFailure marks for now-passing tests --------- Co-authored-by: ShaharNaveh <shaharnaveh@users.noreply.github.com> Co-authored-by: CPython Developers <>
2 parents ae8d8c7 + 55737ed commit e645761

File tree

7 files changed

+2872
-378
lines changed

7 files changed

+2872
-378
lines changed

‎Cargo.lock‎

Lines changed: 4 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Lib/test/test_tabnanny.py‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -316,7 +316,6 @@ def validate_cmd(self, *args, stdout="", stderr="", partial=False, expect_failur
316316
self.assertListEqual(out.splitlines(), stdout.splitlines())
317317
self.assertListEqual(err.splitlines(), stderr.splitlines())
318318

319-
@unittest.expectedFailure # TODO: RUSTPYTHON; Should displays error when errored python file is given.
320319
def test_with_errored_file(self):
321320
"""Should displays error when errored python file is given."""
322321
with TemporaryPyFile(SOURCE_CODES["wrong_indented"]) as file_path:

0 commit comments

Comments
 (0)