Commits

History for

Commits on Feb 27, 2025

[3.12] gh-125553: Fix backslash continuation in `untokenize` (GH-126010) (#130579)

hugovk
and
tomasr8
authored

Commits on Jan 6, 2025

[3.12] gh-128519: Align the docstring of untokenize() to match the docs (GH-128521) (#128532)

miss-islington
and
tomasr8
authored

Commits on Oct 6, 2024

[3.12] gh-125008: Fix `tokenize.untokenize` roundtrip for `\n{{` (GH-125013) (#125021)
miss-islington
authored

Commits on Feb 19, 2024

[3.12] gh-115154: Fix untokenize handling of unicode named literals (GH-115171) (#115662)

miss-islington
and
pablogsal
authored

Commits on Jun 7, 2023

[3.12] gh-105390: Correctly raise TokenError instead of SyntaxError for tokenize errors (GH-105399) (#105439)
miss-islington
authored

Commits on Jun 5, 2023

[3.12] gh-105324: Fix tokenize module main function for stdin (GH-105325) (#105330)
miss-islington
authored

Commits on May 31, 2023

[3.12] gh-105069: Add a readline-like callable to the tokenizer to consume input iteratively (GH-105070) (#105119)

miss-islington
and
pablogsal
authored

Commits on May 26, 2023

[3.12] gh-104976: Ensure trailing dedent tokens are emitted as the previous tokenizer (GH-104980) (#105000)
miss-islington
authored

Commits on May 22, 2023

gh-102856: Tokenize performance improvement (#104731)
mgmacias95
authored

Commits on May 21, 2023

Commits on Feb 3, 2022

bpo-46565: `del` loop vars that are leaking into module namespaces (GH-30993)
sobolevn
authored

Commits on Aug 24, 2021

Add tests for the C tokenizer and expose it as a private module (GH-27924)
pablogsal
authored

Commits on Jul 31, 2021

bpo-44667: Treat correctly lines ending with comments and no newlines in the Python tokenizer (GH-27499)
pablogsal
authored

Commits on Jan 24, 2021

bpo-43014: Improve performance of tokenize.tokenize by 20-30%
asottile
authored

Commits on May 30, 2019

bpo-5028: Fix up rest of documentation for tokenize documenting line (GH-13686)

asottile
authored and
miss-islington
committed
bpo-5028: fix doc bug for tokenize (GH-11683)

andrewnc
authored and
miss-islington
committed

Commits on May 6, 2019

bpo-36766: Typos in docs and code comments (GH-13116)

GiantRobato
authored and
matrixise
committed

Commits on Dec 22, 2018

bpo-30455: Generate all token related code and docs from Grammar/Tokens. (GH-10370)
serhiy-storchaka
authored

Commits on Jul 6, 2018

bpo-33899: Make tokenize module mirror end-of-file is end-of-line behavior (GH-7891)

ammaraskar
authored and
taleinat
committed

Commits on Jun 5, 2018

bpo-12486: Document tokenize.generate_tokens() as public API (#6957)

takluyver
authored and
willingc
committed

Commits on Apr 23, 2018

bpo-33338: [tokenize] Minor code cleanup (#6573)
ambv
authored

Commits on Apr 11, 2018

bpo-33260: Regenerate token.py after removing ASYNC and AWAIT. (GH-6447)
serhiy-storchaka
authored

Commits on Oct 6, 2017

bpo-30406: Make async and await proper keywords (#1669)

JelleZijlstra
authored and
Yury Selivanov
committed

Commits on May 31, 2017

bpo-25324: copy tok_name before changing it (#1608)

albertjan
authored and
vstinner
committed

Commits on May 24, 2017

bpo-30377: Simplify handling of COMMENT and NL in tokenize.py (#1607)

albertjan
authored and
serhiy-storchaka
committed

Commits on May 18, 2017

bpo-30296 Remove unnecessary tuples, lists, sets, and dicts (#1489)

jdufresne
authored and
rhettinger
committed

Commits on Mar 14, 2017

Add ELLIPSIS and RARROW. Add tests (#666)

DimitrisJim
authored and
Stefan Krah
committed

Commits on Sep 9, 2016

Issue #26331: Implement the parsing part of PEP 515.
brettcannon
committed

Commits on Mar 20, 2016

Commits on Dec 29, 2015

Commits on Oct 26, 2015

Issue 25311: Add support for f-strings to tokenize.py. Also added some comments to explain what's happening, since it's not so obvious.
ericvsmith
committed

Commits on Jul 23, 2015

Issue #24619: Simplify async/await tokenization.
Yury Selivanov
committed