bpo-36654: add example to generate token from another file by Windsooon · Pull Request #12947 · python/cpython

Windsooon · 2019-04-25T05:20:06Z

Add example to generate token from another file.

https://bugs.python.org/issue36654

eamanu

This example confuse me, because the file example.py does not exist. Maybe you need to do something like hello.py

berkerpeksag · 2019-04-27T19:24:21Z

+Example of tokenizing from another file::
+
+    import tokenize
+    f = open('example.py', 'rb')


Using tokenize.open() would be better as it automatically detect the encoding of the file:

cpython/Lib/tokenize.py

Line 449 in 5d90954

encoding, lines = detect_encoding(buffer.readline)

Also, I agree with Emmanuel that using the existing hello.py as an example would make the example easier to follow.

I found the tokenize function already call detect_encoding.

I just realized that tokenize.open() returns a text stream, so passing it to tokenize.tokenize() wouldn't work.

We can use tokenize.generate_tokens() to demonstrate how the str API works:

import tokenize with tokenize.open('hello.py') as f: token_gen = tokenize.generate_tokens(f.readline) for token in token_gen: print(token)

Then fallback to your example to show the usage of the bytes API:

import tokenize with open('hello.py', 'rb') as f: token_gen = tokenize.tokenize(f.readline) for token in token_gen: print(token)

bedevere-bot · 2019-04-27T19:25:31Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

matrixise · 2019-09-11T12:59:26Z

Hi @Windsooon

Could you update your PR with the last comment of @berkerpeksag and with the with statement?

Thank you

matrixise

Thanks for your contribution but you have to update your PR with the recommendations of @berkerpeksag

Windsooon · 2019-09-11T23:30:07Z

Hello, @matrixise, I would love to update the example if needed. However, Serhiy Storchaka didn't agree with it

I do not think a new example is needed. The existing example already demonstrates the use of file's readline method.

https://bugs.python.org/issue36654

berkerpeksag · 2019-09-12T11:36:33Z

I think Serhiy's comment was about the current form of the PR. In my last comment, I was proposing to add examples for both bytes and unicode APIs of the tokenize module.

As a core developer, even I missed that generate_token() was made public. I'm pretty sure more people would like to know that the module now has an API to pass unicode input.

csabella · 2020-01-12T14:02:50Z

@Windsooon, any updates? Thanks!

Windsooon · 2020-01-12T14:05:49Z

Sure, I will fix it in two days. Thank you for reminding.

Windsooon · 2020-01-13T05:37:36Z

Hi, @csabella. I just updated the PR base on @berkerpeksag 's suggestion.

berkerpeksag

Looks pretty good to me, thank you!

Comments have been addressed.

bedevere-bot · 2020-01-25T19:23:03Z

@berkerpeksag: Please replace # with GH- in the commit message next time. Thanks!

miss-islington · 2020-01-25T19:23:03Z

Thanks @Windsooon for the PR, and @berkerpeksag for merging it 🌮🎉.. I'm working now to backport this PR to: 3.7, 3.8.
🐍🍒⛏🤖

…honGH-12947) (cherry picked from commit 4b09dc7) Co-authored-by: Windson yang <wiwindson@outlook.com>

bedevere-bot · 2020-01-25T19:23:14Z

GH-18187 is a backport of this pull request to the 3.8 branch.

…honGH-12947) (cherry picked from commit 4b09dc7) Co-authored-by: Windson yang <wiwindson@outlook.com>

bedevere-bot · 2020-01-25T19:23:21Z

GH-18188 is a backport of this pull request to the 3.7 branch.

…H-12947) (cherry picked from commit 4b09dc7) Co-authored-by: Windson yang <wiwindson@outlook.com>

…hon#12947)

bpo-36654: add example to generate token from another file

77f1024

the-knights-who-say-ni added the CLA signed label Apr 25, 2019

bedevere-bot added docs Documentation in the Doc dir awaiting review labels Apr 25, 2019

eamanu reviewed Apr 27, 2019

View reviewed changes

berkerpeksag requested changes Apr 27, 2019

View reviewed changes

bedevere-bot added awaiting changes and removed awaiting review labels Apr 27, 2019

berkerpeksag added needs backport to 3.7 skip news labels Apr 27, 2019

change example file name

dbafadc

matrixise previously requested changes Sep 11, 2019

View reviewed changes

csabella added the needs backport to 3.8 label Jan 12, 2020

add example for tokenizing file using unicode and bytes

f18f1e2

csabella requested a review from berkerpeksag January 13, 2020 12:39

Update tokenize.rst

943c19d

berkerpeksag approved these changes Jan 25, 2020

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting changes labels Jan 25, 2020

berkerpeksag merged commit 4b09dc7 into python:master Jan 25, 2020

bedevere-bot removed the awaiting merge label Jan 25, 2020

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jan 25, 2020

bpo-36654: Add examples for using tokenize module programmically (pyt…

8725dc9

…honGH-12947) (cherry picked from commit 4b09dc7) Co-authored-by: Windson yang <wiwindson@outlook.com>

bedevere-bot removed the needs backport to 3.8 label Jan 25, 2020

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jan 25, 2020

bpo-36654: Add examples for using tokenize module programmically (pyt…

d529008

…honGH-12947) (cherry picked from commit 4b09dc7) Co-authored-by: Windson yang <wiwindson@outlook.com>

bedevere-bot removed the needs backport to 3.7 label Jan 25, 2020

berkerpeksag pushed a commit that referenced this pull request Jan 25, 2020

bpo-36654: Add examples for using tokenize module programmatically (G…

6dbd843

…H-12947) (cherry picked from commit 4b09dc7) Co-authored-by: Windson yang <wiwindson@outlook.com>

shihai1991 pushed a commit to shihai1991/cpython that referenced this pull request Jan 31, 2020

bpo-36654: Add examples for using tokenize module programmically (pyt…

b881caf

…hon#12947)

Uh oh!

Conversation

Windsooon commented Apr 25, 2019 • edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eamanu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

berkerpeksag Apr 27, 2019

Choose a reason for hiding this comment

Uh oh!

berkerpeksag Apr 27, 2019

Choose a reason for hiding this comment

Uh oh!

Windsooon Apr 28, 2019

Choose a reason for hiding this comment

Uh oh!

berkerpeksag Apr 28, 2019

Choose a reason for hiding this comment

Uh oh!

bedevere-bot commented Apr 27, 2019

Uh oh!

matrixise commented Sep 11, 2019

Uh oh!

matrixise left a comment

Choose a reason for hiding this comment

Uh oh!

Windsooon commented Sep 11, 2019

Uh oh!

berkerpeksag commented Sep 12, 2019

Uh oh!

csabella commented Jan 12, 2020

Uh oh!

Windsooon commented Jan 12, 2020 via email

Uh oh!

Windsooon commented Jan 13, 2020

Uh oh!

berkerpeksag left a comment

Choose a reason for hiding this comment

Uh oh!

bedevere-bot commented Jan 25, 2020

Uh oh!

miss-islington commented Jan 25, 2020

Uh oh!

bedevere-bot commented Jan 25, 2020

Uh oh!

bedevere-bot commented Jan 25, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Windsooon commented Apr 25, 2019 •

edited by bedevere-bot

Loading