Skip to content

gh-140950: Reset shlex state in push_source() so post-EOF pushes are read#150172

Open
LeSingh1 wants to merge 1 commit into
python:mainfrom
LeSingh1:fix-shlex-push-source-state
Open

gh-140950: Reset shlex state in push_source() so post-EOF pushes are read#150172
LeSingh1 wants to merge 1 commit into
python:mainfrom
LeSingh1:fix-shlex-push-source-state

Conversation

@LeSingh1
Copy link
Copy Markdown

Fixes gh-140950.

After shlex.shlex.get_token() reaches end of input it sets self.state to None. A subsequent push_source() did not reset that state, so the next get_token() call returned EOF immediately and the newly pushed source was silently ignored.

pop_source() already resets state to ' ' for the same reason. This change does the same in push_source(), which is the minimal symmetric fix.

Reproducer from the issue:

>>> from shlex import shlex
>>> lexer = shlex('a')
>>> lexer.get_token()
'a'
>>> lexer.get_token()      # EOF, state becomes None
''
>>> lexer.push_source('b')
>>> lexer.get_token()      # before: '' (EOF); after: 'b'
'b'

Push during an active stream is unaffected because state is already ' ' in that case. A regression test is added in Lib/test/test_shlex.py and a Library news entry is added under Misc/NEWS.d/next/Library/.

…s are read

After get_token() reaches end of input it sets self.state to None.
A subsequent push_source() did not reset state, so the next get_token()
call returned EOF immediately and the newly pushed source was silently
ignored.  pop_source() already resets state to ' ' for the same reason;
do the same in push_source().
@python-cla-bot
Copy link
Copy Markdown

python-cla-bot Bot commented May 20, 2026

All commit authors signed the Contributor License Agreement.

CLA signed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

shlex.push_source() fails to reset lexer state, causing new stream to be ignored if called after EOF

1 participant