Hello, and thanks for your contribution!

I'm a bot set up to make sure that the project can legally accept your contribution by verifying you have signed the PSF contributor agreement (CLA).

Our records indicate we have not received your CLA. For legal reasons we need you to sign this before we can look at your contribution. Please follow the steps outlined in the CPython devguide to rectify this issue.

If you have recently signed the CLA, please wait at least one business day
before our records are updated.

You can check yourself to see if the CLA has been received.

Thanks again for your contribution, we look forward to reviewing it!

Yes, just found one. Currently (pre-PR)

def f():
    \
  x = 1
  return x

throws an IndentationError (unmatched dedent) on line 4, because line 3 has indentation level from line 2. IMO indent from line 2 should be ignored, because it's empty.

Now this code is accepted, because we calculate indentation after line continuation and then push it to the stack. C and Python tokenizers should behave the same.

This is no longer relevant after the changes to the tokenizer in 3.12.

@lysnikolaou this change also was fixing the c tokenizer

The C tokenizer tokenizes the file attached to the issue correctly as far as I can tell.

cpython on  main [$?] via C v15.0.0-clang via 🐍 pyenv 3.11.3 
❯ cat linecont.py
\

\

\


cpython on  main [$?] via C v15.0.0-clang via 🐍 pyenv 3.11.3 
❯ ./python.exe -m tokenize linecont.py
0,0-0,0:            ENCODING       'utf-8'        
2,0-2,1:            NL             '\n'           
4,0-4,1:            NL             '\n'           
6,0-6,1:            NL             '\n'           
7,0-7,0:            ENDMARKER      ''