Fix doctest paste: strip prompts and dedent (#15089)

Summary

This PR improves how IPython handles pasting doctest and xdoctest code snippets. When pasting code like:

>>>     print(1)

IPython now strips the >>> prompt and also removes the extra indentation, so the result is:

print(1)

Previously, the result would have been:

    print(1)

which can cause an IndentationError at top-level.

What changed

• inputtransformer2.py
  • Enhanced PromptStripper to:
    • Detect doctest-style pastes when any line matches ^\s*>>>.
    • In doctest-mode, strip ^\s*>>>\s? and ^\s*\.\.\.\s? (continuation).
    • After stripping, apply textwrap.dedent() to the joined block, then return dedented lines.
    • Only apply dedent when doctest prompts were actually stripped (to avoid changing non-doctest pastes).
    • Only treat ... as a continuation prompt when a >>> is present in the same block (avoids confusing Python Ellipsis with doctest continuation).
• test_inputtransformer2_line.py
  • Added/updated tests covering single-line doctest, indented >>>, multi-line >>> + ... doctest, and standalone ... (unchanged when no >>>).

Description of changes

• Update IPython/core/inputtransformer2.py to enhance the PromptStripper:
  • Detect doctest-mode for a pasted block when any line matches ^\s*>>>.
  • In doctest-mode, strip indented doctest prompts:
    • ^\s*>>>\s? (primary prompt)
    • ^\s*\.\.\.\s? (continuation prompt)
  • After stripping doctest prompts, apply textwrap.dedent() to the joined block and return the dedented lines.
  • Only apply dedent() when doctest prompts were actually stripped to avoid changing normal paste semantics.
• Update/add tests in tests/test_inputtransformer2_line.py for the new behavior.

Motivation and context

Users often copy doctest examples from documentation or tools that show doctest prompts and indentation (for example, xdoctest-style indented examples). Previously IPython removed the >>> prompt but preserved the remaining indentation, which could turn a pasted, single-line doctest into an indented top-level statement and raise IndentationError. This PR makes doctest pastes behave as users expect: prompts are removed and the code block is normalized by dedenting.

Examples (Before → After)

Single-line doctest

Input:

>>>     print(1)

Before:

    print(1)

After:

print(1)

Indented xdoctest-style prompt

Input:

    >>>     print(1)

After:

print(1)

Multi-line doctest with continuation

Input:

>>> for i in range(2):
...     print(i)

After:

for i in range(2):
    print(i)

Standalone continuation prompt (unchanged)

Input:

...     print(1)

After (unchanged because no >>> present):

...     print(1)

Implementation notes

• dedent() is applied only after doctest prompts were detected and stripped. This avoids affecting normal paste behavior for non-doctest code, and prevents accidental removal of meaningful indentation in user-pasted code.
• ... is considered a continuation prompt only when the pasted block contains at least one >>>, to avoid confusing Python's Ellipsis literal with doctest continuations.
• Regexes used are tolerant of leading whitespace to support indented >>> prompts (xdoctest-style).

Related issues

• Fixes Removing leading indentation when pasting in a doctest snippet #15089

Testing performed

• Added and updated tests in tests/test_inputtransformer2_line.py to cover:
  • Single-line doctest with extra indentation: ">>> print(1)\n" -> "print(1)\n"
  • Leading whitespace before >>>: " >>> print(1)\n" -> "print(1)\n"
  • Multiline doctest using >>> and ... continuation lines.
  • Standalone ... only lines remain unchanged.

Doc

Contribution by Gittensor, see my contribution statistics at https://gittensor.io/miners/details?githubId=191372963