Skip to content

gh-150942: Optimize stringlib split/splitlines with _PyList_AppendTakeRef - #155922

Open
danielKim614 wants to merge 2 commits into
python:mainfrom
danielKim614:gh-150942-str-split
Open

gh-150942: Optimize stringlib split/splitlines with _PyList_AppendTakeRef#155922
danielKim614 wants to merge 2 commits into
python:mainfrom
danielKim614:gh-150942-str-split

Conversation

@danielKim614

Copy link
Copy Markdown

Append result items to the output list with _PyList_AppendTakeRef instead of PyList_Append followed by Py_DECREF in the stringlib SPLIT_ADD (past the 12-slot prealloc) and SPLIT_APPEND (used by splitlines() for every line) macros, removing a reference-count round-trip per appended item (and a per-append lock on the free-threaded build). This covers split/rsplit/splitlines for str, bytes and bytearray. The result list is freshly allocated and stays local to the split function until it is returned.

Microbenchmarks

  • Plain release build, macOS arm64
  • Interleaved A/B runs (8 rounds alternating baseline/patched binaries), medians
Benchmark workload main this PR speedup
str_split_10k_pieces ("abc " * 10_000).split(" ") 165 us 152 us 1.09×
str_split_whitespace_10k ("word " * 10_000).split() 184 us 168 us 1.10×
str_splitlines_10k ("line\n" * 10_000).splitlines() 182 us 165 us 1.10×
bytes_split_10k_pieces (b"abc " * 10_000).split(b" ") 148 us 139 us 1.06×
geomean 1.09×

test_str and test_bytes pass, including -R 3:3 refleak runs.

Benchmark script
"""Microbenchmark str/bytes split / rsplit / splitlines (result-list building)."""
import pyperf

# Many small pieces (>12): exercises the append path after the 12-slot prealloc.
import pyperf

# Many small pieces (>12): exercises the append path after the 12-slot prealloc.
STR_MANY = "abc " * 10_000            # .split(" ") -> 10001 pieces
STR_WS = "word  " * 10_000            # .split() whitespace variant
BYTES_MANY = b"abc " * 10_000         # bytes.split
LINES = "line\n" * 10_000             # .splitlines(): SPLIT_APPEND for every line

# Controls
STR_FEW = "a b c d e f g h i j"       # 10 pieces: stays in the SET_ITEM prealloc path
STR_BIG = ("x" * 4096 + " ") * 250    # large pieces: copy-dominated


def bench_method(loops, obj, name, *args):
    method = getattr(obj, name)
    range_it = range(loops)
    t0 = pyperf.perf_counter()
    for _ in range_it:
        method(*args)
    return pyperf.perf_counter() - t0


runner = pyperf.Runner()
runner.bench_time_func("str_split_10k_pieces", bench_method, STR_MANY, "split", " ")
runner.bench_time_func("str_split_whitespace_10k", bench_method, STR_WS, "split")
runner.bench_time_func("str_splitlines_10k", bench_method, LINES, "splitlines")
runner.bench_time_func("bytes_split_10k_pieces", bench_method, BYTES_MANY, "split", b" ")
runner.bench_time_func("str_split_10_pieces_ctl", bench_method, STR_FEW, "split", " ")
runner.bench_time_func("str_split_large_pieces_ctl", bench_method, STR_BIG, "split", " ")

@python-cla-bot

python-cla-bot Bot commented Aug 17, 2026

Copy link
Copy Markdown

All commit authors signed the Contributor License Agreement.

CLA signed

@corona10 corona10 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

@github-project-automation github-project-automation Bot moved this to Todo in Sprint Aug 17, 2026
@github-project-automation github-project-automation Bot moved this from Todo to In Progress in Sprint Aug 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: In Progress

Development

Successfully merging this pull request may close these issues.

2 participants