Skip to content

gh-150942: Speed up re.split result building - #155940

Merged
corona10 merged 2 commits into
python:mainfrom
snurf198:gh-150942-re-split-opt
Aug 17, 2026
Merged

gh-150942: Speed up re.split result building#155940
corona10 merged 2 commits into
python:mainfrom
snurf198:gh-150942-re-split-opt

Conversation

@snurf198

@snurf198 snurf198 commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

@hugovk @corona10
Append result items to the output list with _PyList_AppendTakeRef instead of
PyList_Append followed by Py_DECREF, removing a reference-count round-trip
per appended item (and a per-append lock on the free-threaded build). This is
the same change gh-150943 made to re.findall and re.sub/subn, applied to
the one result-building loop it left behind. The error: label only drops
list, and _PyList_AppendTakeRef steals the reference on failure too, so the
error path is unchanged.

Microbenchmarks

Default build

Benchmark main this PR speedup
re_split 4.54 ms 4.31 ms 1.05×
re_split_groups 5.52 ms 5.23 ms 1.06×
re_split_ws 4.18 ms 4.03 ms 1.04×
re_split_bytes 4.08 ms 3.95 ms 1.03×
str_split_control 1.37 ms 1.37 ms 1.00×
geomean 1.03×

Free-threading build

Benchmark main this PR speedup
re_split 6.48 ms 5.84 ms 1.11×
re_split_groups 8.05 ms 6.81 ms 1.18×
re_split_ws 5.74 ms 5.26 ms 1.09×
re_split_bytes 6.13 ms 5.39 ms 1.14×
str_split_control not significant
geomean 1.08×
Benchmark script
"""Microbenchmark re.split (result-list building)."""
import re
import pyperf

WORDS = "the quick brown fox jumps over the lazy dog " * 2000
CSVISH = "field1,field2,field3,field4,field5,field6,field7,field8\n" * 3000
CSVISH_BYTES = CSVISH.encode()

split_re = re.compile(r"[,\n]")
group_re = re.compile(r"([,\n])")
ws_re = re.compile(r"\s+")
bytes_re = re.compile(rb"[,\n]")
nomatch_re = re.compile(r"[;|]")


def b_split():         return split_re.split(CSVISH)
def b_split_groups():  return group_re.split(CSVISH)
def b_split_ws():      return ws_re.split(WORDS)
def b_split_bytes():   return bytes_re.split(CSVISH_BYTES)
def b_split_nomatch(): return nomatch_re.split(CSVISH)
def b_str_split():     return CSVISH.split(",")


if __name__ == "__main__":
    runner = pyperf.Runner()
    runner.bench_func("re_split", b_split)
    runner.bench_func("re_split_groups", b_split_groups)
    runner.bench_func("re_split_ws", b_split_ws)
    runner.bench_func("re_split_bytes", b_split_bytes)
    runner.bench_func("re_split_nomatch", b_split_nomatch)
    runner.bench_func("str_split_control", b_str_split)

@python-cla-bot

python-cla-bot Bot commented Aug 17, 2026

Copy link
Copy Markdown

All commit authors signed the Contributor License Agreement.

CLA signed

@bedevere-app

bedevere-app Bot commented Aug 17, 2026

Copy link
Copy Markdown

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

@corona10 corona10 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI, You don't need NEWS.d for optimization for most of case.

@corona10
corona10 enabled auto-merge (squash) August 17, 2026 06:55
@github-project-automation github-project-automation Bot moved this to Todo in Sprint Aug 17, 2026
@github-project-automation github-project-automation Bot moved this from Todo to In Progress in Sprint Aug 17, 2026
@corona10
corona10 merged commit cbe3a7f into python:main Aug 17, 2026
60 checks passed
@github-project-automation github-project-automation Bot moved this from In Progress to Done in Sprint Aug 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants