Repository navigation
Performance: Optimize csv.writer by reducing time spent re-scanning lineterminator #156955
Description
csv.writer examines every character of every field for membership in
dialect.lineterminator. In join_append_data() (Modules/_csv.c) that test is
a call to PyUnicode_FindChar(), made once per character per pass — and the
function runs two passes over each field, one to size the record buffer and one
to fill it.
perf record on 2000 rows x 4 fields attributes 54% of cycles to
PyUnicode_FindChar:
54.12% python PyUnicode_FindChar
2.02% python PyUnicode_FindChar@plt
25.57% _csv..so join_append
3.12% python listiter_next
Because a dialect's terminator is immutable, we can compute its highest code point once at construction and use that information to avoid unnecessary work.
(PR is already prepped)
Linked PRs
Metadata
Metadata
Assignees
Projects
- Status
Activity
pythongh-156955: Speed up csv.writer by not scanning lineterminator p…
pythongh-156955: Cover line terminator quoting in the csv writer tests
pythongh-156955: Keep the line terminator test additions in their own…
gh-156955: Speed up csv.writer by caching the set of special characte…
pythongh-156955: Speed up csv.writer by caching the set of special ch…