Skip to content

gh-95534: Improve gzip reading speed by 10% #97664

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
gpshead merged 33 commits into from
Oct 17, 2022
Merged

gh-95534: Improve gzip reading speed by 10% #97664

Changes from 1 commit
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
1e13a89
Add code from python-isal project
rhpvorderman Sep 28, 2022
6a5cdfd
Reorder code
rhpvorderman Sep 28, 2022
809ad5f
Add ZlibDecompressor
rhpvorderman Sep 28, 2022
03254b8
Add zlibdecompressor object
rhpvorderman Sep 28, 2022
669848a
Fix compile warnings
rhpvorderman Sep 28, 2022
69ff613
Do not use class input
rhpvorderman Sep 28, 2022
6fa43ae
Fix lock stuff
rhpvorderman Sep 28, 2022
cdc5972
Fix incorrect error handling
rhpvorderman Sep 28, 2022
7820627
Rework _GzipReader to be more efficient
rhpvorderman Sep 28, 2022
6f8b64a
Properly initialize zstate
rhpvorderman Sep 30, 2022
3e2a4f5
Add blurb for increased gzip read speed
rhpvorderman Sep 30, 2022
070df1c
Make sure self->initialised is set to 0. Reword some comments.
rhpvorderman Sep 30, 2022
70b7d4d
Add appropriate doctype in blurb
rhpvorderman Sep 30, 2022
22d3893
Merge branch 'main' into gh-95534
rhpvorderman Sep 30, 2022
18a7692
Add missing NULL member to ZlibDecompressor_Members
rhpvorderman Sep 30, 2022
d54c8b5
Merge branch 'gh-95534' of github.com:rhpvorderman/cpython into gh-95534
rhpvorderman Sep 30, 2022
c90096f
Remove double comment
rhpvorderman Sep 30, 2022
1c15839
Use READ_BUFFER_SIZE in python -m gzip command line application
rhpvorderman Sep 30, 2022
d0ff4f0
Fix error in news entry
rhpvorderman Sep 30, 2022
afd92ab
minor edit, use +=
gpshead Sep 30, 2022
922ac5c
Throw compile warning on zlib versions that are too old
rhpvorderman Oct 2, 2022
dc7de61
Use bool instead of int
rhpvorderman Oct 2, 2022
ca12c1f
Correct spelling of insufficient
rhpvorderman Oct 2, 2022
1ce342b
Put brackets around if statement
rhpvorderman Oct 2, 2022
0b7735e
Remove strange default case
rhpvorderman Oct 2, 2022
043a376
Remove unnecessary zero op
rhpvorderman Oct 2, 2022
2a653a9
Change RetVal to return_value
rhpvorderman Oct 2, 2022
475aef6
Change char to bool
rhpvorderman Oct 2, 2022
41ba076
Properly bracketify if-else clause
rhpvorderman Oct 2, 2022
5f1901d
Prefix underscore to _ZlibDecompressor name
rhpvorderman Oct 2, 2022
c5d6888
Copy explanation about zdict from python docs into function docstring
rhpvorderman Oct 2, 2022
9d60339
Merge branch 'gh-95534' of github.com:rhpvorderman/cpython into gh-95534
rhpvorderman Oct 2, 2022
e3da415
Add tests for _ZlibDecompressor
rhpvorderman Oct 3, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Make sure self->initialised is set to 0. Reword some comments.
  • Loading branch information
@rhpvorderman
rhpvorderman committed Sep 30, 2022
commit 070df1cfeb8d4d7f6be59e31342929baf566936a
20 changes: 12 additions & 8 deletions Modules/zlibmodule.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1469,10 +1469,12 @@ arrange_output_buffer(uint32_t *avail_out,
return ret;
}

/* Decompress data of length d->bzs_avail_in_real in d->state.next_in. The output
buffer is allocated dynamically and returned. At most max_length bytes are
returned, so some of the input may not be consumed. d->state.next_in and
d->bzs_avail_in_real are updated to reflect the consumed input. */
/* Decompress data of length self->avail_in_real in self->state.next_in. The
output buffer is allocated dynamically and returned. If the max_length is
of sufficiently low size, max_length is allocated immediately. At most
max_length bytes are returned, so some of the input may not be consumed.
self->state.next_in and self->avail_in_real are updated to reflect the
consumed input. */
static PyObject*
decompress_buf(ZlibDecompressor *self, Py_ssize_t max_length)
{
Expand All @@ -1486,8 +1488,9 @@ decompress_buf(ZlibDecompressor *self, Py_ssize_t max_length)

int err = Z_OK;

/* In Python 3.10 sometimes sys.maxsize is passed by default. In those cases
we do want to use DEF_BUF_SIZE as start buffer. */
/* When sys.maxsize is passed as default use DEF_BUF_SIZE as start buffer.
In this particular case the data may not necessarily be very big, so
it is better to grow dynamically.*/
if ((max_length < 0) || max_length == PY_SSIZE_T_MAX) {
hard_limit = PY_SSIZE_T_MAX;
obuflen = DEF_BUF_SIZE;
Expand Down Expand Up @@ -1543,13 +1546,14 @@ decompress_buf(ZlibDecompressor *self, Py_ssize_t max_length)
if (err == Z_STREAM_END) {
self->eof = 1;
self->is_initialised = 0;
/* Unlike the Decompress object we end here, since there is no
backwards-compatibility issues */
/* Unlike the Decompress object we call inflateEnd here as there are no
backwards compatibility issues */
err = inflateEnd(&self->zst);
if (err != Z_OK) {
zlib_error(state, self->zst, err, "while finishing decompression");
goto error;
}
self->is_initialised = 0;
Comment thread
rhpvorderman marked this conversation as resolved.
Outdated
} else if (err != Z_OK && err != Z_BUF_ERROR) {
zlib_error(state, self->zst, err, "while decompressing data");
}
Expand Down