Skip to content
Merged
Changes from 1 commit
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
3895a79
Do no allocate large temp buffers in _Pickler.dump
ogrisel Nov 9, 2017
9fc8d67
Add comment to justify iteration over chunks
ogrisel Nov 9, 2017
964a67f
Concatenate short bytes
ogrisel Nov 9, 2017
d7216c1
Add entry to NEWS
ogrisel Nov 9, 2017
410d73a
Reduce overhead for small bytes objects
ogrisel Nov 10, 2017
37b4e5b
Even more overhead reduction for small bytes buffers
ogrisel Nov 10, 2017
ba73a71
Fix no-copy for protocol < 4
ogrisel Nov 10, 2017
8eac6a2
More overhead reduction for small bytes
ogrisel Nov 10, 2017
6adc17b
Direct write for large bytes in the C-pickler
ogrisel Nov 10, 2017
e795e66
Update NEWS to include the C pickler
ogrisel Nov 11, 2017
bb4d3eb
No-copy dump for large unicode in C pickler
ogrisel Nov 12, 2017
5d52bd7
Update NEWS to mention str [ci skip]
ogrisel Nov 12, 2017
054c94b
Simpler NEWS entry [ci skip]
ogrisel Nov 12, 2017
31c3afa
C-code style fixes
ogrisel Nov 12, 2017
dcce63a
Fixed leak
ogrisel Nov 12, 2017
d233208
Do not wrap large objects in frames
ogrisel Nov 12, 2017
dfe6314
Fix bug in function call result handling
ogrisel Nov 12, 2017
61611b2
Fix frameless blobs test for pickletools.optimize
ogrisel Nov 12, 2017
e67df81
Flush to file after each frame commit
ogrisel Nov 12, 2017
d1e9a7d
Extend NEWS entry to mention write-on-frame-commit
ogrisel Nov 12, 2017
7f08831
Make proto 4 Python pickler issue (2 * n_frames + 1) calls to write
ogrisel Nov 17, 2017
750ae86
Cleanup _Pickler_CommitFrame
ogrisel Nov 17, 2017
c0c8973
Get rid of the frameless_blobs=False exception: actually it works out…
ogrisel Nov 17, 2017
304571a
Implement frameless blobs for pickletools.optimize
ogrisel Nov 17, 2017
4ee2ee9
Do not skip test silently if self.pickler is renamed
ogrisel Nov 17, 2017
b4c978b
Remove on attribute lookup
ogrisel Nov 17, 2017
e60a3d5
Typo in comment
ogrisel Nov 17, 2017
8181630
Add comments for tests that require self.pickler
ogrisel Nov 17, 2017
25e4e04
revert variable renaming
ogrisel Nov 24, 2017
b57ef9d
Disable test_framed_write_sizes in OptimizedPickleTests
ogrisel Nov 24, 2017
727b5c2
Small code style fix
ogrisel Nov 24, 2017
8af9321
Add comments to explain tradeoffs in Python pickle frame commit
ogrisel Nov 27, 2017
c7bbffa
Implement no-copy support for delayed writers in Python pickler
ogrisel Dec 3, 2017
796c469
Fix renamed test_framed_write_sizes_with_delayed_writer in test_pickl…
ogrisel Dec 3, 2017
a2e87e9
Add comment to explain flush after commit in _Pickler_OpcodeBoundary
ogrisel Jan 4, 2018
8edaa64
Removed unused argument payload_size
ogrisel Jan 4, 2018
609ebfe
Update the changelog to reflect the new behavior of the Python pickler
ogrisel Jan 4, 2018
28f6297
Fix phrasing
ogrisel Jan 4, 2018
7c91f06
Fixed typos in comments
ogrisel Jan 5, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix no-copy for protocol < 4
  • Loading branch information
ogrisel committed Nov 13, 2017
commit ba73a71abdc20c43f12d2e0affc374ea57c485c3
21 changes: 11 additions & 10 deletions Lib/pickle.py
Original file line number Diff line number Diff line change
Expand Up @@ -217,17 +217,18 @@ def write(self, data):
return self.file_write(data)

def write_large_bytes(self, opcode, size_header, payload):
if len(payload) >= self._FRAME_SIZE_TARGET and self.current_frame:
# Terminate the current frame to write the next frame directly into
# the underlying file to skip the unnecessary memory allocations
# of a large temporary buffer.
self.commit_frame(force=True)
if len(payload) >= self._FRAME_SIZE_TARGET:
write = self.file_write
frame_size = len(opcode) + len(size_header) + len(payload)
write(FRAME + pack("<Q", frame_size))

# Be careful to not concatenate the header and the payload prior to
# calling 'write' as do not want to allocate a large temporary
if self.current_frame:
# Terminate the current frame to write the next frame directly
# into the underlying file to skip the unnecessary memory
# allocations of a large temporary buffer.
self.commit_frame(force=True)
frame_size = len(opcode) + len(size_header) + len(payload)
write(FRAME + pack("<Q", frame_size))

# Be careful not to concatenate the header and the payload prior to
# calling 'write' as we do not want to allocate a large temporary
# bytes object.
write(opcode + size_header)
write(payload)
Expand Down