From 47cef3177c7c2f87d0b345a9f2b443eb4be58a70 Mon Sep 17 00:00:00 2001 From: "Jonathan G. Underwood" Date: Sun, 11 Feb 2018 10:02:14 +0000 Subject: [PATCH 01/10] Add packages to requirements.txt --- requirements.txt | 2 ++ 1 file changed, 2 insertions(+) diff --git a/requirements.txt b/requirements.txt index dff7e4c8..9dcfe71f 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,4 +1,6 @@ tox pytest +pytest-runner setuptools_scm deprecation +pkgconfig From ff5707e03c728718722b45e5715e6997b94a68ab Mon Sep 17 00:00:00 2001 From: "Jonathan G. Underwood" Date: Sun, 11 Feb 2018 10:05:08 +0000 Subject: [PATCH 02/10] Add future package to requirements.txt --- requirements.txt | 2 ++ 1 file changed, 2 insertions(+) diff --git a/requirements.txt b/requirements.txt index 9dcfe71f..3ff12fe6 100644 --- a/requirements.txt +++ b/requirements.txt @@ -4,3 +4,5 @@ pytest-runner setuptools_scm deprecation pkgconfig +future + From 9e14082e0a4cb08053c790bdf31459e93d636b50 Mon Sep 17 00:00:00 2001 From: "Jonathan G. Underwood" Date: Sun, 11 Feb 2018 10:24:58 +0000 Subject: [PATCH 03/10] Update README.rst --- README.rst | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/README.rst b/README.rst index e12c8ebd..4a7b51c6 100644 --- a/README.rst +++ b/README.rst @@ -32,7 +32,10 @@ The API provided by the frame format bindings follows that of the LZMA, zlib, gzip and bzip2 compression libraries which are provided with the Python standard library. As such, these LZ4 bindings should provide a drop-in alternative to the compression libraries shipped with Python. The package provides context managers -and file handlers support. +and file handler support. + +The bindings drop the GIL when calling in to the underlying LZ4 library, and is +thread safe. An extensive test suite is included. Documenation ============ From 4ecd261c879bcc0931a20b2c880a58ecb8be1746 Mon Sep 17 00:00:00 2001 From: "Jonathan G. Underwood" Date: Sun, 11 Feb 2018 10:26:18 +0000 Subject: [PATCH 04/10] Update intro.rst --- docs/intro.rst | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/intro.rst b/docs/intro.rst index df3ed8a1..4bf7e2af 100644 --- a/docs/intro.rst +++ b/docs/intro.rst @@ -20,3 +20,12 @@ standard container for the compressed data. In the frame format, the data is compressed into a sequence of blocks. The frame format defines a frame header, which contains information about the compressed data such as its size, and defines a standard end of frame marker. + +The API provided by the frame format bindings follows that of the LZMA, zlib, +gzip and bzip2 compression libraries which are provided with the Python standard +library. As such, these LZ4 bindings should provide a drop-in alternative to the +compression libraries shipped with Python. The package provides context managers +and file handler support. + +The bindings drop the GIL when calling in to the underlying LZ4 library, and is +thread safe. An extensive test suite is included. From 3a93e4aea6cae1c8b28d2282c74e5a3e8d9ad29f Mon Sep 17 00:00:00 2001 From: "Jonathan G. Underwood" Date: Wed, 14 Feb 2018 22:12:06 +0000 Subject: [PATCH 05/10] Fix broken call to flush() in LZ4FrameCompressor.finalize() --- lz4/frame/__init__.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lz4/frame/__init__.py b/lz4/frame/__init__.py index f6c98910..a75cc35c 100644 --- a/lz4/frame/__init__.py +++ b/lz4/frame/__init__.py @@ -273,7 +273,7 @@ def finalize(self): `LZ4FrameCompressor.flush()`. """ - result = flush() + result = self.flush() return result def reset(self): From 45bcd96fc2646533fdb553078a4679c08e883a08 Mon Sep 17 00:00:00 2001 From: "Jonathan G. Underwood" Date: Wed, 14 Feb 2018 22:17:56 +0000 Subject: [PATCH 06/10] Make examples in quickstart.rst run under doctest --- docs/quickstart.rst | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/docs/quickstart.rst b/docs/quickstart.rst index fdccbf9f..9d0cfe1c 100644 --- a/docs/quickstart.rst +++ b/docs/quickstart.rst @@ -13,13 +13,15 @@ provides interoperability with other implementations and language bindings. The simplest way to use the frame bindings is via the :py:func:`compress` and :py:func:`decompress` functions:: +.. doctest:: + >>> import os >>> import lz4.frame >>> input_data = 20 * 128 * os.urandom(1024) # Read 20 * 128kb >>> compressed = lz4.frame.compress(input_data) >>> decompressed = lz4.frame.decompress(compressed) >>> decompressed == input_data - Out[6]: True + True The :py:func:`compress` function reads the input data and compresses it and returns a LZ4 frame. A frame consists of a header, and a sequence of blocks of @@ -34,6 +36,8 @@ Working with data in chunks It's often inconvenient to hold the full data in memory, and so functions are also provided to compress and decompress data in chunks:: +.. doctest:: + >>> import lz4.frame >>> import os >>> input_data = 20 * 128 * os.urandom(1024) @@ -41,7 +45,7 @@ also provided to compress and decompress data in chunks:: >>> compressed = lz4.frame.compress_begin(c_context) >>> compressed += lz4.frame.compress_chunk(c_context, input_data[:10 * 128 * 1024]) >>> compressed += lz4.frame.compress_chunk(c_context, input_data[10 * 128 * 1024:]) - >>> compressed += compress_flush(c_context) + >>> compressed += lz4.frame.compress_flush(c_context) Here a compression context is first created which is used to maintain state across calls to the LZ4 library. This is an opaque PyCapsule object. @@ -59,11 +63,13 @@ time without ending the frame by calling :py:func:`compress_flush` with Decompressing data can also be done in a chunked fashion:: +.. doctest:: + >>> d_context = lz4.frame.create_decompression_context() >>> d1, b, e = lz4.frame.decompress_chunk(d_context, compressed[:len(compressed)//2]) >>> d2, b, e = lz4.frame.decompress_chunk(d_context, compressed[len(compressed)//2:]) >>> d1 + d2 == input_data - Out[12]: True + True Note that :py:func:`decompress_chunk` returns a tuple ``(decompressed_data, bytes_read, end_of_frame_indicator)``. ``decompressed_data`` is the decompressed @@ -78,6 +84,8 @@ is more convenient to use the :py:class:`LZ4FrameCompressor` and :py:class:`LZ4FrameDecompressor` classes which provide context manager functionality:: +.. doctest:: + >>> import lz4.frame >>> import os >>> input_data = 20 * 128 * os.urandom(1024) @@ -85,12 +93,12 @@ functionality:: ... compressed = compressor.begin() ... compressed += compressor.compress(input_data[:10 * 128 * 1024]) ... compressed += compressor.compress(input_data[10 * 128 * 1024:]) - ... compressed += compressor.finalize() + ... compressed += compressor.flush() >>> with lz4.frame.LZ4FrameDecompressor() as decompressor: ... decompressed = decompressor.decompress(compressed[:len(compressed)//2]) ... decompressed += decompressor.decompress(compressed[len(compressed)//2:]) >>> decompressed == input_data - Out[13]: True + True Working with compressed files @@ -102,11 +110,15 @@ replacement for that offered in the Python standard library for bz2, gzip and LZMA compressed files. The :py:func:`lz4.frame.open()` function is the most convenient way to work with compressed data files:: +.. doctest:: + >>> import lz4.frame >>> import os >>> input_data = 20 * os.urandom(1024) >>> with lz4.frame.open('testfile', mode='wb') as fp: - ... fp.write(input_data) + ... bytes_written = fp.write(input_data) + ... bytes_written == len(input_data) + True >>> with lz4.frame.open('testfile', mode='r') as fp: ... output_data = fp.read() >>> output_data == input_data From ce846db226668d35c05982965b559f13365c5fde Mon Sep 17 00:00:00 2001 From: "Jonathan G. Underwood" Date: Wed, 14 Feb 2018 22:22:52 +0000 Subject: [PATCH 07/10] Build html docs and run doctest on travis --- .travis.yml | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/.travis.yml b/.travis.yml index 987f8772..665ec180 100644 --- a/.travis.yml +++ b/.travis.yml @@ -4,10 +4,12 @@ python: - 3.4 - 3.5 - 3.6 -install: pip install tox +install: + - pip install tox + - pip install -r docs/requirements.txt script: - tox - + - make -C docs doctest html deploy: - provider: pypi # server: https://test.pypi.org/legacy/ From e7fe190e8d00407d64e77448d0e3b388a026e565 Mon Sep 17 00:00:00 2001 From: "Jonathan G. Underwood" Date: Wed, 14 Feb 2018 22:31:11 +0000 Subject: [PATCH 08/10] Run lz4.block.rst example under doctest --- docs/lz4.block.rst | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/docs/lz4.block.rst b/docs/lz4.block.rst index c938e47d..d50ead59 100644 --- a/docs/lz4.block.rst +++ b/docs/lz4.block.rst @@ -18,11 +18,16 @@ Example usage ------------- To use the lz4 block format bindings is straightforward:: +.. doctest:: + >>> import lz4.block - >>> compressed_data = lz4.block.compress(data) - >>> data == lz4.block.decompress(compressed_data) + >>> import os + >>> input_data = 20 * 128 * os.urandom(1024) # Read 20 * 128kb + >>> compressed_data = lz4.block.compress(input_data) + >>> output_data = lz4.block.decompress(compressed_data) + >>> input_data == output_data True - >>> + Contents ---------------- From e5f1d892a9a9064a800d8fc213c031e7e05f3053 Mon Sep 17 00:00:00 2001 From: "Jonathan G. Underwood" Date: Wed, 14 Feb 2018 22:33:08 +0000 Subject: [PATCH 09/10] Install package before running sphinx on Travis --- .travis.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.travis.yml b/.travis.yml index 665ec180..023acd32 100644 --- a/.travis.yml +++ b/.travis.yml @@ -9,6 +9,7 @@ install: - pip install -r docs/requirements.txt script: - tox + - python setup.py install - make -C docs doctest html deploy: - provider: pypi From 83654d8412d73238c905c8ef7fd73dbf1534d015 Mon Sep 17 00:00:00 2001 From: "Jonathan G. Underwood" Date: Wed, 14 Feb 2018 22:41:23 +0000 Subject: [PATCH 10/10] Add docs/static directory to stop sphinx complaining --- docs/static/.keep | 1 + 1 file changed, 1 insertion(+) create mode 100644 docs/static/.keep diff --git a/docs/static/.keep b/docs/static/.keep new file mode 100644 index 00000000..5444b8b4 --- /dev/null +++ b/docs/static/.keep @@ -0,0 +1 @@ +# This file created to allow us to check an otherwise empty directory in to git \ No newline at end of file