Skip to content

bpo-47095: Use libb2 to provide blake2 implementation (GH-32059) #32059

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tiran merged 7 commits into from
Mar 26, 2022
Merged

bpo-47095: Use libb2 to provide blake2 implementation (GH-32059) #32059

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
fb159a7
Use libb2 to provide blake2 implementation
tiran Mar 22, 2022
4c0ddb7
Remove unused parallel blake2 and test files
tiran Mar 23, 2022
285b254
Use zero memory and store helpers from blake2-impl.h
tiran Mar 23, 2022
4e8f0f0
blurb, docs, cleanup
tiran Mar 23, 2022
5420a94
Markup fix
tiran Mar 23, 2022
693185f
PKG_CHECK_MODULES() no arg needs to be set to something
tiran Mar 24, 2022
1621bee
the libb2, the
tiran Mar 26, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/posix-deps-apt.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ apt-get -yq install \
ccache \
gdb \
lcov \
libb2-dev \
libbz2-dev \
libffi-dev \
libgdbm-dev \
Expand Down
12 changes: 12 additions & 0 deletions Doc/whatsnew/3.11.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -245,6 +245,12 @@ fractions
that an ``isinstance(some_fraction, typing.SupportsInt)`` check passes.
(Contributed by Mark Dickinson in :issue:`44547`.)

hashlib
-------

* :func:`hashlib.blake2b` and :func:`hashlib.blake2s` now prefer `libb2`_
over Python's vendored copy.
(Contributed by Christian Heimes in :issue:`47095`.)

IDLE and idlelib
----------------
Expand Down Expand Up @@ -1060,6 +1066,9 @@ Porting to Python 3.11
<https://github.com/python/pythoncapi_compat>`__ to get these functions
on old Python functions.

* Distributors are encouraged to build Python with optimized Blake2 library
Copy link
Copy Markdown
Member

@gpshead gpshead Mar 26, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

... with the optimized external ...

`libb2`_.


Deprecated
----------
Expand Down Expand Up @@ -1139,3 +1148,6 @@ Removed
* Remove the ``HAVE_PY_SET_53BIT_PRECISION`` macro (moved to the internal C
API).
(Contributed by Victor Stinner in :issue:`45412`.)


.. _libb2: https://www.blake2.net/
2 changes: 1 addition & 1 deletion Makefile.pre.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2482,7 +2482,7 @@ MODULE_CMATH_DEPS=$(srcdir)/Modules/_math.h
MODULE_MATH_DEPS=$(srcdir)/Modules/_math.h
MODULE_PYEXPAT_DEPS=$(LIBEXPAT_HEADERS) @LIBEXPAT_INTERNAL@
MODULE_UNICODEDATA_DEPS=$(srcdir)/Modules/unicodedata_db.h $(srcdir)/Modules/unicodename_db.h
MODULE__BLAKE2_DEPS=$(srcdir)/Modules/_blake2/impl/blake2-config.h $(srcdir)/Modules/_blake2/impl/blake2-dispatch.c $(srcdir)/Modules/_blake2/impl/blake2-impl.h $(srcdir)/Modules/_blake2/impl/blake2-kat.h $(srcdir)/Modules/_blake2/impl/blake2.h $(srcdir)/Modules/_blake2/impl/blake2b-load-sse2.h $(srcdir)/Modules/_blake2/impl/blake2b-load-sse41.h $(srcdir)/Modules/_blake2/impl/blake2b-ref.c $(srcdir)/Modules/_blake2/impl/blake2b-round.h $(srcdir)/Modules/_blake2/impl/blake2b-test.c $(srcdir)/Modules/_blake2/impl/blake2b.c $(srcdir)/Modules/_blake2/impl/blake2bp-test.c $(srcdir)/Modules/_blake2/impl/blake2bp.c $(srcdir)/Modules/_blake2/impl/blake2s-load-sse2.h $(srcdir)/Modules/_blake2/impl/blake2s-load-sse41.h $(srcdir)/Modules/_blake2/impl/blake2s-load-xop.h $(srcdir)/Modules/_blake2/impl/blake2s-ref.c $(srcdir)/Modules/_blake2/impl/blake2s-round.h $(srcdir)/Modules/_blake2/impl/blake2s-test.c $(srcdir)/Modules/_blake2/impl/blake2s.c $(srcdir)/Modules/_blake2/impl/blake2sp-test.c $(srcdir)/Modules/_blake2/impl/blake2sp.c $(srcdir)/Modules/hashlib.h
MODULE__BLAKE2_DEPS=$(srcdir)/Modules/_blake2/impl/blake2-config.h $(srcdir)/Modules/_blake2/impl/blake2-impl.h $(srcdir)/Modules/_blake2/impl/blake2.h $(srcdir)/Modules/_blake2/impl/blake2b-load-sse2.h $(srcdir)/Modules/_blake2/impl/blake2b-load-sse41.h $(srcdir)/Modules/_blake2/impl/blake2b-ref.c $(srcdir)/Modules/_blake2/impl/blake2b-round.h $(srcdir)/Modules/_blake2/impl/blake2b.c $(srcdir)/Modules/_blake2/impl/blake2s-load-sse2.h $(srcdir)/Modules/_blake2/impl/blake2s-load-sse41.h $(srcdir)/Modules/_blake2/impl/blake2s-load-xop.h $(srcdir)/Modules/_blake2/impl/blake2s-ref.c $(srcdir)/Modules/_blake2/impl/blake2s-round.h $(srcdir)/Modules/_blake2/impl/blake2s.c $(srcdir)/Modules/_blake2/blake2module.h $(srcdir)/Modules/hashlib.h
MODULE__CTYPES_DEPS=$(srcdir)/Modules/_ctypes/ctypes.h
MODULE__DECIMAL_DEPS=$(srcdir)/Modules/_decimal/docstrings.h $(LIBMPDEC_HEADERS) @LIBMPDEC_INTERNAL@
MODULE__ELEMENTTREE_DEPS=$(srcdir)/Modules/pyexpat.c $(LIBEXPAT_HEADERS) @LIBEXPAT_INTERNAL@
Expand Down
2 changes: 2 additions & 0 deletions Misc/NEWS.d/next/Library/2022-03-23-12-07-26.bpo-47095.P3YTrh.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
:mod:`hashlib`'s internal ``_blake2`` module now prefers ``libb2`` from
https://www.blake2.net/ over Python's vendored copy of blake2.
12 changes: 5 additions & 7 deletions Modules/_blake2/blake2b_impl.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,25 +21,23 @@
#include "pycore_strhex.h" // _Py_strhex()

#include "../hashlib.h"
#include "blake2ns.h"

#define HAVE_BLAKE2B 1
#define BLAKE2_LOCAL_INLINE(type) Py_LOCAL_INLINE(type)

#include "impl/blake2.h"
#include "impl/blake2-impl.h" /* for secure_zero_memory() and store48() */
#include "blake2module.h"

#ifndef HAVE_LIBB2
/* pure SSE2 implementation is very slow, so only use the more optimized SSSE3+
* https://bugs.python.org/issue31834 */
#if defined(__SSSE3__) || defined(__SSE4_1__) || defined(__AVX__) || defined(__XOP__)
#include "impl/blake2b.c"
#else
#include "impl/blake2b-ref.c"
#endif
#endif // !HAVE_LIBB2

#define HAVE_BLAKE2B 1

extern PyType_Spec blake2b_type_spec;


typedef struct {
PyObject_HEAD
blake2b_param param;
Expand Down
3 changes: 1 addition & 2 deletions Modules/_blake2/blake2module.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,7 @@
#endif

#include "Python.h"

#include "impl/blake2.h"
#include "blake2module.h"

extern PyType_Spec blake2b_type_spec;
extern PyType_Spec blake2s_type_spec;
Expand Down
21 changes: 16 additions & 5 deletions Modules/_blake2/blake2ns.h → Modules/_blake2/blake2module.h
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,13 @@
/* Prefix all public blake2 symbols with PyBlake2_
*/
#ifndef Py_BLAKE2MODULE_H
#define Py_BLAKE2MODULE_H

#ifndef Py_BLAKE2_NS
#define Py_BLAKE2_NS
#ifdef HAVE_LIBB2
#include <blake2.h>

#else
// use vendored copy of blake2

// Prefix all public blake2 symbols with PyBlake2_
#define blake2b PyBlake2_blake2b
#define blake2b_compress PyBlake2_blake2b_compress
#define blake2b_final PyBlake2_blake2b_final
Expand All @@ -29,4 +33,11 @@
#define blake2sp_init_key PyBlake2_blake2sp_init_key
#define blake2sp_update PyBlake2_blake2sp_update

#endif /* Py_BLAKE2_NS */
#include "impl/blake2.h"

#endif // HAVE_LIBB2

// for secure_zero_memory(), store32(), store48(), and store64()
#include "impl/blake2-impl.h"

#endif // Py_BLAKE2MODULE_H
12 changes: 5 additions & 7 deletions Modules/_blake2/blake2s_impl.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,25 +21,23 @@
#include "pycore_strhex.h" // _Py_strhex()

#include "../hashlib.h"
#include "blake2ns.h"

#define HAVE_BLAKE2S 1
#define BLAKE2_LOCAL_INLINE(type) Py_LOCAL_INLINE(type)

#include "impl/blake2.h"
#include "impl/blake2-impl.h" /* for secure_zero_memory() and store48() */
#include "blake2module.h"

#ifndef HAVE_LIBB2
/* pure SSE2 implementation is very slow, so only use the more optimized SSSE3+
* https://bugs.python.org/issue31834 */
#if defined(__SSSE3__) || defined(__SSE4_1__) || defined(__AVX__) || defined(__XOP__)
#include "impl/blake2s.c"
#else
#include "impl/blake2s-ref.c"
#endif
#endif // !HAVE_LIBB2

#define HAVE_BLAKE2S 1

extern PyType_Spec blake2s_type_spec;


typedef struct {
PyObject_HEAD
blake2s_param param;
Expand Down
Loading