Skip to content

Python wrapper: migrate from pybind11 to nanobind + scikit-build-core - #1109

Closed
stumpylog wants to merge 16 commits into
zxing-cpp:masterfrom
stumpylog:master
Closed

Python wrapper: migrate from pybind11 to nanobind + scikit-build-core#1109
stumpylog wants to merge 16 commits into
zxing-cpp:masterfrom
stumpylog:master

Conversation

@stumpylog

Copy link
Copy Markdown
Contributor

Summary

This replaces the Python wrapper's build system and binding layer entirely. The pybind11 + setuptools/setup.py stack is replaced with nanobind and scikit-build-core, which are the current recommended tools.

  • scikit-build-core + pyproject.toml replaces setup.py -- fully PEP 517/518 compliant, no more setuptools. python -m build and pip install both work correctly.
  • nanobind replaces pybind11. Standalone CMake builds fetch it via zxing_add_package; scikit-build-core builds get it from the build environment declared in build-system.requires.
  • zint/backend is now correctly included in the sdist via a wrappers/python/zint symlink (mirroring the existing core symlink pattern).

Package structure

The module is now a proper Python package rather than a single flat .so:

zxingcpp/
  __init__.py          # re-exports everything from _zxingcpp_core
  _zxingcpp_core.so    # compiled extension (or .pyd on Windows)
  _zxingcpp_core.pyi   # checked-in type stub (PEP 484)
  py.typed             # PEP 561 marker (enables type checker support)
  stubs.pats           # nanobind stub pattern file

Performance improvements

  • numpy fast path: read_barcode and read_barcodes have a dedicated nb::ndarray<nb::numpy, uint8_t> overload that skips the Python buffer protocol and passes the array directly to the C++ core with zero extra copies.
  • GIL release: The GIL is released around all ReadBarcodes and WriteBarcode/WriteSVG calls. The C++ core is CPU-bound and does not call back into Python, so this is safe and allows Python threads to run concurrently with decode/encode operations.
  • nb::is_final() on Barcode, Image, and ImageView: enables CPython 3.11+ LOAD_ATTR specialization (~1.22x faster attribute/method access on those types).
  • nb::rv_policy::move on all Barcode-returning functions: avoids an extra copy when returning decode results to Python.

What did not change

The public Python API is fully backwards-compatible. All existing import zxingcpp code continues to work unchanged. No tests were changed, only some additions

stumpylog and others added 16 commits May 6, 2026 10:45
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When building from the top-level repo with -DZXING_PYTHON_MODULE=ON,
the root zxing.cmake is already included, making the macro available.
Bare find_package(nanobind CONFIG REQUIRED) had no FetchContent fallback.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Restore find_package(Python ...) needed by nanobind in CMakeLists.txt.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- throw nb::python_error() after raise_from instead of re-throwing the
  original exception (which discarded the chained TypeError)
- wrap post-validation code in try-catch so Py_buffer is always released
  even if narrow_cast or ReadBarcodes throws
- add nb::keep_alive<0,1>() on ImageView.__init__ to keep the caller's
  buffer object alive for the ImageView's lifetime

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…rcodes

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Move zxing.cpp into zxingcpp/ package directory, rename C extension
to _zxingcpp_core, add __init__.py and py.typed for PEP 561 compliance,
and use nanobind_add_stub for .pyi generation at build time.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…sdist

Add the nanobind-generated type stub to the source tree so editors and
type checkers work without a build step. Include it in sdist.include so
it is shipped in source distributions as well.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Fix sdist.include paths (relative to pyproject.toml, not repo root)
- Remove module-scope `None: ErrorType` alias from stub (Python keyword)
- Skip zxing.cmake include when SKBUILD is set; use find_package(nanobind)
  directly instead, since scikit-build-core installs nanobind from
  build-system.requires before invoking CMake

The sdist -> wheel path was previously broken because:
1. sdist.include paths had wrong wrappers/python/ prefix
2. zxing.cmake is a symlink preserved as-is in the sdist, so
   zxing_add_package was unavailable in isolated builds

Note: zint/backend/ is not in the sdist (pre-existing limitation); wheel
builds from sdist require -DZXING_WRITERS=OFF or a separate zint source.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Python 3.13's tarfile.data_filter rejects symlinks that resolve outside
the extraction directory. The sdist contained two such symlinks:

- LICENSE (-> ../../LICENSE): excluded via sdist.exclude in pyproject.toml;
  wheel builds from the git tree still get the file via wheel.license-files.
- zxing.cmake (-> ../../zxing.cmake): excluded via sdist.exclude; not needed
  in scikit-build-core (SKBUILD) builds since find_package(nanobind) is used
  directly.

The libzint sources were missing from the sdist because zint is a git
submodule (not enumerated by git ls-files). Add wrappers/python/zint ->
../../zint mirroring the existing core/ symlink so MANIFEST.in's
graft zint/backend can follow it on the filesystem and include the 626
backend files needed to compile the writer backend.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…d setup

Hoist the duplicated find_package(Python) and nanobind discovery out of
the hasParent if/else in CMakeLists.txt so the SKBUILD-vs-zxing_add_package
choice is expressed once. Extract make_reader_options, make_writer_options,
and barcode_from_content helpers in zxing.cpp to remove the four-way
ReaderOptions duplication, two-way WriterOptions duplication, and the
str/bytes content dispatch shared between create_barcode and write_barcode.
All 21 wrapper tests still pass.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When ZXing is built as a shared library (BUILD_SHARED_LIBS=ON), the stub
generator cannot import _zxingcpp_core because ZXing.dll is not on PATH
for the custom build command. Standalone and scikit-build-core builds
default to static linking and are not affected.

Guard nanobind_add_stub with `if (NOT hasParent OR SKBUILD)` so it only
runs in contexts where the DLL is either absent or findable. Always
install the checked-in source-tree stub regardless.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…alization

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…afety

The reader paths (ReadBarcodes via buffer protocol, ImageView, and ndarray
fast-path) already released the GIL before this change. Add matching
gil_scoped_release guards to write_barcode_to_image() and
write_barcode_to_svg() so Python threads are not blocked during
CPU-bound encoding. All Python object construction and PyBuffer calls
remain inside the GIL.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ode/read_barcodes stub signatures

Before this change the stub emitted two @overload entries per function --
one with NDArray[numpy.uint8] (ndarray fast-path) and one with plain
object (buffer/PIL path).  The result was confusing to both type checkers
and IDE autocompletion.

nb::sig() is now applied to all four .def() registrations, giving every
overload the same explicit union type:

    NDArray[numpy.uint8] | PIL.Image.Image | ImageView

A nanobind stub-generator pattern file (stubs.pats) then collapses the
two identical @overload entries for each function into a single
non-overloaded signature, preserving the full docstring via \doc.  This
produces a clean, accurate, single-entry stub for both read_barcode and
read_barcodes.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@stumpylog

stumpylog commented May 6, 2026

Copy link
Copy Markdown
Contributor Author

I do apologize for the size, but swapping out one wrapping method for another isn't really something possible piecemeal.

It starts from the basis of #948 (comment), updating the packaging structure then the conversion and a few nanobind niceties on top. It also makes the sdist a complete package. Untar it and it is buildable as is, since it now includes zint, using the same symlink model as core

This replaces #948 and #1108.

axxel added a commit to axxel/zxing-cpp that referenced this pull request May 15, 2026
This is mainly based on the work of @Doekin from zxing-cpp#948 with additions from
@raymondqin and some cherry picked improvemtns from @stumpylog's zxing-cpp#1109.

Co-authored-by: Doekin <Leetimemp@gmail.com>
Co-authored-by: Trenton H <797416+stumpylog@users.noreply.github.com>
axxel added a commit to axxel/zxing-cpp that referenced this pull request May 16, 2026
This is mainly based on the work of @Doekin from zxing-cpp#948 with additions from
@raymondqin and some cherry picked improvemtns from @stumpylog's zxing-cpp#1109.

Co-authored-by: Doekin <Leetimemp@gmail.com>
Co-authored-by: Trenton H <797416+stumpylog@users.noreply.github.com>
axxel added a commit to axxel/zxing-cpp that referenced this pull request May 16, 2026
This is mainly based on the work of @Doekin from zxing-cpp#948 with additions from
@raymondqin and some cherry picked improvemtns from @stumpylog's zxing-cpp#1109.

Co-authored-by: Doekin <Leetimemp@gmail.com>
Co-authored-by: Trenton H <797416+stumpylog@users.noreply.github.com>
axxel added a commit to axxel/zxing-cpp that referenced this pull request May 16, 2026
* replace pybind11 with nanobind
 * automatic pyi generation
* replace setuptools with scikit_build_core
* enable stable ABI builds for python 12+ (only one wheel for those)
* drop win32 wheels
* move cibuildwheel config to pyproject.toml

This is mainly based on the work of @Doekin from zxing-cpp#948 with additions from
@raymondqin and some cherry picked improvemtns from @stumpylog's zxing-cpp#1109.

Co-authored-by: Doekin <Leetimemp@gmail.com>
Co-authored-by: Trenton H <797416+stumpylog@users.noreply.github.com>
axxel added a commit that referenced this pull request May 17, 2026
* replace pybind11 with nanobind
 * automatic pyi generation
* replace setuptools with scikit_build_core
* enable stable ABI builds for python 12+ (only one wheel for those)
* drop win32 wheels
* drop Python 3.9 wheels
* move cibuildwheel config to pyproject.toml

This is mainly based on the work of @Doekin from #948 with additions from
@raymondqin and some cherry picked improvemtns from @stumpylog's #1109.

Co-authored-by: Doekin <Leetimemp@gmail.com>
Co-authored-by: Trenton H <797416+stumpylog@users.noreply.github.com>
@axxel

axxel commented May 17, 2026

Copy link
Copy Markdown
Collaborator

Thanks for sharing your work on this. I had a closer look at your PR over last couple days but decided to largely stick with the approach in #948. I had issues with the following aspects of your approach:

  • the ndarray "fast path" is a duplication of code that adds not benefit (according to my understanding)
  • I did not like the raw CPython API usage
  • I don't want to include the auto-generated stub file in the git repo
  • the scikit_build_core setup can be simpler
  • the python package layout in the repo did not convince me after all

If you find any issues or missing optimization opportunities with my pushed version, please let me know.

@axxel axxel closed this May 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants