diff --git a/Doc/extending/embedding.rst b/Doc/extending/embedding.rst index e64db373344038..207c62fcc06e08 100644 --- a/Doc/extending/embedding.rst +++ b/Doc/extending/embedding.rst @@ -275,6 +275,91 @@ will depend on the details of the C++ system used; in general you will need to write the main program in C++, and use the C++ compiler to compile and link your program. There is no need to recompile Python itself using C++. +.. _freezingandembeddingmanually: + +Freezing Modules in Manually Embedded Python +============================================ + +It is possible to use frozen modules in embedded Python if the main module +is also frozen. It seems people are bitten when they don't have their main module +frozen and use ``PyRun_SimpleString`` that contains their main module(s) but +also depend on an frozen module (like the ``__hello__`` module). However if we +wanted to add our own module to the frozen list we would normally do this:: + + #include + #include + #include "mymodule.h" + + static const struct _frozen _PyImport_FrozenModules[] = { + /* importlib */ + {"_frozen_importlib", _Py_M__importlib, (int)sizeof(_Py_M__importlib)}, + {"_frozen_importlib_external", _Py_M__importlib_external, + (int)sizeof(_Py_M__importlib_external)}, + /* mymodule */ + {"mymodule", M_mymodule, (int)sizeof(M_mymodule)}, + {0, 0, 0} /* sentinel */ + }; + + const struct _frozen * PyImport_FrozenModules = _PyImport_FrozenModules; + +.. note:: + The reason why this example includes ``importlib.h`` and + ``importlib_external.h`` is because of the fact that the embedded + interpreter might need them to run properly (load the Python standard + library from an zip file or the sources like normally). Also in order + for your compiler to work you must point it to the Python subdirectory + in an active clone of the cpython repository or source tarball. + However it is not required to manually build the python core if you are + using an clone or source tarball of an version installed on your system. + +As you can see the above code will compile (with an warning on Windows +which will export ``PyImport_FrozenModules`` on the embedded python program). +This is not what we want. And if we were to run it with ``myprogram`` we will +get the following traceback on running the string-based main module(s). + +.. code-block:: py + + Traceback (most recent call last): + File "", line 1, in + ModuleNotFoundError: No module named 'mymodule' + +There is got to be a better way to have what we want but to also make python +aware of our ``mymodule`` being an frozen module. This module could be +anything from an import hook or anything else where you want to freeze +it similar to how ``importlib`` is frozen to support how your code base +currently is. + +The fix to this is to change the C code above to:: + + #include + #include + #include "mymodule.h" + + static const struct _frozen _PyImport_FrozenModules[] = { + /* importlib */ + {"_frozen_importlib", _Py_M__importlib, (int)sizeof(_Py_M__importlib)}, + {"_frozen_importlib_external", _Py_M__importlib_external, + (int)sizeof(_Py_M__importlib_external)}, + /* mymodule */ + {"mymodule", M_mymodule, (int)sizeof(M_mymodule)}, + {0, 0, 0} /* sentinel */ + }; + +And then in the main() C or C++ function in your embedded interpreter add this line:: + + PyImport_FrozenModules = _PyImport_FrozenModules; + +Now your embedded Python should be able to load your frozen modules perfectly fine. + +.. note:: + This logic was borrowed from Programs/_freeze_importlib.c. Also using this method is + the only way to use frozen modules in Windows as well. On further note + Programs/_freeze_importlib.c in the cpython source tree (with minimal changes) could + be changed to act like the normal python freeze tool as the normal freeze tool + currently does not work on Windows so it makes it only possible to freeze to an + header file, make the frozen module list manually, write the embedded python + entry manually and then compile to have an fully featured embedded python interpreter + with your own frozen modules. .. _compiling: diff --git a/Misc/NEWS.d/next/Documentation/2017-12-19-19-57-41.bpo-32353.cHpGHn.rst b/Misc/NEWS.d/next/Documentation/2017-12-19-19-57-41.bpo-32353.cHpGHn.rst new file mode 100644 index 00000000000000..3131725c025b03 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2017-12-19-19-57-41.bpo-32353.cHpGHn.rst @@ -0,0 +1,2 @@ +Added more documentation on manually embedding python with custom frozen +modules.