Describe the enhancement requested
InterpreterPoolExecutor's are to be introduced in 3.14, and backported to 3.13 python/cpython#124548, backport:
At present, pyarrow fails with: "ImportError: module pyarrow.lib does not support loading in subinterpreters"
See following example: TPE and PPE work, IPE does not.
from concurrent.futures import ThreadPoolExecutor
from concurrent.futures import ProcessPoolExecutor
from interpreters_backport.concurrent.futures.interpreter import InterpreterPoolExecutor
def try_tpe():
with ThreadPoolExecutor() as executor:
f = executor.submit(exec, "import pyarrow as pa")
f.result()
def try_ppe():
with ProcessPoolExecutor() as executor:
f = executor.submit(exec, "import pyarrow as pa")
f.result()
def try_ipe():
with InterpreterPoolExecutor() as executor:
f = executor.submit(exec, "import pyarrow as pa")
f.result()
if __name__ == "__main__":
try_tpe()
try_ppe()
try_ipe()Result:
Traceback (most recent call last):
File "c:\gitother\iscratch\.venv\Lib\site-packages\interpreters_backport\concurrent\futures\interpreter.py", line 181, in run
self._exec(script)
~~~~~~~~~~^^^^^^^^
File "c:\gitother\iscratch\.venv\Lib\site-packages\interpreters_backport\concurrent\futures\interpreter.py", line 127, in _exec
raise ExecutionFailed(excinfo)
interpreters_backport.concurrent.futures.interpreter.ExecutionFailed: ImportError: module pyarrow.lib does not support loading in subinterpreters
Uncaught in the interpreter:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "c:\gitother\iscratch\.venv\Lib\site-packages\interpreters_backport\concurrent\futures\interpreter.py", line 111, in _call_pickled
cls._call(fn, args, kwargs, resultsid)
~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\gitother\iscratch\.venv\Lib\site-packages\interpreters_backport\concurrent\futures\interpreter.py", line 100, in _call
res = func(*args or (), **kwargs or {})
File "<string>", line 1, in <module>
File "c:\gitother\iscratch\.venv\Lib\site-packages\pyarrow\__init__.py", line 65, in <module>
import pyarrow.lib as _lib
ImportError: module pyarrow.lib does not support loading in subinterpreters
Tested on pyarrow 18.0.0rc0 on 3.13 amd64 using the https://pypi.org/project/interpreters-pep-734/ backport which backports the InterpreterPoolExecutor to 3.13.
Component(s)
Python
Describe the enhancement requested
InterpreterPoolExecutor's are to be introduced in 3.14, and backported to 3.13 python/cpython#124548, backport:
At present, pyarrow fails with: "ImportError: module pyarrow.lib does not support loading in subinterpreters"
See following example: TPE and PPE work, IPE does not.
Result:
Tested on pyarrow 18.0.0rc0 on 3.13 amd64 using the https://pypi.org/project/interpreters-pep-734/ backport which backports the InterpreterPoolExecutor to 3.13.
Component(s)
Python