Skip to content

gh-76785: Add concurrent.futures.InterpreterPoolExecutor #116430

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ericsnowcurrently wants to merge 4 commits into from
Closed

gh-76785: Add concurrent.futures.InterpreterPoolExecutor #116430

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
fd6989e
Simplify subclassing ThreadPoolExecutor.
ericsnowcurrently Mar 1, 2024
9774ffd
Add InterpreterPoolExecutor.
ericsnowcurrently Oct 20, 2023
6a40cd0
Support Interpreter.prepare_main() in InterpreterPoolExecutor.
ericsnowcurrently Mar 1, 2024
c0841e6
Support submitting a script to the executor.
ericsnowcurrently Mar 1, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Support submitting a script to the executor.
  • Loading branch information
@ericsnowcurrently
ericsnowcurrently committed Mar 6, 2024
commit c0841e68fe0ff6072e7c768a89f3e8d1d71cf114
31 changes: 20 additions & 11 deletions Lib/concurrent/futures/interpreter.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,24 +32,33 @@ def initializer(ctx):

@classmethod
def _normalize_task(cls, fn, args, kwargs):
pickled = pickle.dumps((fn, args, kwargs))
def wrapped(ctx):
interp, results = ctx
interp.exec(f'''if True:
import pickle
fn, args, kwargs = pickle.loads({pickled!r})
res = fn(*args, **kwargs)
_results.put(res)
''')
return results.get_nowait()
if isinstance(fn, str):
if args or kwargs:
raise ValueError(f'a script does not take args or kwargs, got {args!r} and {kwargs!r}')
script = fn
def wrapped(ctx):
interp, _ = ctx
interp.exec(script)
return None
else:
pickled = pickle.dumps((fn, args, kwargs))
def wrapped(ctx):
interp, results = ctx
interp.exec(f'''if True:
import pickle
fn, args, kwargs = pickle.loads({pickled!r})
res = fn(*args, **kwargs)
_interp_pool_executor_results.put(res)
''')
return results.get_nowait()
return wrapped, (), {}

@classmethod
def _run_worker(cls, *args):
interp = interpreters.create()
interp.exec('import test.support.interpreters.queues')
results = interpreters.create_queue()
interp.prepare_main(_results=results)
interp.prepare_main(_interp_pool_executor_results=results)
ctx = (interp, results)
try:
_thread._worker(ctx, *args)
Expand Down