Skip to content

gh-128002: use per threads tasks linked list in asyncio #128869

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ambv merged 22 commits into from
Feb 6, 2025
Merged

gh-128002: use per threads tasks linked list in asyncio #128869

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
1a39d41
switch to per thread tasks list
kumaraditya303 Jan 11, 2025
28735e7
traverse linked lists of all threads
kumaraditya303 Jan 12, 2025
3052330
cleanup
kumaraditya303 Jan 14, 2025
acef821
add comments
kumaraditya303 Jan 15, 2025
237a089
add a per interp tasks list
kumaraditya303 Jan 16, 2025
c12f271
fixup for regular builds
kumaraditya303 Jan 16, 2025
d36270f
fix missing start world
kumaraditya303 Jan 16, 2025
f863cd5
Merge branch 'main' of https://github.com/python/cpython into per-thr…
kumaraditya303 Jan 21, 2025
d0fbbc2
add interp llist
kumaraditya303 Jan 21, 2025
5f5b95e
add more tests
kumaraditya303 Jan 21, 2025
708b410
fix test
kumaraditya303 Jan 21, 2025
2c1e93e
try fix compilation
kumaraditya303 Jan 21, 2025
c0ce361
remove supp
kumaraditya303 Jan 21, 2025
aa4c2d1
Merge branch 'main' of https://github.com/python/cpython into per-thr…
kumaraditya303 Jan 24, 2025
fd3fc0b
Merge branch 'main' of https://github.com/python/cpython into per-thr…
kumaraditya303 Jan 24, 2025
c45c9d7
fix merge
kumaraditya303 Jan 24, 2025
f0a9e53
Merge branch 'main' into per-thread-tasks
kumaraditya303 Jan 25, 2025
5b33c45
code review
kumaraditya303 Jan 29, 2025
1d73843
cleanup
kumaraditya303 Jan 29, 2025
4014e70
add comment for asyncio_tasks_lock
kumaraditya303 Jan 29, 2025
83aec26
Reword explanation
ambv Feb 6, 2025
e52741f
Merge branch 'main' into per-thread-tasks
ambv Feb 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
cleanup
  • Loading branch information
@kumaraditya303
kumaraditya303 committed Jan 29, 2025
commit 1d7384346a91a20e6535c56c22d7e6e7c5992215
11 changes: 7 additions & 4 deletions Modules/_asynciomodule.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4012,7 +4012,7 @@ add_tasks_llist(struct llist_node *head, PyListObject *tasks)
// if it is concurrently getting deallocated in another thread,
// otherwise it gets added to the list.
if (_Py_TryIncref((PyObject *)task)) {
if (_PyList_AppendTakeRef((PyListObject *)tasks, (PyObject *)task) < 0) {
if (_PyList_AppendTakeRef(tasks, (PyObject *)task) < 0) {
// do not call any escaping calls here while the world is stopped.
return -1;
}
Expand All @@ -4034,16 +4034,19 @@ add_tasks_interp(PyInterpreterState *interp, PyListObject *tasks)
return -1;
}

int ret = 0;
// traverse the task lists of thread states
_Py_FOR_EACH_TSTATE_BEGIN(interp, p) {
_PyThreadStateImpl *ts = (_PyThreadStateImpl *)p;
head = &ts->asyncio_tasks_head;
if (add_tasks_llist(head, tasks) < 0) {
return -1;
ret = -1;
goto exit;
}
}

return 0;
exit:
_Py_FOR_EACH_TSTATE_END(interp);
return ret;
}

/*********************** Module **************************/
Expand Down
3 changes: 2 additions & 1 deletion Python/pystate.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -644,6 +644,7 @@ init_interpreter(PyInterpreterState *interp,
#endif
llist_init(&interp->mem_free_queue.head);
llist_init(&interp->asyncio_tasks_head);
interp->asyncio_tasks_lock = (PyMutex){0};
for (int i = 0; i < _PY_MONITORING_UNGROUPED_EVENTS; i++) {
interp->monitors.tools[i] = 0;
}
Expand Down Expand Up @@ -1702,7 +1703,7 @@ PyThreadState_Clear(PyThreadState *tstate)
Py_CLEAR(((_PyThreadStateImpl *)tstate)->asyncio_running_task);


PyMutex_LockFlags(&tstate->interp->asyncio_tasks_lock, _Py_LOCK_DONT_DETACH);
PyMutex_Lock(&tstate->interp->asyncio_tasks_lock);
// merge any lingering tasks from thread state to interpreter's
// tasks list
llist_concat(&tstate->interp->asyncio_tasks_head,
Expand Down