gh-146452: Fix pickle segfault on concurrent mutation of dict and list#146470
Conversation
|
The set has a better way of doing this, should we go that way? |
| if (dict_size - total == 1) { | ||
| /* gh-146452: Prevent concurrent dict mutation from | ||
| invalidating the borrowed refs from PyDict_Next(). */ | ||
| Py_BEGIN_CRITICAL_SECTION(obj); |
There was a problem hiding this comment.
The critical section should be on the outer loop, not just on a single call according to the PyDict_Next() documentation:
Py_BEGIN_CRITICAL_SECTION(self->dict);
while (PyDict_Next(self->dict, &pos, &key, &value)) {
...
}
Py_END_CRITICAL_SECTION();There was a problem hiding this comment.
@vstinner thanks, there are two ways to do this, either go with the current go-to for wrapping up with the critical section, or make a new batch_dict_impl then wrap it with criticial_section_begin or critical_section_end inside the batch_dict. Tradeoff is adding an extra function, but way cleaner than the go-to approach because of its multiple layers of condition check based returns. I'm making a modification to my PR in a while.
04b34bc to
8f55561
Compare
|
Please avoid |
|
Admitting the mistake. should i open a new pr again or work on it? i couldn't catch the UBsan failure locally. but another run fixed it on remote. |
|
Thanks @overlorde for the PR, and @kumaraditya303 for merging it 🌮🎉.. I'm working now to backport this PR to: 3.14. |
|
GH-149940 is a backport of this pull request to the 3.14 branch. |
…n pickle (python#146470) Co-authored-by: Kumar Aditya <kumaraditya@python.org>
batch_dict_exact()in_pickle.citerates dict items usingPyDict_Next()which returns borrowed references. Without a critical section, a concurrent dict mutation can invalidate the borrowed reference beforePy_INCREF, causing a segfault.The fix wraps
PyDict_Next()+Py_INCREFinPy_BEGIN_CRITICAL_SECTION(obj)to prevent the dict from being mutated while converting borrowed refs to owned refs. Same approach as the existing set iteration path in the same file (line 3656).Crashes on both 3.14t (stock install) and main (with ASan). Reproducer in the linked issue.
test_picklepasses (1000 tests, no regressions)test_free_threading/test_pickle.pythat segfaults without the fix and passes with it