bpo-31179: Make dict.copy() up to 5.5 times faster.#3067
Conversation
methane
left a comment
There was a problem hiding this comment.
I commented some personal preferences.
There was a problem hiding this comment.
I prefer old, orig, or source to ms.
And I prefer "declare and initialize at same time" style because we can use C99 syntax.
There was a problem hiding this comment.
I prefer "early return" style without "else" clause.
|
@methane Thanks for the review! Please take a another look, I think I've addressed all of your comments. |
There was a problem hiding this comment.
I meant all local variables can be defined where it's initialized.
for example,
for (Py_ssize_t i = 0; i < keys->dk_nentries; i++) {
PyDictKeyEntry *entry = &ep0[i];
...https://google.github.io/styleguide/cppguide.html#Local_Variables
C++ allows you to declare variables anywhere in a function. We encourage you to declare them in as local a scope as possible, and as close to the first use as possible. This makes it easier for the reader to find the declaration and see what type the variable is and what it was initialized to.
C99 is same to C++ in this case.
methane
left a comment
There was a problem hiding this comment.
LGTM except my personal preference coding style.
There was a problem hiding this comment.
Are these test changes actually related to the code changes? Or are they just test hardening that you thought of because of the new code? If so, the test improvements should go in regardless of whether or not the code changes do, and ideally as a separate commit.
There was a problem hiding this comment.
The one case it slowed down was an empty dict ... is it worth a special path to drop out and just create a new empty dict then?
There was a problem hiding this comment.
Added a fast-path for empty dicts.
There was a problem hiding this comment.
Am I missing some context, or is this refusing to use the fast path for shared-key dicts, when the fast path would still be good for 2/3 arrays, and the other just replaced by an INCREF?
There was a problem hiding this comment.
I don't want to complicate the code for speeding up shared-key dicts copy. Copying a shared dict is a very rare operation.
There was a problem hiding this comment.
Any reason not to at least use presized?
There was a problem hiding this comment.
This change isn't concerned with updating/touching the slow path.
|
|
||
|
|
||
| static PyObject * | ||
| clone_dict(PyDictObject *orig) |
There was a problem hiding this comment.
would you mind to rename the function to "clone_combined_dict()"? It requires ma_values==NULL.
| (2) 'mp' is not a split-dict; and | ||
|
|
||
| (3) if 'mp' is non-compact ('del' operation does not resize dicts), | ||
| do fast-copy only if it has at most 3 non-used keys. |
There was a problem hiding this comment.
Instead of an absolute number, why not using a ratio? 3 unused is may a lot if the dict contains 16 items, but what is the dict contains 2^16 items?
Would it make sense to use a ratio, tolerate a maximum of 10% unused entries? Or 1%?
What does impact performances of dict.set(): the absolute number or the ratio?
If it's the absolute number, just explain that above, something like: "each unused entry make dict.set() slower, so create a compacted dict for optimal performances".
| assert(orig->ma_keys->dk_refcnt == 1); | ||
|
|
||
| Py_ssize_t keys_size = _PyDict_KeysSize(orig->ma_keys); | ||
| PyDictKeysObject *keys = PyObject_MALLOC(keys_size); |
There was a problem hiding this comment.
MALLOC is an an alias, use directly PyObject_Malloc().
| } | ||
|
|
||
| if (PyDict_CheckExact(mp) && | ||
| mp->ma_values == NULL && mp->ma_keys->dk_refcnt == 1 && |
There was a problem hiding this comment.
dk_refcnt==1 test seems redundant, the inner assertion is enough IMHO.
Extract of dictobject.c:
The DictObject can be in one of two forms.
Either:
A combined table:
ma_values == NULL, dk_refcnt == 1.
Values are stored in the me_value field of the PyDictKeysObject.
Or:
A split table:
ma_values != NULL, dk_refcnt >= 1
Values are stored in the ma_values array.
Only string (unicode) keys are allowed.
All dicts sharing same key must have same insertion order.
dk_refcnt can only be != 1 if ma_values is not NULL.
|
When you're done making the requested changes, leave the comment: |
|
The overall change LGTM, but I have a few questions and small change requests. |
|
I have made the requested changes; please review again. |
|
@vstinner I've addressed your comments; I agree using a ratio looks like a more sensible approach. I've set it to 20%. The number is pretty arbitrary but the good news we can always adjust it. |
|
@vstinner Pushed another update; now integer arithmetic is used to compute the ratio as you suggested. |
|
Thanks for the updates @1st1, the new change now LGTM. |
https://bugs.python.org/issue31179