ENH: Optimize np.unique with adaptive thresholding for hashing vs sorting#31970
Open
rajat315315 wants to merge 1 commit into
Open
ENH: Optimize np.unique with adaptive thresholding for hashing vs sorting#31970rajat315315 wants to merge 1 commit into
np.unique with adaptive thresholding for hashing vs sorting#31970rajat315315 wants to merge 1 commit into
Conversation
…c array performance
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes: #31969
Proposed Changes
This PR implements adaptive thresholding for
np.uniqueto select the most efficient pathway (hashing vs sorting) based on the input dtype and size:StringDType: Always routed to_unique_hashas hashing variable-width strings remains faster than sorting._unique_hashonly if the array size is< 100elements (where Python/C setup overhead of sorting dominates). For larger sizes, it directly uses the SIMD-accelerated sorting-based unique pathway.Benchmarks (1,000,000 elements)
int64(All unique)int64(10 unique)'U15'(Distinct strings)StringDType(1M repeated)Benchmarks (10 elements)
int64Verification
All set operation tests pass:
Output:
117 passed in 0.73sFirst time committer introduction
I use it a lot for data chuking.
AI Disclosure
Used Gemini 3.1 Pro in benchmarking the results.