The shortest way to remove duplicates from a Python list while keeping the original order is list(dict.fromkeys(items)), because dictionary keys are unique and keep insertion order since Python 3.7. A loop with a seen set does the same and can deduplicate by a key, while NumPy and pandas have their own tools for large data.
original_list = [5, 1, 2, 4, 2, 3, 1]
unique = list(dict.fromkeys(original_list)) # [5, 1, 2, 4, 3]
unordered = list(set(original_list)) # [1, 2, 3, 4, 5], order lostLet us explore these methods, compare their requirements and performance, and handle unhashable items and case-insensitive text. The examples run on Python 3.14 with NumPy 2.5 and pandas 3.0.
1. Different Methods to Remove Duplicates and Maintain Order
The pure Python methods need hashable elements, which rules out lists and dictionaries as elements, while NumPy and pandas compare values in their own way.
| Method | When to Use | Needs hashable items |
|---|---|---|
| dict.fromkeys() | Simple lists, shortest code | Yes |
| Seen set loop | Deduplicate by a key, such as a lowercased name or an ID | Yes, for the key |
| NumPy np.unique() | Large numerical arrays | No, but elements must be comparable |
| pandas drop_duplicates() | Tabular data and Series | No for most types |
| Loop with a list of seen items | Unhashable items such as lists or dicts | No, but slow for large lists |
2. Seen Set: Removing Duplicates using Iteration
This method keeps a set of the elements it has met. For each element, it checks the set, and if the element is new, adds it to both the set and the result, so the first occurrence wins.
def remove_duplicates_seen(lst):
seen = set()
result = []
for item in lst:
if item not in seen:
seen.add(item)
result.append(item)
return result
result = remove_duplicates_seen(original_list) # [5, 1, 2, 4, 3]The seen set stores the elements themselves, so a list of lists raises TypeError: cannot use ‘list’ as a set element (unhashable type: ‘list’). The advantage of the loop is that it can compare by a key, as section 6 shows.
3. dict.fromkeys() and OrderedDict
The dict.fromkeys() method creates a dictionary with the list elements as keys, which drops repeated values and keeps the first position of each. Before Python 3.7, collections.OrderedDict.fromkeys() was needed for the order guarantee; on current versions it gives the same result with more overhead.
from collections import OrderedDict
with_dict = list(dict.fromkeys(original_list)) # [5, 1, 2, 4, 3]
with_ordered = list(OrderedDict.fromkeys(original_list)) # [5, 1, 2, 4, 3]4. NumPy Array: Removing Duplicates using np.unique()
The np.unique() function returns the unique values sorted, not in their original order. With return_index=True, it also returns the first position of each value, and sorting those positions restores the original order.
import numpy as np
arr = np.array(original_list)
sorted_unique = np.unique(arr) # array([1, 2, 3, 4, 5])
_, idx = np.unique(arr, return_index=True)
ordered_unique = arr[np.sort(idx)] # array([5, 1, 2, 4, 3])5. Pandas: Removing Duplicates from a Series or DataFrame
For tabular data, drop_duplicates() keeps the first occurrence by default and preserves the order. On a DataFrame, the subset argument names the columns that define a duplicate.
import pandas as pd
s = pd.Series(original_list)
unique_series = s.drop_duplicates().tolist() # [5, 1, 2, 4, 3]
df = pd.DataFrame({"email": ["a@x.com", "b@x.com", "a@x.com"], "visit": [1, 2, 3]})
first_visits = df.drop_duplicates(subset="email")["visit"].tolist() # [1, 2]6. Removing Duplicates by a Key
Real data often counts two values as the same even when they differ, such as email addresses in different case or records with the same ID. The seen-set loop compares a key and keeps the original value.
emails = ["Ana@x.com", "bob@x.com", "ana@X.com"]
seen, unique_emails = set(), []
for e in emails:
key = e.lower()
if key not in seen:
seen.add(key)
unique_emails.append(e)
# unique_emails = ['Ana@x.com', 'bob@x.com']For unhashable items such as dictionaries, we turn each one into a hashable key first, for example tuple(sorted(d.items())) when the values are hashable.
rows = [{"id": 1}, {"id": 2}, {"id": 1}]
seen, unique_rows = set(), []
for r in rows:
key = tuple(sorted(r.items()))
if key not in seen:
seen.add(key)
unique_rows.append(r)
# unique_rows = [{'id': 1}, {'id': 2}]7. Performance
Both dict.fromkeys() and the seen-set loop take time proportional to the list length. On a list data of one million random integers with about 1,000 distinct values, list(dict.fromkeys(data)) took about 23 ms and the seen-set loop about 28 ms on Python 3.14, because the first one loops in C. A loop that checks if item not in result on a list compares every item with every value kept so far, so it slows down sharply as the number of distinct values grows.
8. Duplicate Removal FAQs
Keeping the last copy and listing the repeated values need small changes to the same code.
8.1. How do we keep the last occurrence instead of the first?
We reverse the list, deduplicate and reverse again, list(dict.fromkeys(reversed(items)))[::-1]. In pandas, drop_duplicates(keep=”last”) does it in one call.
8.2. How do we find which values were duplicated?
The collections.Counter class counts every value, so [v for v, c in Counter(items).items() if c > 1] lists the repeated ones. See the most frequent elements in a list for more counting options.
9. Conclusion
In everyday code, list(dict.fromkeys(items)) removes duplicates and keeps the order in one line. The seen-set loop handles keys such as lowercased text, NumPy needs return_index to keep the order, and pandas uses drop_duplicates() with an optional subset.
10. References
Happy Learning !!
