How to Remove Duplicates from a List in Python

You can remove duplicates from a Python using the dict.fromkeys(), which generates a dictionary that removes any duplicate values. You can also convert a list to a set. You must convert the dictionary or set back into a list to see a list whose duplicates have been removed.

When you’re working with a list in Python, there may be a scenario where you want to remove duplicates from the list. For example, say you are a chef who is merging the old and new menus for your restaurant. You may want to remove any duplicates that occur from combining the two menus.

This tutorial will cover using the dict.fromkeys() method converting a list to a set. We’ll discuss how to convert the output from these two methods back into a list so you can interpret your data as a list.

Python Remove Duplicates From a List

You can remove duplicates using a Python set or the dict.fromkeys() method.

The dict.fromkeys() method converts a list into a dictionary. Dictionaries cannot contain duplicate values so a dictionary with only unique values is returned by dict.fromkeys().

Sets, like dictionaries, cannot contain duplicate values. If we convert a list to a set, all the duplicates are removed.

Python Remove Duplicates from List: dict.fromkeys()

Python dictionaries store data in a key/value structure. Dictionaries map keys to values and create a pair that stores data in Python.

Dictionaries in Python cannot include duplicate keys. If we convert our list to a dictionary, it will remove any duplicate values. That’s where the fromkeys() method comes in.

dict.fromkeys() Example

fromkeys() is a built-in function that generates a dictionary from the keys you have specified. Because dictionaries cannot include duplicate keys, the function will remove any duplicate values from our list. Then, we can convert our dictionary back to a list.

The syntax for the fromkeys() method is as follows:

dictionary_name.fromkeys(keys, value)

fromkeys takes two parameters: keys and value. The keys parameter is the iterable, which specifies the keys of the new dictionary you want to create. Our value parameter is optional and holds the value for all keys. We do not need this parameter for this example, so we will ignore it.

Let’s say that we are a chef who is merging the old and new pizza menus into one big menu. The new menu includes a number of new pizzas that we have been working on for months. But, we want to offer the pizzas on the old menu. We do not want duplicates to appear on the menu, because that would confuse customers.

To remove any duplicates that may exist in the new menu, we could use the following code:

new_menu = ['Hawaiian', 'Margherita', 'Mushroom', 'Prosciutto', 'Meat Feast', 'Hawaiian', 'Bacon', 'Black Olive Special', 'Sausage', 'Sausage']

final_new_menu = list(dict.fromkeys(new_menu))

print(final_new_menu)

Our code removes our duplicated elements and returns:

['Hawaiian', 'Margherita', 'Mushroom', 'Prosciutto', 'Meat Feast', 'Bacon', 'Black Olive Special', 'Sausage']

On the first line, we declare a Python variable called new_menu which stores all the pizzas on the new menu. Note that this list contains duplicate values — both Sausage and Hawaiian appear twice in the Python lists.

We use the dict.fromkeys() method to create a dictionary from our new_menu variable. Next, we use list() to convert our data from a dictionary to a Python array, or list. On the final line of code, we print out our revised list.

As you can see, our program has removed the duplicate values from our data, and Sausage and Hawaiian appear once in our new list.

Remove Duplicates from List Python: Sets

The Python set object is a data structure that is both unordered and unindexed. Sets store collections of unique items in Python. Unlike lists, sets cannot store duplicate values.

So, because sets cannot store duplicate items, we can convert our list to a set to remove them. In Python, sets are a list of comma-separated items enclosed within curly brackets. Here’s an example of a set in Python that stores a list of pizzas:

pizzas = ('Margherita', 'Hawaiian', 'Mushroom', 'Meat Feast')

Remove Duplicates Using a Set Example

Let’s take our example from above to illustrate how sets can be used to remove duplicate values from a list.

Say that we are a pizza chef who wants to merge both our old and new menus. But there may be duplicates between these menus, which we will want to remove before creating our final menu. To remove these duplicates using the set approach, we could use the following code:

new_menu = ['Hawaiian', 'Margherita', 'Mushroom', 'Prosciutto', 'Meat Feast', 'Hawaiian', 'Bacon', 'Black Olive Special', 'Sausage', 'Sausage']

final_new_menu = list(set(new_menu))

print(final_new_menu)

Here’s our list after removing duplicate values:

['Mushroom', 'Black Olive Special', 'Meat Feast', 'Bacon', 'Margherita', 'Sausage', 'Prosciutto', 'Hawaiian']

At the start of our program, we declare a list called new_menu which stores all the pizzas on the new menu.

Then, we declare a variable called final_new_menu which stores our revised menu. This variable converts the new_menu variable to a set using set(), then converts the set back to a list using list(). This process removes all the duplicate values in our new_menu variable.

Finally, our program prints the new menu to the console. All the unique elements from the original list object appear. We now have a unique list.

Conclusion

Removing duplicate items from a list is a common operation in Python. There is no official method that should be used to remove duplicates from a list. The most common approaches are to use the dictionary fromkeys() function or convert your data into a set.

In this guide, we discussed how to use the dictionary fromkeys() function and sets to remove duplicates from a list in Python.

"Career Karma entered my life when I needed it most and quickly helped me match with a bootcamp. Two months after graduating, I found my dream job that aligned with my values and goals in life!"

Venus, Software Engineer at Rockbot

Find Your Bootcamp Match

To learn more about Python programming, read our How to Learn Python guide.