How to drop duplicates in Pandas by specific column

Pandas helps in processing data to high extent. Many a times a user may need to drop rows based on specific column value. For example for a specific customer name there can be many rows, and the need would be to keep only one. This article covers how to drop duplicates in Pandas by specific column.

John has customer dataset available at customer ID level. He founds that Customer Bourke Ismay has 3 records with his name. Similarly Ranice Jaye has 2 records stored. He is looking to drop these duplicate values from Dataframe.

Below are the key steps John is working to achieve the results.

• Step 1: Check the shape of the existing Data.

              Customer_data.shape

• Step 2: The function we will use to drop is drop_duplicates. As we need duplicates removal at specific key level, we will use subset parameter. Here we are looking to drop duplicates at column First Name and Last Name level. So, the below code will be used. Keep option is ‘first’ to keep the first value of duplicate rows.

              Customer_data_2 = Customer_data.drop_duplicates(subset=['First Name','Last Name'], keep='first')

• Step 3: Check to see if the new data is looking ok to proceed. Refer head rows, and shape to check the same.

              Customer_data_2.head(10)

               Customer_data_2.shape

John is able to reach to desired results. In case you are looking for dropping dulicates at whole row level, please refer this link.

Thus, John is able to create Dataframe as per his requirement in Python.

To get top certifications in Python and build your resume visit here. Also, you can read books listed here to build strong knowledge around Python.

Visit below to see video tutorial:

 Looking to practice more with this example? Drop us a note, we will email you the Code file: