Dropping Duplicate Rows in PySpark DataFrame

Spread the love

In the domain of big data processing, Apache Spark is one of the leading platforms. PySpark is the Python API for Spark, which provides a high-level programming interface for Spark and allows you to write Spark applications using Python. With PySpark, you can work with Resilient Distributed Datasets (RDDs) and DataFrames, distributed collections of data.

One of the common tasks when working with PySpark DataFrames is dropping duplicate rows. Just like a table in a relational database, a DataFrame is a distributed collection of data organized into named columns. When working with DataFrames, duplicate rows can create numerous problems. They can skew the results of your analysis, take up unnecessary storage space, and cause your computations to run slower. Thus, it becomes imperative to handle these duplicate rows effectively.

In this article, we’ll explore how to drop duplicate rows in a PySpark DataFrame, the methods available, and some examples to help you understand better.

Basic Usage of dropDuplicates()

PySpark’s DataFrame API provides a dropDuplicates() method to remove duplicate rows from a DataFrame. Here’s how you can use it:

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

data = [("James", "Smith", "USA"), 
        ("James", "Smith", "USA"), 
        ("Michael", "Rose", "USA"), 
        ("Robert", "Williams", "USA"),
        ("Robert", "Williams", "USA")]

df = spark.createDataFrame(data, ["FirstName", "LastName", "Country"])

df = df.dropDuplicates()

df.show()

In this example, dropDuplicates() is called without any arguments. This will consider all columns in the DataFrame while identifying duplicates.

Selective Dropping of Duplicates

The dropDuplicates() method also accepts column names as arguments, allowing you to choose which columns to consider when identifying duplicate rows. If you pass column names to dropDuplicates(), it will only consider those columns and ignore the others.

Here’s an example:

df = df.dropDuplicates(["FirstName", "LastName"])

df.show()

In this example, dropDuplicates() only considers the “FirstName” and “LastName” columns. This means that two rows with the same first and last names but different countries would be considered duplicates, and one of them would be dropped.

Counting Duplicates

Sometimes, before dropping duplicates, you might want to know how many duplicate rows are there. You can use the groupBy() and count() methods for this. Rows with count > 1 are duplicates.

df.groupBy(df.columns)\
    .count()\
    .where("`count` > 1")\
    .show()

Order of Duplicate Rows

When using dropDuplicates(), if two or more rows are duplicates, the first one will be kept, and the rest will be dropped. The “first” row is based on the order in which the rows appear in the DataFrame.

If you want to control which duplicate row to keep, you can use the orderBy() method to sort the DataFrame first before calling dropDuplicates(). The sort order will then determine which row is kept.

df = df.orderBy("Country").dropDuplicates(["FirstName", "LastName"])

In this example, the DataFrame is sorted by “Country” before dropDuplicates() is called. This means if there are duplicate rows with the same first and last names but different countries, the one with the country that comes first in alphabetical order will be kept.

Conclusion

Handling duplicate rows is a common task when working with PySpark DataFrames. PySpark provides the dropDuplicates() method to make this task easy and efficient. With dropDuplicates(), you can remove all duplicate rows from a DataFrame or selectively consider certain columns when identifying duplicates.

Remember that while dropping duplicates can be necessary to clean up your data, it’s important to understand why duplicates appeared in your data in the first place. It might be due to a data entry error, a problem with your data collection process, or maybe duplicates are valid data in your specific use case. Understanding your data and its context is key to effective data analysis and processing with PySpark.

Leave a Reply