How to Remove Duplicate Rows in R

When working with data sets in R, one of the most common tasks you might encounter is the need to remove duplicate rows. Duplicates can introduce noise and inaccuracies in your analysis, which is why it’s crucial to know how to deal with them effectively.

In this article, we’ll explore multiple methods to remove duplicate rows in R, covering the duplicated() function, the unique() function, and the dplyr package. We’ll also discuss the criteria for identifying duplicates.

Table of Contents

1. Understanding Duplicate Rows
2. The duplicated() Function
3. The unique() Function
4. Using dplyr Package
5. Custom Functions for Removing Duplicates
6. Identifying Duplicates Based on Selected Columns
7. Conclusion

1. Understanding Duplicate Rows

Duplicate rows are rows that have the same values across all columns or specific columns. Whether you are working with data frames or tibbles, removing duplicates is a crucial step for data cleaning. Duplicates can occur for various reasons, including data entry errors, merging of data sources, or even by design, depending on how the data was collected.

2. The duplicated( ) Function

One of the easiest ways to identify and remove duplicate rows from a data frame in R is by using the duplicated() function. This function returns a logical vector, where TRUE indicates that the row is a duplicate.

Basic Usage:

Here is an example of using duplicated() to remove duplicate rows in a data frame:

data <- data.frame(
  col1 = c(1, 2, 3, 4, 4),
  col2 = c('a', 'b', 'c', 'd', 'd')
)

# Identify duplicates
duplicates <- duplicated(data)

# Remove duplicates
unique_data <- data[!duplicates, ]

In the code above, !duplicates inverts the logical vector, keeping only the unique rows in the data frame.

Pros and Cons

• Pros: Simple to use, doesn’t require additional packages.
• Cons: Limited in functionality, can only remove duplicates based on all columns.

3. The unique( ) Function

The unique() function can also remove duplicates but works in a slightly different way. It keeps the first unique occurrence of each row and removes subsequent duplicates.

Basic Usage:

unique_data <- unique(data)

Pros and Cons

• Pros: Simple to use, retains the first occurrence of each row.
• Cons: Similar to duplicated(), lacks advanced functionalities.

4. Using dplyr Package

The dplyr package, part of the tidyverse suite, offers more powerful and flexible options for handling duplicates.

Basic Usage:

To remove duplicates, you can use the distinct() function:

library(dplyr)

unique_data <- distinct(data)

Advanced Usage:

You can also specify which columns to consider when identifying duplicates:

unique_data <- distinct(data, col1)

5. Custom Functions for Removing Duplicates

If you have very specific needs for identifying and removing duplicates, you can write your custom functions.

Here’s a simple example:

remove_duplicates <- function(data) {
  duplicated_rows <- duplicated(data)
  return(data[!duplicated_rows, ])
}

6. Identifying Duplicates Based on Selected Columns

In cases where you want to identify duplicates based on selected columns but retain other columns’ values, you can still use dplyr.

unique_data <- data %>%
  group_by(col1) %>%
  filter(row_number() == 1) %>%
  ungroup()

7. Conclusion

Removing duplicates is a common data cleaning task, and R offers multiple ways to handle it. For simple tasks, built-in R functions like duplicated() and unique() are quick and easy to use. For more control and advanced functionalities, packages like dplyr are incredibly useful.