Selecting Columns From PySpark DataFrame

Spread the love

Apache Spark is a popular open-source, distributed computing system used for big data processing and analytics. PySpark is the Python library for Spark and one that is widely used in the field of data science and big data. In PySpark, data is primarily manipulated using a distributed data structure called a DataFrame. A DataFrame is a collection of data distributed across multiple machines, organized into named columns, which provides a higher-level abstraction for manipulating your data.

This article focuses on one fundamental operation in PySpark, specifically within PySpark DataFrames: selecting columns.

Understanding PySpark DataFrames

Before we delve into the mechanics of selecting columns, it’s important to understand the structure we’re dealing with: the DataFrame. In PySpark, a DataFrame is a distributed collection of data organized into named columns. It is conceptually equivalent to a table in a relational database or a data frame in Python (with pandas) or R, but with much more optimization for memory and speed, especially on big data.

Each column in a DataFrame has a name and an associated type (string, integer, float, etc.), and operations to manipulate data are typically done on these columns.

Selecting Columns in PySpark DataFrame

You can select columns from a DataFrame using either DataFrame operations or SQL queries. Here, we will focus on DataFrame operations, particularly the select() function.

The select() function is a transformation operation that is used to select the desired columns from a DataFrame, producing a new DataFrame. Here’s a simple example:

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

# Create a DataFrame
data = [("James", "Smith", "USA", "CA"), ("Michael", "Rose", "USA", "NY"), ("Robert", "Williams", "USA", "CA")]
df = spark.createDataFrame(data, ["FirstName", "LastName", "Country", "State"])

# Select column(s) from the DataFrame"FirstName", "LastName").show()

In this code, we created a PySpark DataFrame and used the select() function to select the “FirstName” and “LastName” columns. The show() action prints the new DataFrame to the console.

Advanced Column Selection Techniques

While the select() function is simple and powerful, there are also more complex column selection techniques that can be employed in PySpark.

Using Column Expressions

Column expressions allow you to perform operations on the data of a column within the select() function. For example, you can add a value to a numeric column, concatenate string columns, or even use a conditional statement within the column selection.

# Add 1 to a numeric column + 1)

# Concatenate two string columns + " " + df.LastName)

# Conditional column selection, df.LastName, when(df.Age > 30, "Old").otherwise("Young"))

These column expressions allow you to create complex column selection and transformation operations in PySpark.

Selecting Nested Columns

PySpark also supports complex data types such as Arrays, Maps, and Structs. In such cases, you might need to select nested columns. You can use dot notation (“.”) to refer to the nested fields. If the column names include dot characters, you should use backticks (“`”) to avoid confusion.

# Create a DataFrame with StructType
from pyspark.sql.types import StructType, StructField, StringType

data = [("James", "Bond", "USA", "NY", {'city': 'New York', 'state': 'NY'}),
        ("Michael", "Rose", "USA", "CA", {'city': 'Los Angeles', 'state': 'CA'})]

schema = StructType([
  StructField("FirstName", StringType(), True),
  StructField("LastName", StringType(), True),
  StructField("Country", StringType(), True),
  StructField("State", StringType(), True),
  StructField("Properties", MapType(StringType(), StringType()), True)

df = spark.createDataFrame(data, schema)

# Select a nested field"").show()

Using selectExpr

The selectExpr() function allows you to run SQL-like expressions within the column selection. It’s a variant of the select() function and accepts SQL expressions in a string format.

df.selectExpr("FirstName as name", "Country as country").show()

Here we’ve selected columns and renamed them using SQL-like syntax in the same line.


Selecting columns is a fundamental operation in PySpark, whether you’re performing simple transformations or preparing your data for complex analytics. By understanding the select() function and its more advanced uses, you can unlock the power of PySpark for big data processing and manipulation. Whether you’re dealing with simple data types or complex nested structures, PySpark provides the tools you need to manipulate and transform your data.

Leave a Reply