
To compute the summary statistics of a column in PySpark, we can use the describe() method. This method takes numeric columns and calculate the count, mean, standard deviation, min, and max of these columns.
Read a Dataset –
Let’s read a dataset to illustrate it. We will use the clothing store sales data.
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
df = spark.read.format('csv') \
.options(header='true', inferSchema='true') \
.load('../data/clothing_store_sales.csv')
df.show(5)

Compute Summary Statistics with describe() method –
To calculate the summary statistics, we can call the describe() method on the dataframe like this –
df.describe().show()
Let’s only select the numeric columns and then compute the summary statistics.
# select only numeric columns
df_numeric = df.select('Customer','Items','Net Sales','Age')
df_numeric.describe().show()

Related Posts –
- Count Number of Rows in a Column or DataFrame in PySpark
- How to Compute the Mean of a Column in PySpark?
- How to Compute Standard Deviation in PySpark?
- Compute Minimum and Maximum value of a Column in PySpark