How to sum by group in Pyspark

Aggregation of fields is one of the basic necessity for data analysis and data science. Pyspark provide easy ways to do aggregation and calculate metrics. Finding sum value for each group can also be achieved while doing the group by. The function that is helpful for finding the sum value is sum(). The below article explains with the help of an example How to sum by Group in Pyspark.

John has store sales data available for analysis. There are five columns present in the data, Geography (country of store), Department (Industry category of the store), StoreID (Unique ID of each store), Time Period (Month of sales), Revenue (Total Sales for the month). John is looking forward to calculate sum revenue for each stores. As there are 4 months of data available for each store, there will be one sum value out of the four.

Find the Total sales for each store in Pyspark

• Step 1: Firstly, Import all the necessary modules.

import pandas as pd
import findspark
findspark.init()
import pyspark
from pyspark import SparkContext
from pyspark.sql import SQLContext 
sc = SparkContext("local", "App Name")
sql = SQLContext(sc)

• Step 2: Use sum() function along with groupby operation. As we are looking forward to group by each StoreID, “StoreID” works as groupby parameter. The Revenue field contains the sales of each store. To find the sum total value, we will be using “Revenue” for sum total value calculation. For the current example, syntax is:

df1.groupBy("StoreID").agg({'Revenue':'sum'}).show()

Example 2: Calculate Sum value for each Department

• Here we are looking forward to calculate the sum value across each department. So, the field in groupby operation will be “Department”

df1.groupBy("Department").agg({'Revenue':'sum'}).show()

Thus, John is able to calculate value as per his requirement in Pyspark. This kind of extraction can be a requirement in many scenarios and use cases. This example talks about one of the use case.

To get top certifications in Pyspark and build your resume visit here. Additionally, you can read books listed here to build strong knowledge around Pyspark. 

Visit us below for video tutorial: