GroupBy in Pandas Python (UPDATED)

groupby pandas python code

This tutorial covers one of the most used functions while performing data analysis or munging tasks. Let’s explore GroupBy in python pandas with code snippets and examples.

“Group by” operation involves one or more of the following steps:

  • Splitting the data into groups based on some criteria.
  • Applying one or more functions to each group independently.
  • Combining the results into a data frame/data structure.

Table of Contents

  1. GroupBy on Single Column (Summary)
  2. GroupBy on Multiple Columns (NEW: Tuple, NamedAggregation)
    1. Apply Single Functions on Columns
    2. Apply Multiple Functions on Columns
  3. Operations on Groups
    1. View Groups
    2. Select a Group

First, let’s create a DataFrame of having person first_name, degree, nationality, age, and salary.

first_namedegreenationalityagesalary
0SamPhDUSA2590000
1ZivaMBAIndia2980000
2KiaPhDIndia19110000
3RobinMSUSA21110000
4KimMBAIndia33110000
5RamPhDUK3880000
6ZenPhDUSA24130000
7JohnMSUSA26130000
8ArielPhDIndia3290000
9RonPhDUK29120000

If you have noticed, the salary is randomly generated between 80K to 150K having steps (multiple) of 10K using randrange function.

1. GroupBy on Single Column Python Pandas

Let’s find out the average salary of each degree type. GroupBy can be used to summarize degree column can using THREE methods listed below:

degreesalary
0MBA125000.000000
1MS115000.000000
2PhD116666.666667

TIP: The group by output will have an index or multiple indexes (headers) corresponding to your chosen grouping variables. To avoid setting this index, pass “as_index=False” or “.reset_index()” to the groupby operation.

We have seen the application of the mean (average) function above. Pandas already provide a full range of groupby aggregation functions list built into the base Pandas package:

FunctionDescription
countNumber of non-null observations
sumSum of values
meanMean of values
madMean absolute deviation
medianArithmetic median of values
………so on

Pandas Data Aggregation: Find GroupBy Count

Let’s look into the application of the .count() function. Count function is used to counts the occurrences of values in each group. In the case of the degree column, count each type of degree present.

Python Pandas Aggregation GroupBy Count
Python| Pandas GroupBy Count Example
degreesalary
0MBA2
1MS2
2PhD6

2. GroupBy on Multiple Columns Python Pandas

Group By on two or more columns is possible and easy using Pandas.

1. Apply Single Functions on Columns

nationalitydegreesalaryage
0IndiaMBA19000033
1IndiaPhD20000032
2UKPhD20000038
3USAMS24000026
4USAPhD22000025

For Nationality India and degree MBA, the maximum age is 33.

2. Apply Multiple Functions on Columns

Multiple functions can be applied to a single column. In the above example, we can show both the minimum and maximum value of the age column.

Pandas Tuple Aggregations (Recommended):

Introduced in Pandas 0.25.0, Pandas has added new groupby behavior “named aggregation” and tuples, for naming the output columns when applying multiple aggregation functions to specific columns.

nationalitydegreemean_salarymin_agemax_age
0IndiaMBA1100002933
1IndiaPhD1100001932
2UKPhD1350002938
3USAMS1150002126
4USAPhD1150002425

Pandas Named Aggregation:

For clearer naming, Pandas also provides the NamedAggregation named-tuple, which can be used to achieve the same as normal tuples:

This will give the same output as the above table.

3. Rename columns name in Groupby Python Pandas (Automatically)

Sometimes there are a lot of columns in the data frame. So, manually renaming each row while performing group by is a cumbersome task. This step can be automated using str. Let’s check the example below:

age_minage_maxage_mean
degree   
MBA293331
MS212623.5
PhD193827.83333

3. Operations on Groups

1. View Groups

The DataFrame df, seems jumbled. So to have a clearer view, we have two options either sort the data or view by groups. Let’s structure data using groups.

 first_nameagesalary
nationalitydegree   
IndiaMBAZiva29110000
 PhDKia19140000
UKPhDRam38140000
USAMSRobin21130000
 PhDSam2590000

2. Select Group

Using the get_group() function, we can select a single group. For example, degree has 3 groups: MS, MBA and PhD. Let’s select all values falls under MS group.

first_namedegreenationalityagesalary
3RobinMSUSA21130000
7JohnMSUSA26100000

Conclusion

Groupby is also one of the most frequent functions used while data analysis. In this tutorial, we have mastered groupby aggregation and summarization which is really straightforward in python pandas. For official documentation, click here.

Leave a Comment