TDM 10100: Project 13 — Fall 2026
Motivation: The groupby function can be used to split a data set, and then we can analyze each group of the data. This sounds complicated, but after you experiment with a few examples, this is straightforward and powerful!
Context: To use the groupby function, we need a Pandas data frame. We split the data frame according to the data in one of the columns. Then we can analyze group of the data.
Scope: using the groupby function on a Pandas data frame
Dataset(s)
This project will use the following datasets:
-
/anvil/projects/tdm/data/election/itcont1980.txt -
/anvil/projects/tdm/data/icecream/combined/products.csv -
/anvil/projects/tdm/data/flights/subset/1990.csv -
/anvil/projects/tdm/data/death_records/DeathRecords.csv -
/anvil/projects/tdm/data/beer/reviews_sample.csv -
/anvil/projects/tdm/data/olympics/athlete_events.csv
|
It is OK to use 4 cores in this project, to prevent the kernel from dying when you load your data. If your kernel does die while you are working, it is OK to select Kernel from the menu in Jupyter Lab, and then Restart Kernel from the dropdown under Kernel, and re-run your analysis for that question. |
Three examples of the groupby function:
Example 1 Using the 1980 election data, we can find the amount of money donated in each state.
import pandas as pd
myDF = pd.read_csv("/anvil/projects/tdm/data/election/itcont1980.txt", header=None, sep='|')
myDF.columns = ["CMTE_ID", "AMNDT_IND", "RPT_TP", "TRANSACTION_PGI", "IMAGE_NUM", "TRANSACTION_TP", "ENTITY_TP", "NAME", "CITY", "STATE", "ZIP_CODE", "EMPLOYER", "OCCUPATION", "TRANSACTION_DT", "TRANSACTION_AMT", "OTHER_ID", "TRAN_ID", "FILE_NUM", "MEMO_CD", "MEMO_TEXT", "SUB_ID"]
We group the data according to the state where the donation was made. Then, looking at the transaction amount column, we can find the sum of the amounts of the transactions (in each state).
myDF.groupby('STATE')['TRANSACTION_AMT'].sum()
because there are many states in the output, you can ask Jupyter Lab to show you all of the rows of output (one row per state) as follows:
pd.set_option('display.max_rows', None)
Example 2 Using the ice cream products data, we can find the average rating for each brand.
import pandas as pd
myDF = pd.read_csv("/anvil/projects/tdm/data/icecream/combined/products.csv")
We split the data according to the brand of the ice cream, and then we take the average (also called the mean) of the ratings for each brand.
myDF.groupby('brand')['rating'].mean()
Example 3 Using the 1990 airport data, we can find the average departure delay for flights from each airport.
import pandas as pd
myDF = pd.read_csv("/anvil/projects/tdm/data/flights/subset/1990.csv")
|
When you load the head of the data using
so that you can see all of the columns. |
We split the data according to the Origin airport, in other words, according to where the flight departed. Then we look at the departure delay and find the average (also called the mean) departure delay for each airport.
myDF.groupby('Origin')['DepDelay'].mean()
|
There are so many states in the output that you will not be able to see them all. So, just like we told Jupyter Lab earlier that we did not want a maximum number of columns, we can also tell Jupyter Lab that we do not want a maximum number of rows either:
In this way, we will be able to see the results from all of the states. |
Questions
Question 1 (2 pts)
Using the 1990 airport data from
/anvil/projects/tdm/data/flights/subset/1990.csv
find the average arrival delay for flights arriving to each airport.
|
The arrival delays are in the |
Question 2 (2 pts)
In the death records file:
/anvil/projects/tdm/data/death_records/DeathRecords.csv
Find the mean Age of death for each Sex.
-
Show the mean
Ageof death for eachSex.
Question 3 (2 pts)
In this file of beer reviews /anvil/projects/tdm/data/beer/reviews_sample.csv
please consider the mean beer scores on each date.
Find the three dates on which the mean score is a 5.
|
It is possible to use |
-
Show the three dates on which the mean
scoreis a 5.
Submitting your Work
Once you have completed the questions, save your Jupyter notebook. You can then download the notebook and submit it to Gradescope.
-
firstname_lastname_project13.ipynb
|
You must double check your You will not receive full credit if your |