TDM 10100: Project 5 - Manipulating Data

The flights dataset is huge, with files from 1987 to 2023, each with respective subset datasets just to make the data reasonable to work with. The flights data provides numerous opportunities for data exploration with 7,141,922 rows of data from 2006 and 5,411,843 rows from 1997 alone. These subsets contain information about when each flight took place, as well as different factors like how long they took, specifics like flight numbers, and more. There are, of course, empty or messy values, but there is so much data that this does not make too much of an impact for what we will be doing.

There are 29 columns and millions of rows of data. Some of these columns include:

The Ice Cream dataset is on the smaller side, containing just a few hundred rows of data. This dataset contains information about four different ice cream brands' products, with product information that would make purchasing easier, such as the description, how the product was rated, and what ingredients it contains.

Some of the columns (description, ingredients) of this dataset are not particularly useful given our current skillset, and can make looking at rows of this data not so easy. Which makes taking specific columns (like the rating column) and isolating them for work ideal.

The rating column is each ice cream’s customer rating on a scale of 0 - 5.

The flights dataset is much too large to work with all at once, so we are going to be working with the 2006 subset. This dataset contains all of the same columns as the other flights datasets, just only containing the rows for which the year value is 2006. Read in the 2006 flights subset data as myDF.

You will notice a star (see below) at the beginning of the code line for this dataset, as it takes some time to load in Anvil due to its large size:

Create pa_flights containing only the flights that have a destination of the Philadelphia International Airport (PHL).

Remember, when subsetting in R, we use the myDF[row, column] logic. For instance:

Using the flights' origins, use tapply() to find the average AirTime for each flight in pa_flights data. Remember to remove NA values (na.rm=TRUE option in tapply()) so this result is meaningful. This output will show the average air time for each unique flight origin, with a destination of PHL.

Make a second temporary dataframe containing the data for flights ending at PHX. Use tapply() to view the average AirTime across each flight origin.

In question 1, we only used the destination of two selected airports. Here, use the top 20 destinations when comparing each average air time of each origin-and-destination pair.

There is a function names() that will return the names or labels of a search. Use names() and make a new variable top_20 to save the first 20 rows of your sorted table displaying the highest counted number of flight destinations.

We already practiced how %in% works: it checks whether a value is included in another set of values. In this example, we can use %in% to filter myDF so that we only keep the rows where the destination (Dest) is one of the values inside the vector top_20. This will give you a new data frame, containing only the flights going to those top 20 destinations:

In tapply() function, we were also able to work with column/variable pairs. For example, we can show the average AirTime across each Origin and Dest pairs for top 20 destination pair as following:

With the tapply() function, we can also work with pairs of columns (variables). For example, we can calculate the average AirTime for each combination of Origin and Dest. Below is how we can do this for the top 20 destinations:

Read in the ice cream products file as ice_cream. View the table of the rating column. This shows the counts of each rating (from 0 to 5), and is helpful, but there is something else we want to find.

To better understand this column and how each ice cream was received, we could add labels to each range of rating. To decide what ranges to use, view the summary of the dataset. The rating column is broken down by

The cut() function in R is used to divide continuous numeric data into "bins". It converts numeric values into factor levels, making it useful for grouping or categorizing data.

With ice_cream data, use the cut() command to classify the four rating ranges:

Wrap the results from the cut in a table().

In the cut() function, add labels corresponding to the quality of the ice cream. Save this as a new column rating_phrases.

Read in the 1997 flights subset dataset as my_flights. This dataset has the same structure as the 2006 flights data we used earlier, but it only includes flights from the year 1997.

In this data, the DepTime column tracks what time each flight departs. This column doesn’t display time like we would expect, instead using a range from 1 (00:01am) to 2400 (midnight). There are a lot of different values in this column, but it is OK to print out the table(my_flights$DepTime) to view what sort of times peoples' flights depart.

One way to make this data more readable is to add some set ranges using the cut() command. Like before with the ice cream data, this would allow us to analyze the data based on a smaller number of sets rather than each individual time.

Build a cut() function to break the DepTime column up into sections. Display the table of this to get each groups' number of occurrences.

Add a corresponding label to each group from the broken up column (for example: night, early morning, etc.), and save this to a new column depart_times. This can be viewed as a table of each label and their respective count, or we could make a plot.

Create a barplot of depart_times. Make sure to include a title and axis labels that make sense for how you have grouped this data. Customize this plot however else you would like to.

Going back to the original my_flights dataset, look at the ArrTime column. This can relate some to the DepTime column as this new column tells what time each flight arrived rather than when it departed. Using the same breaks and labels as before, create a new plot showing the counts of each arrival time of the flights.

Still within the 1997 flights dataset, there are the columns Month, DayOfWeek, and AirTime. Something interesting we could find is how the day of the week and the month out of the year affect the total air time.

Use subset() to make a saved selection of rows for a specific month. For example:

View the table of the DayOfWeek column for this month (march in our example). As expected, the counts of the flights for each day of this month are all within reasonable range of each other.

Do this again, just on all of the months (together) from the original dataframe.

To view the total AirTime for each DayOfWeek, write a tapply() function. Remember to remove NA values.
You should have 7 value categories.

Using tapply(), we’re able to create a more complicated table, where we choose an x-axis and a y-axis and for what values the cells in the table are being calculated. In this case, use AirTime as the values, and show Month and DayOfWeek on the axes, as following:

Save this table as flight_times, and make a heatmap showing the results.

Create another table showing the total AirTime across the months, for each day of the month. Make a heatmap of this and compare to your first visualization.