STAT 19000: Project 6 — Fall 2021

Motivation: `tapply` is a powerful function that allows us to group data, and perform calculations on that data in bulk. The "apply suite" of functions provide a fast way of performing operations that would normally require the use of loops. If you have any familiarity with SQL, it `tapply` is very similar to working with the `GROUP BY` clause — you first group your data using some rule, and then perform some operation for each newly created group.

Context: The past couple of projects have studied the use of loops and/or vectorized operations. In this project, we will introduce a function called `tapply` from the "apply suite" of functions in R.

Scope: r, tapply

Learning Objectives
• Demonstrate the ability to use the following functions to solve data-driven problem(s): mean, var, table, cut, paste, rep, seq, sort, order, length, unique, etc.

• Read and write basic (csv) data.

• Explain and demonstrate: positional, named, and logical indexing.

• List the differences between lists, vectors, factors, and data.frames, and when to use each.

• Demonstrate a working knowledge of control flow in r: if/else statements, while loops, etc.

• Demonstrate using tapply to perform calculations on subsets of data.

Make sure to read about, and use the template found here, and the important information about projects submissions here.

Dataset(s)

The following questions will use the following dataset(s):

• `/depot/datamine/data/amazon/tracks.csv`

• `/depot/datamine/data/amazon/tracks.db`

Questions

Question 1

Load the `tracks.csv` file into an R data.frame called `tracks`. Immediately after loading the file, run the following.

``str(tracks)``

What happens?

 The C in CSV is not true for this dataset! You’ll need to take advantage of the `sep` argument of `read.csv` to read in this dataset.

Once you’ve successfully read in the data, re-run the following.

``str(tracks)``
Items to submit
• Code used to solve this problem.

• Output from running the code.

Question 2

Great! `tapply` is a very cool, very powerful function in R.

First, let’s say that we wanted to see what the average `duration` (a column in the `tracks` data.frame) of songs were by each `year` (a column in the `tracks` data.frame). If you think about how you would approach solving this problem, there are a lot of components to keep track of!

• You don’t know ahead of time how many different years are in the dataset.

• You have to associate each sum of `duration` with a specific `year`.

• Etc.

Its a lot of work!

In R, there is a really great library that allows us to run queries on an sqlite database and put the result directly into a dataframe. This would be the SQL and R solution to this problem.

``````library(RSQLite)

con <- dbConnect(SQLite(), dbname = "/depot/datamine/data/amazon/tracks.db")
myDF <- dbGetQuery(con, "SELECT year, AVG(duration) AS average_duration FROM songs GROUP BY year;")

Use `tapply` to solve the same problem! Are your results the same? Print the first 5 results to make sure they are the same.

 `tapply` can take a minute to get the hang of. I like to think about the first argument to `tapply` as the column of data we want to perform an operation on, the second argument to `tapply` as the column of data we want to group by, and the third argument as the operation (as a function, like `sum`, or `median`, or `mean` or `sd`, or `var`, etc.) we want to perform on the data.
Items to submit
• Code used to solve this problem.

• Output from running the code.

Question 3

Plot the results of question (2) with any appropriate plot that will highlight the duration of music by year, sequentially. What patterns do you see, if any?

Items to submit
• Code used to solve this problem.

• Output from running the code.

Question 4

Ha! Thats not so bad! What are the `artist_name` of the artists with the highest median `duration` of songs? Sort the results of the `tapply` function in descending order and print the first 5 results.

 This may take a few minutes to run — this function is doing a lot and there are a lot of artists in this dataset!
Items to submit
• Code used to solve this problem.

• Output from running the code.

Question 5

Explore the dataset and come up with a question you want to answer. Make sure `tapply` would be useful with your investigation, and use `tapply` to calculate something interesting for the dataset. Create one or more graphics as you are working on your question. Write 1-2 sentences reviewing your findings. It could be anything, and your findings do not need to be "good" or "bad", they can be boring (much like a lot of research findings)!

Items to submit
• Question you want to answer.

• Code used to solve this problem.

• Output (including graphic(s)) from running the code.

• 1-2 sentences reviewing your findings.

Question 6 (optional, 0 pts)

Use the following SQL and R code and take a crack at solving a problem (any problem) you want to do with R and SQL. You can use the following code to help. Create a cool graphic with the results!

``````library(RSQLite)

con <- dbConnect(SQLite(), dbname = "/depot/datamine/data/amazon/tracks.db")
myDF <- dbGetQuery(con, "SELECT year, AVG(duration) AS average_duration FROM songs GROUP BY year;")
myDF``````
 Please make sure to double check that your submission is complete, and contains all of your code and output before submitting. If you are on a spotty internet connection, it is recommended to download your submission after submitting it to make sure what you think you submitted, was what you actually submitted.