TDM 20100: Project 2 — Working with the bash shell

In your $HOME directory, you can store only 25GB of data, but in your $SCRATCH directory, you can store up to 200TB of data.

Your $SCRATCH directory is not intended for long-term storage, and it can be erased by the system administrators at regular points in time. Nonetheless, it can be very helpful for working on data sets that do not need to be stored for a long time.

Your project templates (and all of your Jupyter Lab files) should be stored in your $HOME directory, but it is OK to put temporary data files into your $SCRATCH directory.

Make a new file called myflights.csv in the $SCRATCH directory that has only the first line of the 1987.csv file.

Now take all of the csv data files called 1987.csv through 2008.csv from the /anvil/projects/tdm/data/flights/subset/ directory and add their rows of data, one at a time, to the myflights.csv file. Be sure to not add the headers of these files. To accomplish this, use the grep command with the -h and -v options. (The -h option is used to hide the name of the file in the results, and the -v option is used to avoid any lines of the files that have the word "Year".) To append data to the end of a file, use ">>".

Now check that the resulting file has the correct number of lines.

The original files 1987.csv through 2008.csv have a total of 118914480 lines.

The file myflights.csv has all of these lines, except for the 22 header lines from the 22 respective files, plus it has the header from the 1987.csv file. So it should have a total of 118914480 - 22 + 1 = 118914459 lines.

Note: wc, which stands for word count, is actually capable of much more than simply counting the words in a file! Take a look at some of the below examples, along with this man page, for some ideas about the power of wc. The wc command gives the number of lines, bytes, or characters within a file.

Another note: The du command (which stands for disk usage) measures the total disc space occupied by files and directories. Again, review the man page for du and the below examples, and then move onto the tasks for the final set of tasks for this project.

Sometimes we want to copy files directly. Let’s create a new directory in our $SCRATCH folder and copy all of those files with flight data (1987.csv through 2008.csv) into that directory. Call the directory myfolder. Inside that folder, after those files are copied, build another file (like in Question 1) called myflightsremix.csv. Finally, compare these two files, using cmp $SCRATCH/myflights.csv $SCRATCH/myfolder/myflightsremix.csv (if the files are exactly the same, there should be no output because the files have no differences). Also compare them by running: ls -la $SCRATCH/myflights.csv $SCRATCH/myfolder/myflightsremix.csv which should demonstrate that they are the same size. Check wc $SCRATCH/myflights.csv $SCRATCH/myfolder/myflightsremix.csv to ensure that they have the same number of lines, words, and bytes.

Now go back to the scratch directory and remove this folder and its contents, using: cd $SCRATCH and then rm -r $SCRATCH/myfolder

Copy the files itcont1980.txt through itcont2024.txt from the directory /anvil/projects/tdm/data/election into your $SCRATCH directory. Then create a new directory called mytemporarydirectory in your $SCRATCH directory and move all of these election files into that new directory. Finally, put the content from all of these election files into a new file called myelectiondata.txt. Check the size of this new file using the wc command. When you are finished, it is OK to remove the directory myelectiondata from the $SCRATCH directory.

Extract the Origin and Destination columns from all of the files 1987.csv to 2008.csv in the directory /anvil/projects/tdm/data/flights/subset. Save these origins and destinations into a file called $SCRATCH/myoriginsanddestinations.txt

Then sort this data and save the results to: $SCRATCH/mysortedoriginsanddestinations.txt

Then use the uniq -c command to get the counts corresponding to the number of times that each flight path occurred: $SCRATCH/mycounts.txt Note: you need to sort the file before using uniq -c

Now sort the file again, this time in numerical order, using sort -n and save the results to $SCRATCH/mysortedcounts.txt

Finally display the tail of the file, which contains the 10 most popular flight paths from the years 1987 to 2008 and the number of times that airplanes flew on each of these flight paths.

Use the cut command with the flags -d, -f7 to extract the STORE_R values from this file:

/anvil/projects/tdm/data/8451/The_Complete_Journey_2_Master/5000_transactions.csv

Then use the techniques that you learned in Question 4, to discover how many times that each of the STORE_R values appear in the file.