TDM 20200: Web Scraping Project 1 — Spring 2025

When scraping data from websites using Beautiful Soup, we first import requests and BeautifulSoup:

Afterwards, we can use requests to scrape the data from a website, such as (for instance) The Data Mine staff directory:

and then we can parse this content using BeautifulSoup:

Afterwards, we can use select statements to extract elements from the website. For instance, the names of The Data Mine staff members are given as the text after the p tags with class attribute purdue-home-cta-grid__card-name.

We can extract the names of all 22 staff members at once, by selecting the data in these p tags with class attribute purdue-home-cta-grid__card-name from the page, as follows:

With a list comprehension, we can get these 22 names into a list:

Now that you have the 22 staff members' names in a list, use a similar operation to extract the 22 staff members' job titles.

Finally, make a Pandas data frame with 22 rows and 2 columns, namely, 1 row per staff member, with their name in the left column and their job title in the right column.

The National Park Service homepage at www.nps.gov lists 56 states and territories. Their names are given as the text after the a tags with class attribute dropdown-item dropdown-state.

Extract the 56 names of the states and territories, and remove the whitespace from the text, using the strip() function.

Now that you have these 56 names, we can get the locations of the webpages devoted to each state and territory, by extracting the href attribute from each tag. If your data is stored in element, then the href attribute can be retrieved as element['href']. Append the string 'https://www.nps.gov' to the front of each string.

Finally, make a Pandas data frame with 56 rows and 2 columns, namely, 1 row per state or territory, with their name in the left column and a string displaying the URL for that state or territory in the right column.

For instance, the row for Indiana should have 'Indiana' in the left column and 'https://www.nps.gov/state/in/index.htm' in the right column.

The demo website Books To Scrape does not have real prices for books. It is only a demonstration website, located at books.toscrape.com/

This website has numerous categories in the left-hand sidebar. The names of the categories are given in an li tag, then a ul tag, then another li tag, and finally an a tag. The names of the categories are the text after the a tags.

Extract the 50 category types as the text after this combination of li then ul then li then a tags, and remove the whitespace from the text, using the strip() function. Hint: 'Travel' should be the first category, and 'Crime' should be the last category.

Now that you have these 50 categories, we can get the locations of the webpages devoted to each category, by extracting the href attribute from each tag. If your data is stored in element, then the href attribute can be retrieved as element['href']. Append the string 'https://books.toscrape.com/' to the front of each string.

(As a very minor point for sharp readers: In question 2, we appended 'https://www.nps.gov' without an additional forward slash, because in the NPS website, the slash was already in the href attribute.)

Finally, make a Pandas data frame with 50 rows and 2 columns, namely, 1 row per category, with their name in the left column and a string displaying the URL for that category in the right column.

For instance, the row for Poetry should have 'Poetry' in the left column and 'https://books.toscrape.com/catalogue/category/books/poetry_23/index.html' in the right column.

For academic purposes only now we extract a Snoopy comic from the internet. As many students know, Dr Ward loves the Woodstock character from the Peanuts comic strip. Although Woodstock first appeared on March 4, 1966, he was not named until June 22, 1970. We can extract the comic from June 22, 1970, as follows:

Load the comic at this website: www.gocomics.com/peanuts/1970/06/22

In Firefox, right click on the comic (or Control-click on a Mac), and "Inspect" the image in Firefox. If we look into some of the html content for the picture, we will see:

In particular, if we look for an img tag with alt attribute that has value 'Peanuts Comic Strip for June 22, 1970 ' then we can extract the src attribute. Hint: It is necessary to put the space after the year in the string, on this website.

Verify that this URL contains the comic for the day that Woodstock got named: assets.amuniversal.com/2181aa70f895013014ff001dd8b71c47

Now load the Peanuts comic for two other days, and explain your steps. In particular, specify which two other days you explored, and give the location of the comic image for those two days, just like for June 22, 1970, the comic image is located here: assets.amuniversal.com/2181aa70f895013014ff001dd8b71c47

This website www.scrapethissite.com/pages/forms/ has data about hockey teams, which students can use to practice scraping tables.

We can view 100 rows of this data at a time, for instance, as follows: www.scrapethissite.com/pages/forms/?page_num=4&per_page=100 which gives the 4th page of the data. In other words, this page shows rows 301 through 400.

Indeed, there are only 582 rows altogether. By asking for 582 or more rows at a time, in this particular website, we can actually get all 582 rows at once, like this: www.scrapethissite.com/pages/forms/?per_page=600

(This is website dependent! Not every website will allow you to do this.)

Now we can extract the entire table from this website. First we need to import Pandas, and also io from StringIO:

Then, as in the previous two questions, we can extract the contents of the website as follows:

and then we can read the entire table, using StringIO and Pandas, as follows:

which will show rows 0 through 4 and also rows 577 through 581.

(Notice that we did not actually use Beautiful Soup at all, for Question 5 !)