TDM 20200: Web Scraping Project 2 — Spring 2025

When scraping data from websites using lxml, we again import requests but this time we also import from lxml like this:

Afterwards, just like in the previous project, we can use requests to scrape the data from a website, such as (for instance) The Data Mine staff directory:

and then we can parse this content using html.fromstring (which comes from lxml) like this:

When selecting the parts of the webpage where we want to extract elements, this time we will use an xpath to the location that we want, in the webpage. The specification for an xpath looks a lot like the way that we specify a location in BeautifulSoup, but there are a few differences. We start the string for the location with // which indicates that we want to look in the entire document. We also put an @ symbol (an ampersand) before each attribute.

We can extract the names of all 22 staff members at once, by selecting the data in p tags with class attribute purdue-home-cta-grid__card-name from the page. Now that we are using lxml instead of Beautiful Soup, we change this statement:

to INSTEAD using this statement:

which gives us the 22 elements we wanted to find. Now, just like we did with Beautiful Soup, we can use a list comprehension to get these 22 names into a list. Our previous statement from Beautiful Soup

will INSTEAD look like this with lxml:

Now that you have the 22 staff members' names in a list, use a similar operation in lxml (instead of Beautiful Soup) to also extract the 22 staff members' job titles.

Finally, make a Pandas data frame with 22 rows and 2 columns, namely, 1 row per staff member, with their name in the left column and their job title in the right column.

As in the previous project, we will extract the 56 names of the states and territories from the National Park Service homepage at www.nps.gov (again) using the fact that their names are given as the text after the a tags with class attribute dropdown-item dropdown-state.

We will (as before) remove the whitespace from the text, using the strip() function.

To get the locations of the webpages devoted to each state and territory, instead of using element['href'] to extract the href attribute from each element, since we are working with lxml, we will use element.attrib['href'] and (as before) we will append the string 'https://www.nps.gov' to the front of each string.

Finally, make a Pandas data frame with 56 rows and 2 columns, namely, 1 row per state or territory, with their name in the left column and a string displaying the URL for that state or territory in the right column.

For instance, the row for Indiana should have 'Indiana' in the left column and 'https://www.nps.gov/state/in/index.htm' in the right column.

Just like in Project 1, Question 3, we would like you to extract from books.toscrape.com/ each of the 50 category types and the 50 URLs corresponding to each of these categories. In this way, you can make a data frame with 50 rows and 2 columns, namely, 1 row per category, with their name in the left column and a string displaying the URL for that category in the right column.

The names of the categories are given in a double set of li tags, and then an li tag, and then an a tag. The names of the categories are the text after the a tags.

Instead of using:

we will use an xpath like this:

Extract the 50 category types as the text after the a tags, and remove the whitespace from the text, using the strip() function. Hint: 'Travel' should be the first category, and 'Crime' should be the last category.

Now that you have these 50 categories, we can get the locations of the webpages devoted to each category, by extracting the href attribute from each tag. If your data is stored in element, then the href attribute can be retrieved as element['href']. Append the string 'https://books.toscrape.com/' to the front of each string.

(As a very minor point for sharp readers: In question 2, we appended 'https://www.nps.gov' without an additional forward slash, because in the NPS website, the slash was already in the href attribute.)

Finally, make a Pandas data frame with 50 rows and 2 columns, namely, 1 row per category, with their name in the left column and a string displaying the URL for that category in the right column.

For instance, the row for Poetry should have 'Poetry' in the left column and 'https://books.toscrape.com/catalogue/category/books/poetry_23/index.html' in the right column.

For academic purposes only now we extract the Snoopy comic for June 22, 1970 from the URL www.gocomics.com/peanuts/1970/06/22 but this time, we use lxml instead of Beautiful Soup.

Instead of using:

we will use an xpath like this:

In this way, using lxml instead of Beautiful Soup, we can extract the URL for the comic, for this particular day (June 22, 1970) that Woodstock got named: assets.amuniversal.com/2181aa70f895013014ff001dd8b71c47

Now use lxml to find and check the location of the Peanuts comic for two additional days, and explain your steps.

Consider the website for Indiana parks from the National Park Service:

www.nps.gov/state/in/index.htm

There are four parks listed on this page.

First, scrape the types of all four parks (National Historical Park, National Park, National Historic Trail, National Memorial).

Next, scrape the names of all four parks (George Rogers Clark, Indiana Dunes, Lewis & Clark, Lincoln Boyhood).

Next, scrape the names of all four locations (Vincennes, IN; Porter, IN; Sixteen States: IA,ID,IL,IN,KS,KY,MO,MT,NE,ND,OH,OR,PA,SD,WA,WV; Lincoln City, IN).

Finally, make a Pandas data frame with 4 rows and 3 columns, namely, 1 row per park, with the type of park in the left column, the name of the park in the middle column, and the location of the park in the right column.