TDM 30100: Project 11 — 2023

Motivation: In general, scraping data from websites has always been a popular topic in The Data Mine. In addition, it was one of the requested topics. We will continue to use "" to practice scraping skills

Context: This is a second project focusing on web scraping combined with the BeautifulSoup library

Scope: Python, web scraping, selenium, BeautifulSoup

Learning Objectives
  • Use Selenium and XPath expressions to efficiently scrape targeted data.

  • Use BeautifulSoup to scrape data from web pages

Make sure to read about, and use the template found here, and the important information about projects submissions here.

In the previous project, you learned how to get the 'Music' category link in the webpage of "", and how to use Selenium to scrape books' information. The follow is the sample code for the solution for question 1.

import time
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from import By

firefox_options = Options()
# Headless mode means no GUI

driver = webdriver.Firefox(options=firefox_options)

e_t = driver.find_element("xpath",'//article[@class="product_pod"]/h3/a')
e_p = driver.find_element("xpath",'//p[@class="price_color"]')
fst_b_t = e_t.text
fst_b_p =e_p.text

# find book entitled "how music works"
book_link = driver.find_element(By.LINK_TEXT, "How Music Works")

#scrape and print book information : product description, UPC and availability
table = driver.find_element(By.XPATH, "//table[@class='table table-striped']")
upc= table.find_element(By.XPATH, ".//th[text()='UPC']/following-sibling::td[1]")
upc_value = upc.text

availability = table.find_element(By.XPATH, ".//th[text()='Availability']/following-sibling::td[1]")
availability_value = availability.text
In this project we will include BeautifulSoup in our webscraping journey. BeautifulSoup is a python library. You can use it to extract data from HTML or XML files. You may find more BeautifulSoup information here:


Question 1 (2 pts)

  1. Please create a function called "get_category" to extract all categories' names in the website. The function does not need any arguments. The function returns a list of categories' names.

  • Use BeautifulSoup for this question

    from bs4 import BeautifulSoup
  • You can parse the page with BeautifulSoup

    bs = BeautifulSoup(driver.page_source,'html.parser')
  • Review the page source of the website’s homepage, including categories located at the sidebar. The BeautifulSoup "select" method is useful to get names, like this:

categories = [c.text.strip() for c in'.nav-list li a')]

Question 2 (2 pts)

  1. Please create a function called "get_all_books" to get first page books for a given category name from question 1. Use "Music_14" to test the function. The argument is a category name. The function returns a list of book objects with book titles, book price and book availability from the first webpage.

  • Review the page source, you may find one "article" tag holds one book information. You may use find_all to find all "article" tags, like

  • You may create an object to hold the book information, like:

    book = {
  • You may use a loop to go through the books, like

    for article in articles:
        title = article.h3.a.attrs['title']
        price = article.find('p',class_='price_color').text
        availability = article.find('p',class_='instock availability').text
    # create a book object with the extract information
  • You may need a list to hold all book objects, and add all books to it, like

  • You may use different ways to solve the question, like use function "map" etc.

Question 3 (2 pts)

You may have noticed that some categories like "fantasy_19" have more than one page of books.

  1. Please update the function "get_all_books" from question 2 so that the function can be used to get all books, even if there are multiple pages for the category.

  • Look for pagination link "next"

Question 4 (2 pts)

  1. Look through the website "", and pick anything that interests you, and scrape and display those data.

Project 11 Assignment Checklist

  • Jupyter Lab notebook with your code, comments and output for the assignment

    • firstname-lastname-project11.ipynb

  • Submit files through Gradescope

Please make sure to double check that your submission is complete, and contains all of your code and output before submitting. If you are on a spotty internet connection, it is recommended to download your submission after submitting it to make sure what you think you submitted, was what you actually submitted.

In addition, please review our submission guidelines before submitting your project.