TDM 20200: Web Scraping Project 5 — Spring 2025

Write a function called myscrapingfunction that works as follows:

It takes a url called myurl as input and returns a list of all of the books from that url’s category.

For instance,

myscrapingfunction('https://books.toscrape.com/catalogue/category/books/travel_2/index.html') should return a list of 11 url’s like this:

and, as another example,

myscrapingfunction('https://books.toscrape.com/catalogue/category/books/childrens_11/index.html') should return a list of 29 url’s like this:

Here is pseudocode for how Dr Ward defined his function myscrapingfunction with input myurl:

import requests

from lxml import html

Make an empty list called mytemplist.

Let mytree be defined for myurl just like in Project 4 (and earlier).

Create mystub from myurl by using a string replace of 'index.html' by '' (in other words, remove 'index.html' from myurl).

Extend mytemplist by the list [mystub + element.attrib['href'] for element in mytree.xpath('//h3/a')] (this basically gets the books from page 1 of the category corresponding to myurl).

Set mycounter to be 1.

While len([element for element in mytree.xpath('//li[@class="next"]/a')]) is strictly positive (in other words, while there are still "next" pages to handle, do the following:

Increment the value of mycounter by 1.

Create a variable mynewurl from concatenating together these four things:

Create mytree from mynewurl

and then extend mytemplist by the list [mystub + element.attrib['href'] for element in mytree.xpath('//h3/a')] (this basically gets the books from the page numbered mycounter of the category corresponding to myurl)

Now the while loop is over, and the function should return mytemplist.

Run the function myscrapingfunction on each element of mylistofurls.

You should get a list and each element should itself be a list of the URLs for the books in the respective categories.

Create an empty list called mybiglist. Then use list comprehension to extend mybiglist for each element in the list from question 2. As a result, mybiglist should have the URLs for all 1000 books at the Books To Scrape website.

Write a function that takes a category from Books To Scrape, and lists all of the individual prices of the books from that category. Do not run a function on the individual book URLs! Instead, modify the two occurrences of mytemplist.extend from Question 1, so that you extract all of the prices from the category pages. To get the price of the books on a category page, search for a p tag with @class="price_color".

For instance, when we run the function on:

books.toscrape.com/catalogue/category/books/childrens_11/index.html

it should print:

or if you run it on:

books.toscrape.com/catalogue/category/books/science_22/index.html

it should print:

Now add the prices of all 1000 books. You will need to remove the British pound symbol from each (you can use the replace function for this). Verify that the total amount of the costs of all of the books from the Books To Scrape website is 35070.35 British pounds altogether.