requests is a Python package used to make web requests over Hypertext Transfer Protocol, or HTTP.
If you’ve used the internet for a while, you’ve probably noticed "http" or "https" at the beginning of most links and websites. HTTP is a protocol for communication between servers and clients. A server would be a fancy computer running in an AWS (Amazon Web Services) warehouse, while a client could be you, surfing the web on your computer!
HTTP has requests and responses. A client, such as your browser, sends a request to a server, and server then returns a response to the client.
The basis of HTTP is methods such as GET, POST, PUT, HEAD, DELETE, PATCH, and OPTIONS. You can read more about those here. The important thing is to realize that the
requests package enables you to easily "use" these methods in Python. For example, the GET method is used to get data from a server:
import requests response = requests.get("https://datamine.purdue.edu/") print(response)
In most instances, you will only use
How do I scrape the HTML from datamine.purdue.edu/?
response = requests.get("https://datamine.purdue.edu/") print(response.text[:500])
<!DOCTYPE html> <html lang="en"> <head> <!-- general meta data Here --> <title>The Data Mine</title> <meta content="The first large-scale living learning community for undergraduates from all majors, focused on Data Science for All." name="description" /> <meta content="Marketing and Media" name="author" /> <meta charset="utf-8" /> <meta content="IE=edge" http-equiv="X-UA-Compatible" /> <meta content="width=device-width, initial-scale=1" name="viewport" /> <lin
Here is the official documentation for the
requests library. Our examples are usually introductory and searching official documentation is a skill you’ll want to take far into the future. Never be afraid to go beyond these pages for help!