4.6. Web Scraping Basics

Used when data is available on websites but not through APIs.

Web scraping is a technique used to extract information from websites. It becomes particularly useful when the data you need is not accessible through application programming interfaces (APIs). An API is a structured way for software to communicate and retrieve data safely and efficiently. However, some websites only display data visually, requiring the use of web scraping techniques to collect that information directly from the HTML content.

Imagine you are trying to pick fruit from a tree. If the fruit is hanging low enough, you can reach and grab it directly. This is like using an API to get your data. But if the fruit is at the top of a tall tree and you can't reach it, you must find a way to climb or access it differently. This is similar to web scraping, where you must navigate through the website's code to collect your needed information.

Tools: requests, BeautifulSoup

To perform web scraping, we commonly use two Python libraries: 'requests' and 'BeautifulSoup'. The 'requests' library helps us send HTTP requests to a specified URL, enabling us to retrieve the webpage's content. The 'BeautifulSoup' library is then used to parse the retrieved HTML content, making it easier to navigate through and extract specific data, such as text or images.

Think of it as ordering a book online. First, you send a request (like writing the order form) to the bookstore's website. Once they process your order, you receive a package (the website's HTML). You then open the package and read the book (using BeautifulSoup to parse the code) to find the information you are looking for.

from bs4 import BeautifulSoup
import requests

url = 'https://example.com'
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')
titles = soup.find_all('h2')
for t in titles:
    print(t.text)

This code snippet demonstrates the basic process of web scraping using Python. First, we import the necessary libraries. We define a URL from which we want to scrape data and use 'requests.get(url)' to fetch the content of that webpage. The response is then processed by 'BeautifulSoup', which formats the HTML into a navigable structure. The line 'soup.find_all('h2')' searches for all header tags (h2) on the page. Finally, we loop through the found tags and print their text content, effectively listing all h2 headings from the specified page.

Imagine you’re following a recipe book. First, you open the book (fetching the webpage), then you look at every chapter header (h2 tags) to find the sections you want to read. For each chapter, you write down the title (printing the text) so you can reference it later.

Important: Always check the site's robots.txt and terms of use before scraping.

Before scraping any website, it is ethically essential to check the robots.txt file associated with that site. This file tells web crawlers which parts of the website are open for scraping and which parts are not. Additionally, it is vital to review the website's terms of use, as some sites explicitly prohibit scraping. Respecting these guidelines not only maintains good relationships with website owners but also prevents legal issues.

Consider entering a library. There are areas with open access and areas marked as private or restricted. To avoid trouble, you must follow the library's guidelines—this is analogous to checking the robots.txt file before accessing data from a website.

--