4.1. What is Web Scraping?

Web scraping is the technique of extracting data from websites by parsing their HTML content.

Web scraping involves the use of programming techniques to retrieve data from web pages. It typically revolves around accessing the HTML of a web page and then pulling out the specific pieces of information needed. For instance, if you want to gather all links from a webpage, you would fetch its HTML and parse through it to find the all the <a> tags that contain the hyperlinks.

Think of web scraping like harvesting fruit from a tree. Just as you would pick only the ripest fruits from various branches, web scraping allows you to collect only the specific data you want from web pages, such as prices from an eCommerce site or headlines from news articles.

Example with requests + BeautifulSoup

import requests
from bs4 import BeautifulSoup
url = "https://example.com"
html = requests.get(url).text
soup = BeautifulSoup(html, "html.parser")
for item in soup.find_all("a"):
    print(item["href"])

The example code illustrates a simple web scraping operation. First, it imports the necessary libraries: requests for fetching the web page and BeautifulSoup for parsing the HTML. Then, it sends a request to the webpage at 'https://example.com' and retrieves its HTML content. After that, it parses the HTML with BeautifulSoup, allowing it to extract all links from the webpage by searching for all <a> tags and printing out their href attributes.

Imagine you are an investigator looking for clues in a large book. The requests library is like your magnifying glass that helps you view the fine print, while BeautifulSoup is the sharp eye that discerns relevant clues hidden within the text, allowing you to gather important pieces of information without getting lost in unnecessary details.

● Always check the site’s robots.txt. ● Avoid sending too many requests in a short time. ● Never scrape login-protected or copyrighted data without permission.

Engaging in web scraping requires awareness of ethical and legal standards. The robots.txt file of a website indicates which parts of the site can be scraped and which should not be accessed at all. It's important to respect these rules to prevent overloading the server with requests, which can harm the website's performance. Moreover, scraping data that requires login credentials or is copyrighted without proper authorization could lead to legal issues.

Consider web scraping like exploring a museum. Some areas are open for everyone to visit, while others may be off-limits or require special permissions to enter. Just as you respect the museum's rules, a responsible web scraper respects a website's guidelines to maintain trust and legality in their actions.

--