Web Scraping with Python: BeautifulSoup and Requests

Web scraping means fetching web pages with a program and extracting data from their HTML. It is the tool of last resort for data that is published for people to read but not offered as a download or an API: product listings, public notices, event schedules. Python makes the mechanics easy with two libraries, Requests to fetch and Beautiful Soup to parse. The harder part is doing it responsibly, and that comes first.

Before you scrape anything

• Look for an API or a download. Many sites publish their data officially. An API is more stable, faster and explicitly allowed; see building REST APIs with Python for how APIs are designed.
• Read the site's terms of use. Some prohibit automated access. Scraping personal data raises data-protection obligations regardless of whether the data is visible.
• Check robots.txt. It tells automated clients which paths they may fetch; the rules are standardised in RFC 9309 (RFC 9309).
• Be gentle. Identify your client in the User-Agent header, pause between requests, cache pages you have already fetched, and never run requests in parallel against someone else's site without permission.

The examples here use Quotes to Scrape, a site published specifically for practising scraping.

Parsing HTML with Beautiful Soup

Beautiful Soup turns HTML into a tree you can search. CSS selectors, the same syntax stylesheets use, are the most readable way to find things (Beautiful Soup documentation). Here it parses a fragment of a shop's price list held in a string, so the example runs offline:

from bs4 import BeautifulSoup

html = """
<table id="prices">
  <tr><th>Product</th><th>Price</th><th>Stock</th></tr>
  <tr><td>A5 notebook</td><td>120</td><td class="stock">In stock</td></tr>
  <tr><td>Desk stapler</td><td>450</td><td class="stock">Sold out</td></tr>
</table>
<a class="next" href="/catalogue/page-2.html">next</a>
"""
soup = BeautifulSoup(html, "html.parser")

rows = soup.select("#prices tr")[1:]                  # skip the header row
products = [
    {"product": r.select("td")[0].get_text(strip=True),
     "price": int(r.select("td")[1].get_text(strip=True)),
     "stock": r.select_one("td.stock").get_text(strip=True)}
    for r in rows
]
print(products)
print(soup.select_one("a.next")["href"])

[{'product': 'A5 notebook', 'price': 120, 'stock': 'In stock'}, {'product': 'Desk stapler', 'price': 450, 'stock': 'Sold out'}]
/catalogue/page-2.html

select returns every match and select_one the first. get_text(strip=True) returns the text with surrounding whitespace removed, and element attributes are read like dictionary keys, as with ["href"]. Everything arrives as a string, so convert numbers and dates as you extract them. The products and prices are sample data.

Fetching pages with Requests

Use a Session: it keeps headers and cookies across requests and reuses connections, and always pass a timeout, because a request without one can hang indefinitely (Requests: advanced usage).

import requests
from urllib.robotparser import RobotFileParser

session = requests.Session()
session.headers["User-Agent"] = "glorytecks-tutorial/1.0 (learning example)"

robots = RobotFileParser("https://quotes.toscrape.com/robots.txt")
robots.read()
print("allowed:", robots.can_fetch(session.headers["User-Agent"], "https://quotes.toscrape.com/page/2/"))

response = session.get("https://quotes.toscrape.com/", timeout=10)
response.raise_for_status()
page = BeautifulSoup(response.text, "html.parser")

quotes = [
    {"author": q.select_one("small.author").get_text(strip=True),
     "tags": [t.get_text() for t in q.select("a.tag")]}
    for q in page.select("div.quote")
]
print(len(quotes), quotes[0])
print(page.select_one("li.next > a")["href"])

allowed: True
10 {'author': 'Albert Einstein', 'tags': ['change', 'deep-thoughts', 'thinking', 'world']}
/page/2/

raise_for_status() turns 4xx and 5xx responses into exceptions, so an error page is never parsed as if it were data. RobotFileParser answers the robots.txt question in code (Python docs: robotparser). This practice site has no robots.txt at all; a missing file (a 404) means no restrictions under the standard, which is why the answer is True. On a real site, check the answer before each new path you crawl.

Following pagination

Listing pages usually link to the next page. A loop follows those links until there are none, pausing between requests:

• Build the full URL with urllib.parse.urljoin(current_url, href), because links are often relative, like /page/2/.
• Stop when the "next" link disappears, and also after a maximum number of pages, so a site change cannot send your loop on forever.
• Sleep a second or more between requests, and store each page's raw HTML before parsing, so a parsing bug can be fixed without fetching everything again.

Storing and cleaning what you collect

Separate fetching from parsing. Save each page's HTML to disk with the URL and the time it was fetched, then parse from those files. When the parser needs fixing, and it will, you rerun it on the saved pages instead of fetching the site again. Clean values as you extract them: strip whitespace, convert prices and dates to real types, and record where each value came from. Write the results to CSV or JSON for a one-off job, or to a database table keyed by URL for a recurring one, so a rerun updates rows instead of duplicating them.

A checklist before you run a scraper

1. An official API or download does not exist, or does not cover what you need.
2. The terms of use allow automated access, and robots.txt allows the paths you will fetch.
3. The User-Agent identifies you, every request has a timeout, and there is a pause between requests.
4. There is a hard limit on the number of pages, and fetched pages are cached.
5. Missing fields raise an error or a warning instead of silently producing empty data.

When scraping breaks, and alternatives

Scrapers depend on a site's HTML structure, which changes without notice. Keep selectors in one place, check that every expected field was found, and fail loudly when a page stops matching instead of quietly producing empty results. Pages that build their content with JavaScript show little in the raw HTML; before reaching for a browser-automation tool such as Playwright, open the browser's developer tools and look at the network requests: the data often comes from a JSON endpoint you can call directly, which is simpler and lighter on the site.

Scraped data usually ends up in a file or a database; working with JSON in Python and reading and writing CSV files cover saving it. Working with REST APIs, making requests and web scraping with Beautiful Soup form the APIs and web scraping module of the Python Programming course.

Sources

• Requests: advanced usage — Requests documentation
• Beautiful Soup documentation — Beautiful Soup documentation
• urllib.robotparser — Parser for robots.txt — Python documentation
• RFC 9309: Robots Exclusion Protocol — RFC Editor
• Quotes to Scrape — toscrape.com