Introduction
Web scraping is the skill that turns the internet into your database. This tutorial builds a polite, working scraper with BeautifulSoup: fetch a page, parse the HTML, extract structured data (headlines, links, prices), follow pagination, and export everything to CSV. The same request-and-parse loop powers the weather app — except there the API returned JSON, and here you parse HTML yourself.
The polite part matters: respecting robots.txt, rate limiting, and identifying your scraper are what separate a tool from a nuisance.
Features
?page=N links automatically.Prerequisites
pip install requests beautifulsoup4 pandasStep 1: Create the Script
Save as scraper.py — this example scrapes quote authors and texts from quotes.toscrape.com (a site built for scraping practice):
import requests
from bs4 import BeautifulSoup
import pandas as pd
import time
HEADERS = {"User-Agent": "MyLearningScraper/1.0 (contact: you@example.com)"}
def scrape_page(url):
response = requests.get(url, headers=HEADERS, timeout=10)
response.raise_for_status()
soup = BeautifulSoup(response.text, "html.parser")
results = []
for quote in soup.select("div.quote"):
results.append({
"text": quote.select_one("span.text").get_text(strip=True),
"author": quote.select_one("small.author").get_text(strip=True),
"tags": ",".join(t.get_text() for t in quote.select("a.tag")),
})
next_link = soup.select_one("li.next a")
next_url = "https://quotes.toscrape.com" + next_link["href"] if next_link else None
return results, next_url
def scrape_all(start_url, max_pages=5):
all_rows, url, page = [], start_url, 1
while url and page <= max_pages:
print(f"Scraping page {page}: {url}")
try:
rows, url = scrape_page(url)
all_rows.extend(rows)
except requests.RequestException as e:
print(f" Skipping ({e})")
break
page += 1
time.sleep(1) # be polite
return all_rows
if __name__ == "__main__":
data = scrape_all("https://quotes.toscrape.com/")
df = pd.DataFrame(data)
df.to_csv("quotes.csv", index=False)
print(f"Saved {len(df)} rows to quotes.csv")
print(df.head())Step 2: Run the Scraper
python scraper.pyWatch it page through the site, then open quotes.csv — structured data extracted from raw HTML.
How It Works
BeautifulSoup turns raw HTML into a navigable tree, and soup.select() queries it with CSS selectors — the same syntax you use in stylesheets. div.quote finds containers; span.text finds the headline inside each container. The two-level pattern (select containers, then extract fields per container) is the fundamental scraping loop, and it maps directly to how you'd read the page in browser DevTools: right-click an element, Copy selector, adapt.
Pagination is recursion-lite: each page's HTML contains a li.next a link to the following page. Following it until None walks the whole site — with max_pages as a safety brake, because real sites have surprising link structures.
The politeness layer has three parts: a custom User-Agent that identifies the scraper (many sites block default python-requests), time.sleep(1) between requests so you don't hammer the server, and try/except so a timeout on page 14 doesn't discard pages 1–13.
Common Errors & Fixes
select_one with a check, as the code does per-container.response.content (bytes) to BeautifulSoup and let it detect encoding, rather than response.text.Key Concepts
What to Try Next
urllib.robotparser before each domain.FAQ
Is web scraping legal?
Scraping public data is generally legal in many jurisdictions, but terms of service, copyright, and personal-data laws (GDPR) create real exceptions. Check robots.txt, avoid personal data, and prefer official APIs whenever one exists.
When should I use an API instead?
Always when one exists — APIs are stable, documented, and sanctioned. Scraping is for when the data exists only in HTML.
How do I scrape sites that need login?
Use requests.Session() to maintain cookies through the login POST, then scrape authenticated pages. Only do this where the terms permit it.