Python Web Scraping: Step-By-Step Guide (2026)

If you want to try out web scraping with Python, there’s no better choice of a programming language to get started with. Due to being an easy-to-use scripting language with excellent library support, it’s one of the most popular choices for web scraping in 2026.

In this article, you’ll learn how to use Python’s requests and Beautiful Soup libraries to scrape web pages. After reading this article, you’ll be able to gather book titles, prices, and build a structured dataset by scraping a simulated online bookstore.

What Is Web Scraping?

Web scraping is a technique of gathering HTML data from a website. It’s usually done in an automated way through the use of bots, which are sometimes called scrapers or crawlers.

Most web scraping tools work by fetching the HTML code of a web page and finding the stuff you need. More advanced tools can also include a browser without a GUI (headless browser) to simulate a real user.

Web scraping can be quite tricky and involves plenty of trial and error to get right. In addition, your scripts can easily be broken by changes in the HTML structure or CSS. For this reason, web scraping should be used only when there isn’t an open API to get the data from.

But it’s a technique that you’ll be happy to know when the necessity arises. It’s frequently used to gather data for market research, data analysis, financial analysis, price monitoring, and more.

Why Is Python the Best Language for Web Scraping?

Most programming languages have libraries that you can use for web scraping: all you need to get started is an HTTP client and a way to parse a website’s HTML.

But Python has an ecosystem like none other. You have access to the minimalistic Requests library for handling HTTP requests and Beautiful Soup , a handy library for parsing HTML and XML documents. There are also more advanced tools like Scrapy and Playwright . All of these tools are battle-tested by many users, and there are plenty of tutorials available online.

In addition, Python is a simple language, which makes it great for writing and prototyping code even if you’re not a programmer by profession.

How to Do Python Web Scraping

This tutorial will show how to use web scraping with Python. First, you’ll learn how to get all the titles and prices of the books on the front page of a bookstore. Then, you’ll use the dataset to extract meaningful pricing information.

This tutorial will scrape Books to Scrape (accessible via books.toscrape.com), a safe sandbox website specifically designed for beginners to practice web scraping without worrying about getting blocked.

Setup

For this tutorial, you need to have Python installed on your computer. If you don’t have Python installed, you can download it from the official website .

You also need to install the requests and beautifulsoup4 libraries with the following commands:

pip install requests
pip install beautifulsoup4

Lastly, create a Python file called scraper.py where you’ll put the contents of your script.

Fetching HTML

Basic Python web scraping consists of two tasks: getting the HTML code of a page and finding the information you need.

To fetch the HTML code of the bookstore's front page, you can use Python's Requests library.

import requests

It provides a function called requests.get() that takes a link to the page, connects to it, and returns the HTTP response.

Even though our target site is a sandbox, many real sites rate-limit requests carrying default library user agent strings. It's good practice to identify your scraping application with a descriptive user agent, which you can set via the headers keyword argument of the requests.get() function.

page = requests.get("https://books.toscrape.com/",
                    headers={'User-agent': 'Sorry, learning Python!'})

You can access the code of the web page with the .content property.

html = page.content

This is the code you should have for now:

import requests 

page = requests.get("https://books.toscrape.com/",
                    headers={'User-agent': 'Sorry, learning Python!'})
html = page.content

After you have fetched the page’s HTML code, you just need to find the book titles inside it. To make it easier, you can use a parsing library like Beautiful Soup.

Parsing HTML

First, add an import statement for Beautiful Soup at the top of the file.

from bs4 import BeautifulSoup

Then, parse the html variable with Beautiful Soup’s "html.parser".

soup = BeautifulSoup(html, "html.parser")

This will return a Python object on which you can call methods such as find() and find_all() to search for specific HTML tags.

But which tags should you be searching for? You can find that out by using the inspect option available in browsers like Google Chrome.

It’s simple to use: open books.toscrape.com, find a book title you want to scrape, then right-click and choose 'Inspect'.

This will open the HTML document at the element you have selected.

Now, you need to find a combination of HTML element tags and classes that uniquely identifies the elements you need.

In the case of this bookstore, every book is wrapped in an <article> tag with the class product_pod.

You can use the find_all function with “article” and an additional “product_pod” argument to capture every book container on the page.

book_tags = soup.find_all("article", "product_pod")

But this gets you the entire article block, which contains prices, images, and buttons. To get just the full book title, you need to look inside the <h3> tag and extract the title attribute from the anchor (<a>) tag. You can do this via a list comprehension:

titles = [book.find("h3").find("a")["title"] for book in book_tags]

In the end, you can print out the result in the console.

print(titles)

This is the full code for now:

import requests
from bs4 import BeautifulSoup

page = requests.get("https://books.toscrape.com/",
                    headers={'User-agent': 'Sorry, learning Python!'})
html = page.content

soup = BeautifulSoup(html, "html.parser")
book_tags = soup.find_all("article", "product_pod")
titles = [book.find("h3").find("a")["title"] for book in book_tags]

print(titles)

It will print out the 20 titles on the first page of the store.

But if you want to gather a valuable dataset, scraping just 20 books will not be enough. In the next section, you’ll learn how to enable the scraper to crawl more than one page.

Scraping Multiple Pages

In this part, you’ll expand your script to scrape the first five pages of the bookstore.

When crawling multiple pages, you need to find the link to the next page. Then you can load it, scrape it, find the link to the next page, and continue doing this until you have all the data you need.

In this store, the "next page" button is located inside a <ul> list with the class pager. The button itself is an <li> tag with the class next. So you can select this button by providing “li” and “next” arguments to soup.find().

After selecting the button, you can get the link by navigating to the anchor tag inside via .find(“a”) and accessing the contents of the href attribute. Because it is a relative link, you’ll need to append it to the base URL.

next_url = soup.find("li", "next").find("a")['href']

To scrape more than one page, you’ll need to rewrite your code a little.

First, you need to import the time library. It provides a sleep function that will pause your Python scraper between page requests. This is necessary not to overload the server of the website you’re scraping and make the requests look more natural.

import time

After that, you’ll need to initialize variables at the beginning of the file.

books_data = []
base_url = "https://books.toscrape.com/catalogue/"
next_page = "https://books.toscrape.com/catalogue/page-1.html"

Then, you need to create a for loop that will run 5 times.

It will fetch the current page, scrape the titles and prices, and append them to our books_data list. It will also find the link to the next page, update the next_page variable, and pause for 2 seconds.

Here is the full loop:

import requests
import time
from bs4 import BeautifulSoup
from urllib.parse import urljoin # Used for urljoin below

books_data = []
base_url = "https://books.toscrape.com/catalogue/"
next_page = "https://books.toscrape.com/catalogue/page-1.html"

for current_page in range(1, 6):
    page = requests.get(next_page, headers={'User-agent': 'Sorry, learning Python!'})
    html = page.content

    soup = BeautifulSoup(html, "html.parser")
    book_tags = soup.find_all("article", "product_pod")
    
    # Extract titles and prices
    for book in book_tags:
        title = book.find("h3").find("a")["title"]
        # The price has a "£" symbol we want to remove so we can save it as a number
        price_text = book.find("p", "price_color").get_text().replace('£', '')
        
        books_data.append({
            "title": title,
            "price": float(price_text)
        })

    # Find the next page link
    next_button = soup.find("li", "next")
    if next_button:
        next_url = next_button.find("a")['href']
        # Handle the URL formatting (the first page link differs slightly from the rest)
        if "catalogue/" in next_url:
            next_page = urljoin(next_page, next_url)
        else:
            next_page = base_url + next_url
    else:
        break # Exit loop if no next page exists

    time.sleep(2)

print(f"Scraped {len(books_data)} books!")

Analyzing the Scraped Book Data

So how can you productively use this dataset? One option is to calculate the average price of the books you just scraped, or filter the dataset to find the most expensive titles.

Because we saved the data as dictionaries with numerical prices, it's quite simple:

# Calculate average price
total_price = sum(book["price"] for book in books_data)
average_price = total_price / len(books_data)
print(f"The average book price is £{average_price:.2f}")

# Find books over £50
expensive_books = [book for book in books_data if book["price"] > 50.00]
print(f"Found {len(expensive_books)} books over £50.")

Saving Scraped Data to a File

While printing to the console is fine for quick tests, a practical project needs a way to store data for later analysis. Exporting your results to a structured format ensures your scraped data is safely stored and ready to use long after your script finishes running.

Exporting to CSV

Python has a built-in csv module that handles tabular data exports.

import csv

# Assuming books_data is our list of dictionaries from the previous step
with open('books_dataset.csv', 'w', newline='', encoding='utf-8') as file:
    writer = csv.writer(file)
    writer.writerow(['Title', 'Price (£)']) # Write headers
    
    for book in books_data:
        writer.writerow([book['title'], book['price']])

Exporting to JSON

JSON is a better option for nested or complex structured data. The standard json module exports Python dictionaries and lists directly to a file.

import json

with open('books_dataset.json', 'w', encoding='utf-8') as file:
    json.dump(books_data, file, indent=4)

A Note on pandas

If you plan to analyze the data immediately, you can load it into a pandas DataFrame. Once loaded, pandas provides quick export methods like df.to_csv() or df.to_excel().

Handling Errors and Getting Blocked

Web scrapers fail when websites change their layouts, servers go down, or anti-bot systems block your requests. You need to handle these errors properly to ensure your script runs reliably.

HTTP requests return status codes that indicate the result. The most common codes encountered during scraping include:

• 200 OK. The request was successful.
• 301 / 302 Redirect. The page moved. The requests library handles these automatically by default.
• 403 Forbidden. The server understood the request but refuses to authorize it. This usually means you have been detected as a bot and blocked.
• 404 Not Found. The URL is incorrect, the query strings are malformed, or the page was deleted.
• 429 Too Many Requests. You are hitting the server too fast. You must slow down and respect the site's rate limits.
• 500 Internal Server Error. The website's server crashed or is struggling. This is on their end, not yours.

Wrap your requests in a try/except block to catch connection issues or HTTP errors without crashing:

import requests
from requests.exceptions import RequestException

url = "https://books.toscrape.com/"

try:
    response = requests.get(url, headers={'User-agent': 'Learning Python'}, timeout=10)
    response.raise_for_status() # Raises an exception for 4xx or 5xx codes
    # Continue with BeautifulSoup...
except RequestException as e:
    print(f"An error occurred: {e}")

Retries With Backoff

When you hit temporary errors like a 429 or 500, it's best to use an exponential backoff approach, meaning your script waits a little longer after each failed attempt. You can rely on the built-in retry features from the requests library (using urllib3 adapters) or a dedicated package like tenacity.

Anti-Block Practices

To reduce the likelihood of getting blocked on real websites, use these tips:

• Rotate user agents. On a sandbox site, an honest custom user agent is fine, but on real sites, a rotating pool of current browser user agent strings avoids the blanket filters many servers apply to default library headers.
• Add delays. Introduce randomized time.sleep() intervals between requests so your scraper looks more like organic users.
• Respect rate limits. Don’t overload the server with thousands of requests within a few seconds.
• Rotate IPs. Use residential proxies to distribute your traffic across different IP addresses.

Scraping JavaScript-Rendered Pages

While books.toscrape.com is a fully static website, many modern sites are not. Tools like requests and BeautifulSoup only fetch the initial HTML payload. If a website uses JavaScript to load content dynamically, static parsers will miss the data because they cannot render JavaScript.

To handle dynamic content and scrape dynamic pages, use a headless browser like Playwright or Selenium to execute JavaScript.

Here is a clean example using modern Selenium to wait for and extract dynamic elements on a hypothetical site:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.options import Options
from selenium.common.exceptions import TimeoutException

# Run in headless mode (no visible window)
options = Options()
options.add_argument('--headless=new')

driver = webdriver.Chrome(options=options)

try:
    driver.get("https://quotes.toscrape.com/js/")

    # Wait up to 10 seconds for JavaScript to render the quotes
    WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.CLASS_NAME, "quote"))
    )

    for quote in driver.find_elements(By.CSS_SELECTOR, "div.quote"):
        text = quote.find_element(By.CSS_SELECTOR, "span.text").text
        author = quote.find_element(By.CSS_SELECTOR, "small.author").text
        print(f"{author}: {text}")

except TimeoutException:
    print("The content did not load within 10 seconds.")
finally:
    driver.quit()

Headless browsers consume significantly more system resources than simple HTTP requests, so reserve them only for pages that strictly require JavaScript execution.

Python Web Scraping With Proxies

For serious web scraping activities, it’s recommended to use a proxy server, which is a server that acts as an intermediary between you and the website.

This is because many web page owners don’t really enjoy their page being scraped. Running a simple script a few times is mostly fine. But if you, for example, scrape a large number of pages every day to create a dataset over time, your traffic will most likely get flagged by the server, and you might encounter obstacles like bans, CAPTCHAs, and more.

A proxy server routes your request through a different IP address, so the target site sees the proxy's address rather than your own. Rotating between proxies distributes your requests across many addresses, which spreads the load instead of concentrating it on a single connection.

While there are some free proxies available out there, the service of those is usually not suitable for any serious web scraping effort. Instead, you should pick a paid proxy—the reward will be worth the small cost of a few dollars.

Here’s how you can add an IPRoyal proxy server to your request.

Purchasing Proxies for Python Web Scraping From IPRoyal

First, you need to have an IPRoyal account. If you don’t have one already, you can register for it on the sign-up page .

For this tutorial, we’ll use residential proxies . Instead of buying or renting access to a single server, you buy a set amount of data and connect through whichever of our intermediaries are available or according to your custom settings. That makes proxy rotation straightforward, since you can use a different intermediary on each connection.

To use the service, you need to purchase some amount of data. For the tutorial, 1 GB is more than enough. To make an order, go to the Residential proxies section, and then choose the “Create new order” option.

After finishing the purchase, you will be able to use the proxy service.

Now, you should get your credentials for the proxy service. You can access them further down in the Residential proxies section. Copy the first link from the cURL example section.

Adding IPRoyal Proxies to Your Python Web Scraping Project

Now, go to your script and create a PROXIES variable that will hold the link to these proxies (your link will differ from the one in the example).

PROXIES = {"http": "http://yourusername:[email protected]:12321",
"https": "http://yourusername:[email protected]:12321"}

After that, you can use this variable in the requests.get() function via a proxies keyword argument.

page = requests.get(next_page,
                        headers={'User-agent': 'Just learning Python, sorry!'},
                        proxies=PROXIES)

On each request, the proxy server will be rotated, so your traffic reaches the site from a different IP address each time.

    

Final Thoughts

In this article, you learned about the basics of Python web scraping in 2026. You used two libraries—requests and BeautifulSoup—to scrape book titles and prices across multiple pages. Then we showed you how to save this data and calculate insights from it.

Scraping a dedicated sandbox like books.toscrape.com is quite easy, since it doesn’t use any techniques to obfuscate the HTML or block traffic. This is rarely the case with massive real-world e-commerce sites, which often generate classes dynamically or block bots.

These two libraries are not the only web scraping libraries in the Python ecosystem. For more advanced Python web scraping projects, they might be too basic. In this case, you might want to look into Scrapy, a full-fledged scraping framework, and Playwright, an automation tool that simulates a real browser. For AI-assisted workflows, Gemini web scraping is worth looking into as a complement to these tools.

FAQ