Asynchronous Web Scraping With Python: A Practical Guide
PythonLearn how to speed up your data extraction and scale your infrastructure with this asynchronous Python tutorial using asyncio and aiohttp.
Eugenijus Denisov
Key Takeaways
-
Switching to async Python eliminates idle wait times, which significantly speeds up your I/O-bound web scraping tasks.
-
Using semaphores to control concurrency and implementing exponential backoff helps you extract data without overwhelming target servers.
-
Scaling your scrapers requires solid infrastructure, like proxy rotation, structured logging, and error tracking, to keep your data clean and reliable.
Standard, synchronous scraping scripts waste most of their runtime just waiting for servers to respond. At scale, this makes them incredibly slow. By using asyncio and aiohttp to handle requests concurrently, your script moves on to the next URL instead of stopping, which gives a significant speed boost to your tasks.
In this web scraping tutorial, we’ll walk through setting up your environment, building an async scraper, comparing performance benchmarks, and adding production-grade features to your code.
Asynchronous Web Scraping - Quick Overview
Traditional (synchronous) web scraping forces your script to sit idle while each page loads. Asynchronous programming, however, lets a single CPU thread overlap the waiting periods of many requests instead of handling them one at a time.
Using async def and await, you explicitly tell Python where the connection pauses happen. It lets the central event loop swap between ongoing downloads seamlessly.
Asynchronous web scraping shifts how you handle HTTP requests. By no longer blocking on network I/O, you can regularly see scripts run many times faster.
For our stack, we’ll use aiohttp to fetch the pages concurrently and BeautifulSoup to parse the HTML, which will give you a practical foundation for pulling data at scale.
Project Setup
To avoid dependency conflicts later, you need to build a solid foundation first. Make sure you always isolate your project in a virtual environment before installing tools.
# macOS/Linux
mkdir async-scraper && cd async-scraper
python3 -m venv venv
source venv/bin/activate
# Windows
mkdir async-scraper && cd async-scraper
python -m venv venv
venv\Scripts\activate
Now, let’s install the core libraries that will do the heavy lifting.
pip install aiohttp beautifulsoup4 lxml python-dotenv
Asyncio and Aiohttp Basics
Before we get into coding, let’s look at how asynchronous HTTP requests work:
- async def. Turns functions into coroutines that can pause their execution to let other code run.
- await. Signals exactly where the script should yield control back to the event loop (usually placed in front of network calls).
- asyncio.create_task(). Schedules a coroutine to run in the background while your main script sets up other jobs.
- asyncio.run(main()). Starts the event loop and executes everything you just scheduled.
While asyncio handles the orchestration, making the actual connections falls to aiohttp and its powerful ClientSession object. Establishing a new connection from scratch carries a massive performance penalty.
To avoid this, ClientSession holds open a pool of underlying TCP connections using Keep-Alive mechanisms, which allow you to recycle the same sockets for thousands of consecutive requests.
Building an Asynchronous Scraper Step by Step
We pull down data from Books to Scrape because it provides a reliable sandbox environment for web scraping with Python without putting businesses under severe pressure.
Loading URLs From a CSV
Hardcoding links directly into your Python scripts quickly becomes a maintenance nightmare at scale. Instead, managing your targets in a simple urls.csv file lets you load them cleanly using Python's built-in csv module.
import csv
def create_sample_csv(filepath):
urls = [f"https://books.toscrape.com/catalogue/page-{i}.html" for i in range(1, 51)]
with open(filepath, "w", newline="", encoding="utf-8") as f:
writer = csv.writer(f)
writer.writerow(["url"]) # header
writer.writerows([u] for u in urls)
def load_urls(filepath):
with open(filepath, newline="", encoding="utf-8") as f:
reader = csv.DictReader(f) # reads the 'url' column by name
return [row["url"] for row in reader if row.get("url")]
create_sample_csv("urls.csv")
urls = load_urls("urls.csv")
print(urls)
The Main Function and Event Loop
Wrapping your core fetch logic in an async def main() function gives you a clean entry point to manage the active session. To pull data fast while still playing nice with target servers, you should use a semaphore to limit concurrency.
Then, asyncio.gather() runs all the fetching tasks and waits for the results.
import asyncio
import aiohttp
import csv
def load_urls(filepath):
with open(filepath, newline="", encoding="utf-8") as f:
reader = csv.DictReader(f) # reads the 'url' column by name
return [row["url"] for row in reader if row.get("url")]
async def fetch(session, url, semaphore):
async with semaphore: # cap concurrency at 20
try:
async with session.get(url) as response:
response.raise_for_status() # turn 4xx/5xx into an exception
return await response.text()
except aiohttp.ClientError as e:
print(f"Failed to fetch {url}: {e}")
return None
except (aiohttp.ClientError, asyncio.TimeoutError) as e:
print(f"Failed to fetch {url}: {e}")
return None
async def main():
urls = load_urls('urls.csv')
semaphore = asyncio.Semaphore(20)
timeout = aiohttp.ClientTimeout(total=30)
async with aiohttp.ClientSession(timeout=timeout) as session:
tasks = [fetch(session, url, semaphore) for url in urls]
results = await asyncio.gather(*tasks)
print(f"Finished fetching {len(results)} pages.")
if __name__ == '__main__':
asyncio.run(main())
The Fetch Function
Network issues are inevitable at scale. This is an upgrade to the core fetch function above, adding a retry block with exponential backoff and timeouts to handle rate limits and transient errors.
async def fetch(session, url, semaphore, max_retries=3):
async with semaphore:
for attempt in range(max_retries):
try:
async with session.get(url) as response:
# Retry on transient server errors and rate limiting
if response.status in (429, 502, 503, 504):
if attempt < max_retries - 1:
await asyncio.sleep(2 ** attempt)
continue
print(f"Gave up on {url}: status {response.status}")
return None
response.raise_for_status()
return await response.text()
except (aiohttp.ClientError, asyncio.TimeoutError) as e:
if attempt < max_retries - 1:
await asyncio.sleep(2 ** attempt)
continue
print(f"Failed {url}: {e}")
return None
return None # unreachable in practice, but makes the contract explicit
Parsing HTML With BeautifulSoup
Once you have the raw HTML, standard web scraping techniques take over. Here, you need to use BeautifulSoup and CSS selectors to extract the title, price, and stock availability.
from bs4 import BeautifulSoup
def parse_book(html):
if not html:
return None
soup = BeautifulSoup(html, 'lxml')
title_el = soup.select_one('h1')
price_el = soup.select_one('.price_color')
stock_el = soup.select_one('.instock.availability')
return {
'title': title_el.text if title_el else 'Unknown',
'price': price_el.text.strip() if price_el else '0.00',
'stock': stock_el.text.strip() if stock_el else 'Unknown',
}
Saving Results to JSON
Dumping data into a standard JSON file works for small runs, but larger operations favor JSON Lines (JSONL) so you can append records as a stream. If you’re working with massive datasets, writing a separate file per product is recommended to avoid data loss if the script crashes.
import json
def save_json(records, filepath):
records = [r for r in records if r]
with open(filepath, "w", encoding="utf-8") as f:
json.dump(records, f, indent=2, ensure_ascii=False)
print(f"Saved {len(records)} records to {filepath}")
Here’s the full snippet to run in one go:
import asyncio
import aiohttp
import csv
import json
import os
from bs4 import BeautifulSoup
def create_sample_csv(filepath):
"""Generate urls.csv if it doesn't exist, so the script runs first try."""
if os.path.exists(filepath):
return
urls = [f"https://books.toscrape.com/catalogue/page-{i}.html" for i in range(1, 6)]
with open(filepath, "w", newline="", encoding="utf-8") as f:
writer = csv.writer(f)
writer.writerow(["url"])
writer.writerows([u] for u in urls)
print(f"Created {filepath} with {len(urls)} URLs")
def load_urls(filepath):
with open(filepath, newline="", encoding="utf-8") as f:
reader = csv.DictReader(f)
return [row["url"] for row in reader if row.get("url")]
async def fetch(session, url, semaphore, max_retries=3):
async with semaphore:
for attempt in range(max_retries):
try:
async with session.get(url) as response:
if response.status in (429, 502, 503, 504):
if attempt < max_retries - 1:
await asyncio.sleep(2 ** attempt)
continue
print(f"Gave up on {url}: status {response.status}")
return None
response.raise_for_status()
return await response.text()
except (aiohttp.ClientError, asyncio.TimeoutError) as e:
if attempt < max_retries - 1:
await asyncio.sleep(2 ** attempt)
continue
print(f"Failed {url}: {e}")
return None
return None
def parse_books(html):
"""A catalogue page lists many books, so return a list of records."""
if not html:
return []
soup = BeautifulSoup(html, "lxml")
books = []
for article in soup.select("article.product_pod"):
title_el = article.select_one("h3 a")
price_el = article.select_one(".price_color")
stock_el = article.select_one(".instock.availability")
books.append({
"title": title_el["title"] if title_el else "Unknown",
"price": price_el.text.strip() if price_el else "0.00",
"stock": stock_el.text.strip() if stock_el else "Unknown",
})
return books
def save_json(records, filepath):
with open(filepath, "w", encoding="utf-8") as f:
json.dump(records, f, indent=2, ensure_ascii=False)
print(f"Saved {len(records)} records to {filepath}")
async def main():
create_sample_csv("urls.csv")
urls = load_urls("urls.csv")
semaphore = asyncio.Semaphore(20)
timeout = aiohttp.ClientTimeout(total=30)
async with aiohttp.ClientSession(timeout=timeout) as session:
tasks = [fetch(session, url, semaphore) for url in urls]
pages = await asyncio.gather(*tasks)
# Parsing is synchronous/CPU-bound, so it runs outside the async machinery
records = []
for html in pages:
records.extend(parse_books(html))
save_json(records, "books.json")
print(f"Fetched {sum(1 for p in pages if p)} of {len(urls)} pages, "
f"extracted {len(records)} books.")
if __name__ == "__main__":
asyncio.run(main())
Benchmarking - Async vs Sync
Running raw numbers against your own code proves the massive advantage of asynchronous web scraping, especially when compared directly to the linear crawl of traditional scripts.
A standard synchronous approach loops over the list one by one. Even when reusing connections via a session, the script still completely blocks, waiting for the server to generate and send the HTML payload before it can move to the next item.
import asyncio
import time
import requests
import aiohttp
def scraper_sync(urls):
results = []
with requests.Session() as session:
for url in urls:
response = session.get(url, timeout=30)
results.append(response.text)
return results
async def scraper_async(urls, concurrency=20):
semaphore = asyncio.Semaphore(concurrency)
timeout = aiohttp.ClientTimeout(total=30)
async def _fetch(session, url):
async with semaphore:
async with session.get(url) as response:
return await response.text()
async with aiohttp.ClientSession(timeout=timeout) as session:
return await asyncio.gather(*[_fetch(session, url) for url in urls])
if __name__ == "__main__":
urls = [f"https://books.toscrape.com/catalogue/page-{i}.html"
for i in range(1, 51)]
start = time.perf_counter()
scraper_sync(urls)
sync_time = time.perf_counter() - start
start = time.perf_counter()
asyncio.run(scraper_async(urls))
async_time = time.perf_counter() - start
print(f"Sync: {sync_time:.2f}s")
print(f"Async: {async_time:.2f}s ({sync_time / async_time:.1f}x faster)")
Using time.perf_counter() to measure execution time makes the performance gap obvious. A batch of one hundred URLs that takes forty seconds synchronously can finish in just two or three seconds with async code.
Real-world speeds depend on network latency and server response times, but synchronous requests will always bottleneck your pipeline if the scraping is performed on more than a handful of URLs.
Robust Error Handling and Reliability
If you want to leave your crawler unattended, you’ll need to write logic that assumes things will break at the worst possible moment. Designing a production scraper requires you to anticipate a variety of real-world failures:
- DNS resolution failures that completely obscure the target server.
- Connection resets that tear down the socket mid-download.
- Aggressive 429 rate limits when you hit the endpoint too hard.
- Random 5xx errors triggered by overloaded backends.
- Silent layout updates that suddenly alter your CSS selectors.
When these failures hit, capturing the exact context with Python's logging library saves you massive headaches. Configure your logger to include timestamps automatically, and explicitly record the target URL, status code, and error type. Appending failed URLs to a separate text file also ensures you can easily rerun them later.
You can catch almost all transient errors without hammering the server by implementing a three-retry limit with exponential backoff. If you hit a 429 status, however, retrying won’t be enough since pausing the worker to respect the server’s Retry-After header is the only sustainable way to avoid a permanent IP ban.
Advanced Features for Production-Grade Scrapers
To push your crawler beyond basic functionality, you’ll need to take care of some additional aspects. Use these tips to make sure your web scraping is as resilient and efficient as it can be:
- Rotate your User-Agent strings and language headers on every request to blend in with natural browser traffic.
- Route your connections through a proxy network by passing your proxy URL and authentication details into session.get(url, proxy=..., proxy_auth=...).
- Inject artificial pauses between large batches of requests using asyncio.sleep() to simulate human reading patterns.
- Clamp your per-domain concurrency down to anywhere between five and twenty concurrent connections to avoid triggering automated security tripwires.
Once your logic scales up, you might find that the raw mechanics of your web scraping pipeline need more specialized tooling.
HTTPX
This modern client bridges the gap by supporting both sync and async architectures, while offering first-class support for HTTP/2. Developers lean on HTTPX heavily when modern Web Application Firewalls (WAFs) and CDNs aggressively block legacy HTTP/1.1 bot traffic.
Scrapy
If your operation has outgrown basic scripts, you need to turn to Scrapy , which provides a comprehensive asynchronous programming framework that handles queuing, item pipelines, and more features natively. It’s one of the best options for enterprise-level Python scraping.
Crawlee for Python
Crawlee brings a unified API that simplifies switching between basic extractors and full browser automation depending on the situation. It strips away a lot of the boilerplate you normally write for web scraping .
Playwright
Sometimes the data you need is located inside client-side rendering. When you cannot avoid JavaScript execution, Playwright’s headless browser allows you to render the page fully and get the data that raw HTTP requests can’t reach.
FAQ
When should I use asynchronous web scraping instead of threads or processes in Python?
You should use the event loop for I/O-bound operations where the script spends most of its time waiting for server replies. For example, multiprocessing is better suited for heavy tasks like intense data processing, whereas threading comes with excessive OS overhead.
How many concurrent tasks are safe for asynchronous scraping?
There’s no single answer for every operation, but five to twenty concurrent tasks per domain is a safe baseline for ethical asynchronous scraping. Just make sure you respect rate limits at all times and stop immediately if you see status codes appearing that indicate server stress.
Can I mix synchronous and asynchronous code in the same web scraping project?
Yes, you can use synchronous HTTP requests with asynchronous, but you must run blocking operations to prevent them from freezing the event loop. Writing clean asynchronous code means you need to isolate your fast network calls from heavier, synchronous tasks.
Is asynchronous web scraping always faster than synchronous scraping?
It dominates for network tasks, but if you only need to grab two or three pages for a quick test, the development and setup overhead of an async environment usually isn't worth it. However, for anything beyond a handful of HTTP requests, the performance gains compound quickly.
Do I still need to worry about legal and ethical issues when web scraping asynchronously?
Fetching data faster actually amplifies your ethical risks, as you can easily and accidentally perform a DDoS attack on a small server in seconds. You must always review the site's Terms of Service, respect the robots.txt file, and ensure your scrapers never pull sensitive or protected personal information.
