Web Scraping Wiki: How to Extract Data from Wikipedia with Python

Wikipedia is a massive knowledge source, and for those new to or learning how to scrape, it’s one of the best targets, since the platform’s pages follow a simple HTML structure. Besides that, Wikipedia is also publicly accessible to anyone and covers almost anything you want to know.

This tutorial will show you how to use Python and its three libraries : requests, BeautifulSoup, and pandas. We’ll walk you through inspecting page structure, parsing HTML, extracting tables and text, and saving your scraped results to useful files.

What This Web Scraping Wiki Guide Covers

At a high level, scraping’s essentially an automated process of extracting data from web pages. In simpler terms, web scraping could be compared to copying and pasting, but performed at a much larger scale without you actually having to locate what you want to copy and paste in a separate file or location.

Wikipedia is a great starting point for anyone interested in learning the basics of web scraping because all of its pages use predictable HTML elements, like sortable tables, infoboxes, structured headings, and clean text.

You’ll learn how to scrape Wikipedia and how to understand the Document Object Model (DOM) and its three structures that most browsers use to represent web pages.

Moreover, you’ll also learn about Cascading Style Sheets (CSS) selectors and XPath queries. CSS selectors allow users to target specific elements by class or tag name, while XPath gives a more granular, path-based way to navigate pages.

Planning Your Wikipedia Scraper

Now, before we go into the actual tutorial, it’s important to lay down some ground rules. Namely, you have to define exactly what you want to scrape clearly. Start by picking a specific Wikipedia URL and listing what you want to get from that URL.

For example, a typical scraping project involving Wikipedia could be to scrape the list of the largest companies by revenue page to:

• Extract the main revenue ranking table.
• Get company names, origin, and associated revenue in USD.
• Collect clean data without commas or currency symbols.
• Save the scraped data to a Comma-Separated Values (CSV) file.

So, for this example project, you’d have four distinct goals. In reality, Wikipedia covers a multitude of topics. Every URL you select might require slightly different goals, but they’d all be pretty similar to the provided example.

Legal, Ethical, and Technical Constraints

As it stands today, Wikipedia is a free knowledge library anyone can update. Additionally, the platform’s robots.txt allows most web scraping activities, but you shouldn’t consider this as full access.

It’s necessary to cover Wikipedia’s terms and conditions, which specifically state that all those wishing to scrape the web or run other automated tasks should respect server load. Aggressive scraping can result in rate-limiting and IP blocks.

Residential proxies are commonly used with web scraping tools, particularly for scraping thousands of pages. These proxies can mask your real IP and send HTTP requests through different IP addresses to reduce IP flagging or banning risks.

To steer clear of any kind of trouble, everyone needs to follow these best web scraping practices :

• Limit your requests to 1-2 per second.
• Use a descriptive User-Agent header to make it easier for Wikipedia to identify your web scraper API.
• Set up request slowdown flows when you start getting a 429 Too Many Requests response.
• Include randomized delays between your requests to avoid predictable traffic spikes.

When it comes to using the data you scrape from Wikipedia, all of its HTML content is published under the Creative Commons Attribution-ShareAlike license. This means that you can use scraped data commercially, but you must source it back to Wikipedia.

Inspecting Wikipedia Page Structure

For the purposes of this tutorial, we’ll use Google Chrome throughout the article. Now, open the browser and go to a Wikipedia page you’ve chosen. Right-click on the element you want to extract and select Inspect to open DevTools.

You’ll see a new window open with different sections – hover over any node, and it will be highlighted on the page. That’s essentially how you inspect elements on a page and make sure you target the right elements when web scraping.

Commonly, you’ll see these key elements on most Wikipedia pages:

Setting Up Your Python Environment

After inspecting and selecting the elements you’ll want to scrape, the next step is to use a virtual environment to keep your project dependencies separate from your system Python installation. The main reason for this is to ensure future web scraping projects operate smoothly if you decide to use other libraries.

Here’s how this looks in code:

# macOS / Linux
python3 -m venv venv
source venv/bin/activate

# Windows
python -m venv venv
venv\Scripts\activate

Alternatively, if you use an IDE like PyCharm or VS Code, you can use the terminal there to set up your virtual environment. Once you've done that, you can go ahead and install all required libraries:

pip install requests beautifulsoup4 pandas

Fetching a Wikipedia Page With Requests

Requests handle the HTTP layer. This library sends GET requests to a specified URL, in this case, a Wikipedia page you want to scrape. The destination server, Wikipedia, then sends back a response.

Using a descriptive and realistic User-Agent is so important – it allows Wikipedia to return full responses without issues, as part of the User-Agent is used to tell the server the preferred language, device, browser version, etc. Making HTTP requests from unknown User-Agents may return a malformed response or none at all.

import requests

url = "https://en.wikipedia.org/wiki/List_of_largest_companies_by_revenue"

headers = {
    "User-Agent": "MyWikiScraper/1.0 (https://example.com; [email protected])"
}

response = requests.get(url, headers=headers)
print(response.status_code)

We're using the largest companies by revenue page.

In this step, you need to know the most common HTTP statuses:

• 200 OK: you can continue with parsing.
• 404 Not Found: the page you're attempting to open doesn't exist, skip it and log it.
• 429 Too Many Requests: you're likely rate-limited, retry later.

Also, Wikipedia pages can include non-ASCII characters, symbols, letters, or emojis that aren't part of the standard 128-character English-based ASCII set. If this happens, make sure to use UTF-8 encoding to view all content.

Parsing HTML With BeautifulSoup

After you locate and extract the data you need, it's time for BeautifulSoup . It’s a parsing library that is responsible for turning raw HTML data and XML documents into a recognizable Python object that can be more easily navigated.

Use response.text and "lxml" specifically for parsing data. They're faster and more capable of parsing irregular markups from older Wikipedia pages than Python's html.parser.

from bs4 import BeautifulSoup

soup = BeautifulSoup(response.text, "lxml")

When parsing with BeautifulSoup, you'll work with these methods most:

• soup.find("tag", class_="name"): returns the first matching element.
• soup.find_all("tag"): returns a list of all matching elements.
• element.get_text(strip=True): extracts visible text without whitespace.
• soup.select("table.wikitable"): CSS selector syntax for exact targeting.

For example:

table = soup.select_one("table.wikitable")

headers = [th.get_text(strip=True) for th in table.select("tr th")]

rows = []
for tr in table.select("tr")[1:]:        # skip the header row
    cells = tr.find_all(["td", "th"])
    if not cells:
        continue
    values = [td.get_text(strip=True) for td in cells]
    rows.append(dict(zip(headers, values)))

print(rows[0])

Extracting Tables With Pandas

Pandas is specifically designed to extract, manage, and create structured tables, with pandas.read_html() being the fastest path from HTML to a DataFrame. This line can automatically detect all <table> elements on your selected page.

import pandas as pd
from io import StringIO

tables = pd.read_html(StringIO(response.text))
df = tables[0]  # usually the first table is the one you want
print(df.head())

Once you've successfully extracted tables, it's time to clean the raw results. Use these lines to:

• df.columns = ["Rank", "Company", "Revenue", "Country"]: Rename columns to remove footnote markers and whitespace.
• df.columns = ['_'.join(col).strip() for col in df.columns]: Flatten multi-index headers if Wikipedia uses header spanning.
• $: Convert revenue strings to strip currency signs, commas, and million/billion suffixes.
• df.drop(columns=["Notes"], inplace=True): Drop irrelevant columns.

After cleaning the results, save them to CSV with this line:

df.to_csv("largest_companies.csv", index=False)

Scraping Text, Links, and Images From a Wikipedia Article

Tables are one of the most popular targets when scraping Wikipedia, but they're not the only ones. Some of the other targets you'll encounter will be paragraphs, links, and images.

Extracting Paragraphs

To extract text and paragraphs, you'll need to use the <p> tag in div#mw-content-text containers. You can filter out empty paragraphs and citation noise with this:

import re

paragraphs = soup.select("div#mw-content-text p")
text = [
    re.sub(r'\[\d+\]', '', p.get_text(strip=True))
    for p in paragraphs
    if p.get_text(strip=True)
]

Additionally, the regular expression (re) library is used to strip out citation markers such as [1].

Extracting Links

All internal Wikipedia links use relative paths that start with /wiki/. To scrape links, all you have to do is add the base URL in front of every link you're targeting.

links = []
for a in soup.select("div#mw-content-text a[href^='/wiki/']"):
    links.append("https://en.wikipedia.org" + a["href"])

Extracting Images

Most, if not all, on Wikipedia use protocol-relative URLs (//upload.wikimedia.org/...), so you convert them to full HTTPS URLs before saving. Scope the search to the content container so you don't pick up UI icons, the site logo, or edit sprites:

images = []
for img in soup.select("div#mw-content-text img"):
    src = img.get("src", "")
    if src.startswith("//"):
        src = "https:" + src
    images.append(src)

Storing Mixed Data Types In a Structured Dictionary or JSON File

Finally, you can choose to store mixed data types, like the very ones we just discussed – paragraphs, links, and images – in a single dictionary. To do that, use this line:

import json

data = {"paragraphs": text, "links": links, "images": images}
with open("article_data.json", "w", encoding="utf-8") as f:
    json.dump(data, f, ensure_ascii=False, indent=2)

Handling Multiple Pages

Learning to web scrape is a useful skill, particularly today when data holds such a vital role. However, with every new page you add to your scraping project, the likelihood of issues and failures increases.

Wikipedia has one of the best interconnected article networks, with links back to other articles. This creates a great opportunity to start building URL queues. Use:

• soup.select("div.mCategoryToc a") to collect target URLs from Wikipedia's category pages.
• list(set(urls)) to deduplicate the URL list before looping.
• Consistent extraction logic for every function, so that each page is treated the same.
• time.sleep(1) after every request.

Once that's done, wrap every request in error handling:

import time

for url in urls:
    try:
        response = requests.get(url, headers=headers, timeout=10)
        response.raise_for_status()
        # ... parse and extract
    except requests.exceptions.RequestException as e:
        print(f"Skipping {url}: {e}")
    time.sleep(1)

Storing and Analyzing Scraped Data

After scraping the data you need, store the structured data, as this format is easy to read and works well with Excel and databases. For paragraphs, links, mixed media, and other complex content, use JSON.

# JSON output for mixed / nested data
with open("output.json", "w", encoding="utf-8") as f:
    json.dump(records, f, ensure_ascii=False, indent=2)

Once your desired data is stored, you can use these lines for basic analysis:

• df.describe() for descriptive statistics.
• matplotlib or seaborn for visualizations.
• df.groupby("country")["revenue"].sum() for aggregations.

Common Pitfalls and How to Avoid Them

Note that Wikipedia often restructures articles, changing class names, splitting tables, and removing infoboxes. To keep track of these changes, you could consider setting up alerts to notify you when your scraper returns zero rows.

FAQ