How to Scrape eBay using Python

Web scraping can allow you to gather millions of data points from websites in just a few hours or even minutes. As such, it’s widely used in various business applications where large volumes of information are required.

eBay is one of the many websites where there’s a lot of valuable information to be found. Ranging from pricing data to conducting market research, both individuals and businesses can scrape eBay for a wide variety of reasons.

To build a dedicated eBay web scraper, Python is usually used. The programming language provides a lot of flexibility and a lot of support for bypassing various anti-scraping measures.

Why Scrape eBay?

eBay is one of the most popular marketplaces and e-commerce websites in the world. There’s billions of listings on the website, so there’s enough information for nearly any e-commerce or peer-to-peer transaction use case.

Many businesses, however, tend to narrow down their eBay scraping goals to just one or two of the major use cases:

• Market research

There are enough listings on the platform to get a good sense of products that are becoming more or less popular. Additionally, it can be used to analyze consumer sentiment and behavior of particular products or brands .

• Competitor analysis

As long as a few competitors exist on eBay, scraping the website becomes a decent way to monitor what other companies are doing.

• Price monitoring

While most businesses focus on creating pricing strategies based on their competitors, a more generalized approach can also be performed, looking at specific categories and how their prices change over time.

• Product availability tracking

A somewhat narrow use case where an individual, researcher, or business could monitor the changes in the availability of specific products or categories. Such data can help predict trends, up-and-coming products, and various other behavioral patterns.

These are just some of the possible use cases for eBay scraping. One of the great benefits of having a dedicated web scraping solution for the platform is that you can pivot to other use cases quite quickly.

Due to the versatility of the platform, having a dedicated web scraping solution becomes valuable even if you need to make changes on the fly.

Available eBay Data Points

eBay scraping use cases rely on knowing which data points are available and possible to analyze. Luckily, the platform is one of the few websites that barely hides any information behind login pages.

So, you can easily gather lots of different data points from every listing without having to do anything difficult:

• Listing title : The title of the item being sold.
• Price : The current price of the item, including any bids if it’s an auction.
• Seller information : Username, feedback score, location, and anything else.
• Number of bids : For auction-style listings, the number of bids placed on the item.
• Listing category : Item category, based on eBay’s structure
• Item condition : Condition of the item (e.g., new, used, refurbished).
• Shipping information : Details about shipping costs, delivery options, and estimated delivery times.

All of these data points are available in eBay listings. But you can also scrape data from seller profiles, eBay search results, the homepage, and any other landing pages. While eBay product data is often the most valuable, there are plenty of other use cases for information available in other parts of the website.

How to Start Scraping eBay Data

We’ll be using Python to start scraping eBay data, so you’ll need to install both the programming language and an IDE. PyCharm Community Edition is a good option as it’s free and has all the necessary features.

Installing and Importing Libraries

Once you have both installed, open the IDE and start a new project. First, we’ll need to install all the libraries that’ll be used for scraping eBay data. Open up the Terminal and type in:

pip install requests beautifulsoup4

Requests will allow us to send HTTP requests to the website, while BeautifulSoup is used for data parsing.

Once all the libraries are installed, import them into your project:

import requests
from bs4 import BeautifulSoup

Scraping eBay Listings

There are two areas where you can get eBay product data – search results and the listings themselves. We’ll start from the former and move on to the latter. Note that you’ll nearly always get more eBay product data from listings, but search results can get you a lot of additional URLs and some basic information as well.

Scraping eBay Search Results

We’ll need to send an HTTP request to load a search results page first and then download the HTML file.

url = "https://www.ebay.com/sch/i.html?_nkw=laptop"
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')

So, we send a GET HTTP request to the defined URL and store the response text into a “soup” object.

Then, we’ll need to analyze the response text. Since there are likely to be many items, we need to create a loop that will iterate through the file.

listings = soup.find_all('li', class_='s-item')
for listing in listings:
    # Find the title
    title_element = listing.find('div', class_='s-item__title')
    if title_element and title_element.span:
        title = title_element.span.text
    else:
        title = 'No title available'

    # Find the price
    price_element = listing.find('span', class_='s-item__price')
    if price_element:
        price = price_element.text
    else:
        price = 'No price available'
    
    print(f'Title: {title}, Price: {price}')

So, we first create a set of all objects that have the name “li” and the class “s-item”, which are both used for item listings in search results within eBay.

We then start the loop and iterate over each item in the list. To find the title, we look for the tags with the name “div” and the specific class that has the title. Since the actual title text is stored in the “span” tag, we have to verify that both the title element and the span exist.

If both are true, then we store the title text in a “title” object. Otherwise, we add the string “No title available” to the same object.

We can follow the same process for any eBay product data stored on the search results page. So, we do the same for pricing data.

Scraping Individual Pages

Usually, web scraping involves collecting URLs from search results and putting them into a list. A loop is used to iterate through the list to gather information from a large volume of individual pages.

Since that involves a lot more coding, we’ll start defining our functions for readability and usability. All of the functions are commented on and follow the same structure, so we’ll only explain a single one.

import requests
from bs4 import BeautifulSoup
import time

# Function to scrape details from an individual product page
def scrape_product_page(url):
    response = requests.get(url)
    product_soup = BeautifulSoup(response.text, 'html.parser')

    # Extract the primary price in EUR
    primary_price_element = product_soup.find('div', class_='x-price-primary')
    primary_price = primary_price_element.find('span', class_='ux-textspans').text.strip() if primary_price_element else 'No primary price available'

    # Extract the approximate price in USD
    approx_price_element = product_soup.find('div', class_='x-price-approx')
    approx_price = approx_price_element.find('span', class_='ux-textspans--BOLD').text.strip() if approx_price_element else 'No approximate price available'

    # Extract the seller's username
    seller_element = product_soup.find('div', class_='x-sellercard-atf__info__about-seller')
    seller_username = seller_element.find('span', class_='ux-textspans--BOLD').text.strip() if seller_element else 'No seller information available'

    # Extract the seller's feedback score
    feedback_element = seller_element.find('li', attrs={'data-testid': 'x-sellercard-atf__about-seller'}) if seller_element else None
    feedback_score = feedback_element.find('span', class_='ux-textspans--SECONDARY').text.strip() if feedback_element else 'No feedback score available'

    # Extract the shipping cost and method
    shipping_element = product_soup.find('div', class_='ux-labels-values__values-content')
    shipping_info = shipping_element.find('span', class_='ux-textspans').text.strip() if shipping_element else 'No shipping information available'
    shipping_method = shipping_element.find_all('span', class_='ux-textspans')[3].text.strip() if shipping_element and len(shipping_element.find_all('span', class_='ux-textspans')) > 3 else 'No shipping method available'

    # Extract the location
    location_element = shipping_element.find_all('span', class_='ux-textspans--SECONDARY')[-1].text.strip() if shipping_element else 'No location information available'

    print(f"Primary Price (EUR): {primary_price}")
    print(f"Approximate Price (USD): {approx_price}")
    print(f"Seller Username: {seller_username}")
    print(f"Seller Feedback Score: {feedback_score}")
    print(f"Shipping Cost: {shipping_info}")
    print(f"Shipping Method: {shipping_method}")
    print(f"Location: {location_element}")
    print("-" * 40)

# Function to scrape eBay search results page
def scrape_search_results(url):
    response = requests.get(url)
    soup = BeautifulSoup(response.text, 'html.parser')

    listings = soup.find_all('li', class_='s-item')
    for listing in listings:
        # Extract title
        title_element = listing.find('div', class_='s-item__title')
        title = title_element.span.text.strip() if title_element and title_element.span else 'No title available'

        # Extract price
        price_element = listing.find('span', class_='s-item__price')
        price = price_element.text.strip() if price_element else 'No price available'

        # Extract product URL
        product_url_element = listing.find('a', class_='s-item__link')
        product_url = product_url_element['href'] if product_url_element else None

        print(f"Search Result Title: {title}")
        print(f"Search Result Price: {price}")

        if product_url:
            print(f"Scraping product page: {product_url}")
            scrape_product_page(product_url)
        else:
            print("No product URL available")

        print("=" * 80)

        # To avoid overloading the server, add a small delay between requests
        time.sleep(2)

# Main execution
if __name__ == "__main__":
    search_url = "https://www.ebay.com/sch/i.html?_nkw=laptop"
    scrape_search_results(search_url)

Scraping each of the data points in the listing is quite simple – we find the associated tag and class name and, then, since it’s all nested in “span”, we use it to extract the value.

Our search result function has now changed to also include product URLs. We’ve avoided creating a list by calling the product page scraping function within the search function, however, it may often be better to perform smaller steps when you’re not testing your web scraping script.

Additionally, it’s recommended to use the “pandas” library to create better structured data. Printing output is great for testing and verifying your code, but if you want to perform analysis, exporting your results to a structured data format is necessary.

Avoiding eBay Blocking

While we added a small sleep timer to our search function to avoid triggering anti-bot systems, for most of the use cases you’ll be web scraping so much that blocks are inevitable. You can optimize your user agents, switch things up by sometimes deploying a browser, and use many other strategies.

All of them, however, will only delay the inevitable – an IP address ban. If you run into such a problem, residential proxies are your only bet. Datacenter proxies can also work, but they’re easier to detect and will likely cause significantly more bans than residential proxies.

Additionally, residential proxies usually have significantly more IP addresses and locations, and are easier to rotate. So, they’re often the preferred choice when web scraping difficult websites. Some providers even offer dedicated eBay proxies that are optimized for the platform.

The Python requests library does have proxy support, so implementing them is quite easy as well. All you need to do is to find a reliable proxy provider that will allow you to keep costs low while giving you enough IP addresses to support your web scraping project.

FAQ