How to Use ChatGPT for Web Scraping

Writing even a simple web scraping script takes an annoyingly long time, especially if you’re new to it. You need to master not only basic Python, but also libraries like Beautiful Soup and Requests . More challenging web scraping tasks take even longer as you must navigate dynamic content, pagination, anti-bot measures, and other hurdles.

The availability of Large Language Models (LLMs) is making web scraping accessible to everyone, and ChatGPT is leading the pack. While web scraping with ChatGPT isn’t as straightforward as asking a question, you can generate Python scripts that will do the job, at least to some extent.

What Is ChatGPT?

ChatGPT is a highly advanced AI chatbot created by OpenAI that uses machine learning to process natural language and generate human-like responses.

As such, it can perform many tasks at a near-human level. While it cannot extract data from a simple command, it can answer relevant questions, generate full-length articles, analyze pictures, and even write code.

All of these functions simplify the web scraping process, but since it’s not a dedicated web scraping tool, you must learn to use the prompt-response pattern common to LLMs. You write a prompt, and ChatGPT gives you an answer. Since it stores your previous messages, you can build and update the scraping logic conversationally.

Why Use ChatGPT for Web Scraping?

Web scraping with ChatGPT can significantly simplify your workflow. Instead of relying on documentation, manual coding, and guesswork, you can leverage the AI chatbot as an assistant. While much of the work is still manual, various ChatGPT web scraping techniques help with troubleshooting and generally refining your approach.

• Code optimization assistance. Create first iteration Python scripts with complete parsing logic much faster than writing the whole code manually.
• Debugging. Get help identifying errors, fixing broken links, selectors, or any other unexpected issues.
• Streamlined workflow. ChatGPT can suggest better approaches, cleaner code logic, or additional tools.
• Accessibility. It’s easy to learn scraping basics in practice with ChatGPT for beginners, while advanced users still find use in LLMs for researching websites.

How to Write Web Scraping Code With ChatGPT?

ChatGPT can write code for almost anything, including scripts that extract data from various sources. Unfortunately, your ChatGPT web scraper is unlikely to work out of the box, so you’ll need to refine the code and adjust it to your specific target.

ChatGPT-generated code for extracting data does not match crucial parts of the website’s code, such as CSS selectors. Most sites change their code or obfuscate elements, requiring manual inspection to build a working ChatGPT web scraper. If you ask it to write a web scraper by itself, two outcomes are most likely.

• Since it has been trained on many web scraping tutorials , it might find some information about your target website and compile a statistically probable script. If the website hasn’t changed its structure, the scraper might work.
• ChatGPT will hallucinate and create the HTML code that is likely for your target website, but actually isn’t. The resulting code will look convincing and have sound logic, but won’t actually extract data.

We need to provide ChatGPT with more detailed instructions before it writes any code. Here are the most important details to put in a prompt for a ChatGPT web scraping task:

• A link to the website you’re using.
• The kind of data that you want to scrape.
• XPath or CSS selectors that you can use to scrape this data.

After the first version of code is generated, you’ll need to test and refine it. With some back and forth, the code will work. However, the success of your ChatGPT web scraper depends on your target.

Practice targets, like the website Books to Scrape , might work on the first try. Here’s how your prompting ChatGPT for such a task might look:

I need to write a web scraper.
Here's the link to the website: http://books.toscrape.com/
I need to scrape the titles and the prices of the books and print them out in the console.
Each book title is in an h3 tag. The price of the book is in a p element with a price_color class.
Please write the code using Python. Add the necessary installs separately as "pip install".

Here’s the code that ChatGPT generates:

import requests
from bs4 import BeautifulSoup

# Website URL
url = "http://books.toscrape.com/"

# Send GET request
response = requests.get(url)

# Parse HTML
soup = BeautifulSoup(response.text, "html.parser")

# Find all book containers
books = soup.find_all("article", class_="product_pod")

# Loop through books and print title + price
for book in books:
    # Book title is inside the h3 tag
    title = book.h3.a["title"]

    # Book price is inside p with class "price_color"
    price = book.find("p", class_="price_color").text

    print(f"Title: {title}")
    print(f"Price: {price}")
    print("-" * 40)

ChatGPT Vision Capabilities for Scraping

To scrape a real-world website, you'll need to determine the relevant XPath elements or CSS selectors. It can be done manually with most browsers. Here's how to do it using Google Chrome:

• Open the target web page in your browser.
• Right-click the element you want.
• Click Inspect.
• The element you chose should be highlighted in DevTools.
• From there, you can copy the XPath or CSS selectors.

However, some websites obfuscate elements needed for web scraping and try to restrict the use of DevTools. In such cases, and when you're scraping hundreds of page types or sites, it might be easier to use the vision capabilities of ChatGPT for finding relevant CSS or XPath elements.

Simply take a screenshot of your target website and upload it to ChatGPT for analysis with a request to identify elements and propose possible selectors. An example prompt might look as follows:

I want to scrape data from this webpage. Based on this screenshot, identify the main data elements (e.g., titles, prices, links) and suggest possible CSS selectors or XPath expressions I could use. Assume a typical HTML structure and explain your reasoning briefly.

Note that ChatGPT won’t replace using DevTools yourself. It can only help quickly identify likely patterns or starting points, which is most useful for large-scale projects.

Improving Code Together With ChatGPT

Since ChatGPT remembers the context of the conversation, it’s easy to upgrade the code it has written by giving it additional suggestions and requests for improvement.

One of the most important additions to successfully extract data is to use proxy servers. By acting as intermediaries, routing your requests, proxies hide your original IP address with their own. As such, your access is less likely to be blocked by automated bot detection systems.

IPRoyal's residential proxies are some of the best tools for web scraping. It's recommended to use automatically rotating proxies. The IPs will switch for every request or set period without manual configuration. They can be added to your ChatGPT web scraping project with a simple prompt.

I have a proxy server I can use. Here are the credentials:
http://MMthaEvPm1ISdnmA:xGygceaB5Kfm6V@geo.iproyal.com:12321
Please modify my code to use this proxy correctly, handle possible connection errors or timeouts, and keep the existing scraping logic unchanged.

Here’s the resulting code:

import requests
from requests.exceptions import ProxyError, ConnectTimeout, ReadTimeout, RequestException
from bs4 import BeautifulSoup

# Website URL
url = "http://books.toscrape.com/"

# Proxy configuration
proxy = "http://MMthaEvPm1ISdnmA:[email protected]:12321"

proxies = {
    "http": proxy,
    "https": proxy
}

try:
    # Send GET request through proxy
    response = requests.get(
        url,
        proxies=proxies,
        timeout=10
    )

    # Raise error for bad status codes
    response.raise_for_status()

    # Parse HTML
    soup = BeautifulSoup(response.text, "html.parser")

    # Find all book containers
    books = soup.find_all("article", class_="product_pod")

    # Loop through books and print title + price
    for book in books:
        # Book title is inside the h3 tag
        title = book.h3.a["title"]

        # Book price is inside p with class "price_color"
        price = book.find("p", class_="price_color").text

        print(f"Title: {title}")
        print(f"Price: {price}")
        print("-" * 40)

except ProxyError:
    print("Proxy connection failed.")

except ConnectTimeout:
    print("Connection timed out while connecting to the website.")

except ReadTimeout:
    print("The server took too long to send data.")

except RequestException as e:
    print(f"An error occurred: {e}")

Everything else in the code remains the same as before, so all that's left to do is to replace the example proxy credentials with your own. Follow our quick setup guide for setting up your proxies and finding credentials.

Note: Your proxy credentials contain sensitive information, such as your username and password. Do not share this information with third parties, including ChatGPT. While OpenAI cares about the privacy of your conversations, there have been instances of chat histories leaking to other users.

Multi-Page Scraping

So far, our ChatGPT web scraping script has only extracted data from the first page. Chances are, your real-life project is more complicated and requires scraping beyond the first page. We can prompt ChatGPT to include pagination logic and conditions for when to stop.

Add pagination to my scraper. It should loop through pages, stop when no next page exists, and collect all results into a single list.

The resulting code looks as follows:

import requests
from requests.exceptions import ProxyError, ConnectTimeout, ReadTimeout, RequestException
from bs4 import BeautifulSoup
from urllib.parse import urljoin

# Start URL
base_url = "http://books.toscrape.com/"
url = base_url

# Proxy configuration
proxy = "http://MMthaEvPm1ISdnmA:[email protected]:12321"

proxies = {
    "http": proxy,
    "https": proxy
}

# Store all results here
all_books = []

try:
    while True:
        response = requests.get(url, proxies=proxies, timeout=10)
        response.raise_for_status()

        soup = BeautifulSoup(response.text, "html.parser")

        # Find all book containers
        books = soup.find_all("article", class_="product_pod")

        # Extract data (same logic, just storing instead of only printing)
        for book in books:
            title = book.h3.a["title"]
            price = book.find("p", class_="price_color").text

            all_books.append({
                "title": title,
                "price": price
            })

        # Find next page link
        next_btn = soup.select_one("li.next a")

        if next_btn:
            next_page_url = next_btn["href"]
            url = urljoin(url, next_page_url)
        else:
            break

    # Print all results at the end
    for item in all_books:
        print(f"Title: {item['title']}")
        print(f"Price: {item['price']}")
        print("-" * 40)

except ProxyError:
    print("Proxy connection failed.")

except ConnectTimeout:
    print("Connection timed out while connecting to the website.")

except ReadTimeout:
    print("The server took too long to send data.")

except RequestException as e:
    print(f"An error occurred: {e}")

Handling Dynamic Sites and JavaScript

Many real-world targets are dynamic websites with JavaScript-rendered pages. It complicates the whole data extraction procedure as static HTTP requests return raw responses, not processed by the browser. Such sites require headless browsers, like Playwright or Selenium , for parsing dynamic HTML.

Here's a prompt that might be used:

Please modify my scraper to handle JavaScript-rendered content using an appropriate tool (e.g., Selenium or Playwright).
Keep the existing scraping logic, but ensure the page fully loads before extracting data. Also include any necessary waits to handle dynamic elements.
Search for the newest libraries for proxy integration into browser automation libraries. Add all necessary installs separately.

Following this prompt, ChatGPT completely rewrote our web scraper with Selenium.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from webdriver_manager.chrome import ChromeDriverManager

from bs4 import BeautifulSoup
from urllib.parse import urljoin

# -----------------------------
# Proxy setup (authenticated)
# -----------------------------
proxy_host = "geo.iproyal.com:12321"
proxy_user = "MMthaEvPm1ISdnmA"
proxy_pass = "xGygceaB5Kfm6V"

proxy = f"http://{proxy_user}:{proxy_pass}@{proxy_host}"

# -----------------------------
# Selenium options
# -----------------------------
options = Options()
options.add_argument(f"--proxy-server={proxy}")

# Ensures JS pages behave more like real users
options.add_argument("--disable-blink-features=AutomationControlled")

driver = webdriver.Chrome(
    ChromeDriverManager().install(),
    options=options
)

wait = WebDriverWait(driver, 15)

# -----------------------------
# Scraping state
# -----------------------------
url = "http://books.toscrape.com/"
all_books = []

try:
    while True:
        driver.get(url)

        # -----------------------------
        # WAIT: ensure JS/content loaded
        # -----------------------------
        wait.until(
            EC.presence_of_all_elements_located((By.CSS_SELECTOR, "article.product_pod"))
        )

        soup = BeautifulSoup(driver.page_source, "html.parser")

        # -----------------------------
        # YOUR ORIGINAL SCRAPING LOGIC
        # -----------------------------
        books = soup.find_all("article", class_="product_pod")

        for book in books:
            title = book.h3.a["title"]
            price = book.find("p", class_="price_color").text

            all_books.append({
                "title": title,
                "price": price
            })

        # -----------------------------
        # PAGINATION
        # -----------------------------
        next_btn = soup.select_one("li.next a")

        if next_btn:
            url = urljoin(url, next_btn["href"])
        else:
            break

finally:
    driver.quit()

# -----------------------------
# Output results
# -----------------------------
for item in all_books:
    print(f"Title: {item['title']}")
    print(f"Price: {item['price']}")
    print("-" * 40)

Bypassing Anti-Bot Measures

More anti-scraping systems used on popular websites require your web scraper to mimic real user behavior. Settings for rotating user-agent headers, delays between requests, and other measures might be needed. An example ChatGPT prompt for adding them might look like this:

Modify my Python scraper to reduce bot detection. Add rotating User-Agent headers, random delays between requests, and session handling using requests.Session. Keep the scraping logic unchanged.

The resulting code reuses sessions, introduces rotating user agents, and adds random delays. Note that we use the code with the requests library instead of headless browsers, since the former is more efficient, but more detectable. It’s always a tradeoff in real-world scenarios.

import requests
import random
import time
from bs4 import BeautifulSoup
from urllib.parse import urljoin
from requests.exceptions import ProxyError, ConnectTimeout, ReadTimeout, RequestException

# -----------------------------
# Base URL
# -----------------------------
base_url = "http://books.toscrape.com/"
url = base_url

# -----------------------------
# Proxy (optional, kept from your setup)
# -----------------------------
proxy = "http://MMthaEvPm1ISdnmA:[email protected]:12321"

proxies = {
    "http": proxy,
    "https": proxy
}

# -----------------------------
# Rotating User-Agents
# -----------------------------
USER_AGENTS = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/122.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 13_5) AppleWebKit/537.36 Chrome/121.0 Safari/537.36",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/120.0 Safari/537.36",
    "Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/605.1.15 Version/17.0 Mobile/15E148 Safari/604.1"
]

# -----------------------------
# Session (keeps cookies alive)
# -----------------------------
session = requests.Session()

all_books = []

try:
    while True:
        headers = {
            "User-Agent": random.choice(USER_AGENTS),
            "Accept-Language": "en-US,en;q=0.9",
            "Connection": "keep-alive"
        }

        # Random delay BEFORE request (anti-bot behavior)
        time.sleep(random.uniform(1.2, 3.5))

        response = session.get(
            url,
            headers=headers,
            proxies=proxies,
            timeout=10
        )

        response.raise_for_status()

        soup = BeautifulSoup(response.text, "html.parser")

        # -----------------------------
        # EXISTING SCRAPING LOGIC (UNCHANGED)
        # -----------------------------
        books = soup.find_all("article", class_="product_pod")

        for book in books:
            title = book.h3.a["title"]
            price = book.find("p", class_="price_color").text

            all_books.append({
                "title": title,
                "price": price
            })

        # Pagination
        next_btn = soup.select_one("li.next a")

        if next_btn:
            url = urljoin(url, next_btn["href"])
        else:
            break

except ProxyError:
    print("Proxy connection failed.")

except ConnectTimeout:
    print("Connection timed out.")

except ReadTimeout:
    print("Read timeout occurred.")

except RequestException as e:
    print(f"Request error: {e}")

# -----------------------------
# Output results
# -----------------------------
for item in all_books:
    print(f"Title: {item['title']}")
    print(f"Price: {item['price']}")
    print("-" * 40)

Saving Data to CSV or JSON

At this point, our ChatGPT web scraper can perform some scraping functions, but you likely need it in a structured data format, such as CSV or JSON. A good prompt for such a task should clearly state the format and provide some additional details on the structure.

Modify my web scraper so that instead of printing results, it saves the scraped data into a CSV file with columns for title, price, and link. Keep the scraping logic unchanged.

As instructed, ChatGPT will add a few lines for saving what's scraped into a structured data format we instructed.

# Save to CSV
import csv

csv_file = "books.csv"

with open(csv_file, mode="w", newline="", encoding="utf-8") as file:
    writer = csv.DictWriter(file, fieldnames=["title", "price", "link"])
    writer.writeheader()
    writer.writerows(all_books)

print(f"Saved {len(all_books)} books to {csv_file}")

Advanced Data Analysis

ChatGPT's Advanced Data Analysis feature, previously named Code interpreter, allows users to run Python code in a sandbox environment. It can process files, analyze data, and generate tables, charts, and other outputs.

The environment is limited for use as a ChatGPT web scraper. It has no open internet access, short runtime, and no ability to interact with external systems. As a result, running Python web scraping scripts, even those that ChatGPT itself generated, will return an error.

For a full web scraping process, you must save the code into a file and run it on your machine. Only then will your web scraper be able to make live HTTP requests, handle sessions, and access the full website.

The real utility of Advanced Data Analysis lies in data processing tasks. You can upload your dataset and use it to clean and structure data, parse dynamic HTML snippets, convert formats, and run various analyses. The feature should be treated as an experimentation tool rather than a full scraping engine.

Tips for Making ChatGPT Coding Prompts

ChatGPT is best as a supporting tool for web scraping, like an intelligent assistant, complementing your data collection stack. Beginners will find use in ChatGPT's capabilities to explain concepts and generate starter code. For advanced users, the LLM speeds up the development process while still requiring validation.

• Double-checking answers and code. Always test generated scripts, as ChatGPT tends to produce convincing yet incorrect or inefficient logic.
• Be as specific as possible. Clear instructions with URLs, data targets, selectors, and constraints are crucial for good results.
• Asking for explanations. Understanding the logic behind what ChatGPT generates is not only useful for learning, but it also helps debug and adapt Python scripts for later.
• Include edge cases. Mentioning known complications, like pagination, or restrictions, like rate limits, will make the web scraping process much faster.
• Verifying HTML structure manually. Don't blindly rely on the elements ChatGPT suggests and inspect the DOM in DevTools yourself whenever possible.
• Compare different approaches. Ask ChatGPT to provide multiple scraping strategies and compare them before choosing one. For example, Scrapy, BeautifulSoup, and Selenium all have their pros and cons.
• Asking for optimization suggestions. Sometimes it's more efficient to write parts of the code yourself and then ask ChatGPT to review and improve it.

Limitations of ChatGPT for Web Scraping

ChatGPT can be a powerful assistant in the right hands, but it has important limitations that are crucial to know for web scraping.

• No real code execution or testing. Even with the Advanced Data Analysis, ChatGPT doesn't help you test the code in real-world conditions. Be sure to set up a testing environment yourself and review all the code it generates.
• Struggles with anti-bot detections. ChatGPT can suggest and even implement some basic web scraping techniques, but it does not bypass advanced systems. Cloudflare restrictions, advanced fingerprinting, JavaScript challenges, and other hurdles will have to be solved manually.
• Limited scalability. ChatGPT is not designed to develop large-scale data pipelines or work with important data infrastructure. It can solve separate tasks and help find solutions, but, at least currently, its role is more of an assistant.
• Legal and ethical responsibility. OpenAI does not take any responsibility for the scraping you perform with ChatGPT-generated scripts. The burden of compliance with website terms, robots.txt, and local laws lies entirely with the user.

Conclusion

With only a few exceptions, the tips and instructions here apply to similar LLMs. All of them are limited in similar ways as well, but it might be easier to get the desired results from multiple tools. Consider using other LLMS, such as Claude, DeepSeek, or Gemini , and not rely on web scraping with ChatGPT alone.

FAQ