How to Scrape Dynamic Websites with Python (and Avoid Getting Blocked)

Scraping dynamic websites with Python can be extremely useful for extracting data hidden behind JavaScript. Scraping dynamic web pages that use JavaScript is more difficult for web scrapers to manage because elements change dynamically.

After reading this, you’ll understand the difference between static and dynamic content and how to extract data safely. You’ll also get free code examples to scrape dynamic pages and tips to avoid IP bans.

What Is Dynamic Web Scraping (vs Static)?

A static web page sends all its content in the HTML response. You extract data, parse it, and that’s all. Scraping static websites is relatively fast and easy.

Scraping dynamic web pages is more difficult since they change after the initial load. They rely on JavaScript to render new dynamic content, load new dynamic elements, or trigger user interactions. As a result, you need special tools to render JavaScript and mimic those interactions.

Static websites are easy to scrape and have low bandwidth requirements, but they’re limited to pages with little interactivity.

Dynamic web pages might contain more valuable information that you need, but they’re also more difficult to scrape.

Understanding the Document Object Model (DOM) helps when working with both. It’s basically the map of the page that your script will explore.

Challenges of Scraping Dynamic Web Pages

When a page uses JavaScript to tweak parts of the page after it loads, your web scraper won’t see that dynamic content in raw HTML. A script might call APIs or respond to clicks, which sends new network requests.

Scraping dynamic web pages is trickier due to all of those factors. Those dynamic elements might not exist right away, and they won’t be visible to something that interacts only with HTTP requests.

You may need to wait, detect, or simulate user interactions, such as scrolling or clicking. Without doing that, you risk missing key information or getting blocked by anti-scraping measures.

Tools You Need for Dynamic Web Scraping in Python

Here are some beginner-level tools you should use:

• Selenium /. It mimics a real browser, so it can render JavaScript and handle complex dynamic web interactions.
• BeautifulSoup. Once Selenium has finished loading the page, BeautifulSoup parses the final HTML.

You might also consider browser automation libraries like Playwright or Splash. However, for a classic approach, you may want to stick to Selenium first. You may also be interested in Rust web scraping with Selenium.

Here’s a quick install guide for all these tools, assuming you have an IDE ready:

pip install selenium beautifulsoup4

How to Scrape Dynamic Sites: Step-by-Step Code

Here's a complete example on how to scrape dynamic sites using Python:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from bs4 import BeautifulSoup
import time


def scrape_dynamic_site(url):
    """
    Scrape product data from a dynamic page with infinite scroll.

    Args:
        url: The target URL to scrape.

    Returns:
        A list of dictionaries containing product name, price, and image URL.
    """
    options = webdriver.ChromeOptions()
    options.add_argument("--headless=new")

    driver = webdriver.Chrome(options=options)

    try:
        print(f"Fetching: {url}")
        driver.get(url)

        wait = WebDriverWait(driver, 10)
        wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, ".product-item")))

        last_height = driver.execute_script("return document.body.scrollHeight")

        while True:
            driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
            time.sleep(2)

            new_height = driver.execute_script("return document.body.scrollHeight")

            if new_height == last_height:
                break

            last_height = new_height

        html = driver.page_source
        soup = BeautifulSoup(html, "html.parser")
        items = soup.select(".product-item")

        print(f"Found {len(items)} items!")

        data = []
        for item in items:
            name = item.select_one(".product-name")
            price = item.select_one(".product-price")
            image = item.select_one(".product-image")
            data.append({
                "name": name.text.strip() if name else "No Name",
                "price": price.text.strip() if price else "No Price",
                "image": image.get("src") if image else None
            })

        return data

    except Exception as e:
        print(f"An error occurred: {e}")
        return []

    finally:
        driver.quit()


if __name__ == "__main__":
    url = "https://www.scrapingcourse.com/infinite-scrolling"
    results = scrape_dynamic_site(url)
    print(results[:5])

Here's what's happening in this web scraping process:

• Chrome options. We set --headless=new so the browser runs in the background without popping up a window on your screen.
• Smart waits and throttling. We use WebDriverWait for the initial load to ensure elements are ready. However, for the scrolling action, we use a short time.sleep(2) to allow the server time to render new items after we scroll.
• Handling scroll. We use a simple loop to scroll to the bottom of the page using JavaScript, mimicking a user pressing the "Page Down" key to trigger more content loading.
• Parsing. Once the interactions are done, we hand the raw HTML over to BeautifulSoup, which is much better at searching through HTML tags than Selenium is.

The script defines a main function, scrape_dynamic_site, which initializes the browser, waits for the initial content to load, performs a loop to scroll down the page, and then hands the HTML source code to BeautifulSoup for parsing.

Such a web scraping approach is ideal for scraping dynamic websites. For static websites, you wouldn't always need Selenium, as in most cases, you could use requests and BeautifulSoup. If you want to learn more, you should check out how to use web scraping across different industries and what it requires.

How to Avoid Getting Blocked While Web Scraping

IP bans occur when web pages detect too many calls in a short period. Dynamic websites may watch for repeated network requests, same patterns, or missing user headers. Here's how to avoid that while you extract data from both static and dynamic pages:

• Rotate headers and randomize user agents to mimic real browsers.
• Throttle requests to the web page by adding random delays.
• Use rotating proxies while web scraping dynamic websites to spread across IPs.
• Handle cookies and session IDs to trick anti-bot systems in the web pages.

Follow these best web scraping practices and you will minimize your chances of getting banned while scraping dynamic pages.

Legal and Ethical Considerations

Planning before you start is a crucial step. If you intend to begin web scraping, you need to check Terms and Conditions, privacy regulations, and local laws to avoid trouble.

Public data is usually safe, but it’s not always entirely clear what public data actually encompasses. Scraping behind a login is high-risk. While it may not always be outright illegal, it almost certainly violates Terms of Service, risking an immediate permanent ban of your user account.

To stay ethical, you should also respect robots.txt and rate your request limits. Otherwise, websites will consider you malicious and start blocking your traffic. LinkedIn is notoriously strict. While scraping public data is often legally defensible, LinkedIn aggressively defends its data. They are known to implement strict IP bans and will permanently ban user accounts associated with scraping activity.

Keep in mind that this is not legal advice, and you should always consult a legal professional before scraping.

Comparison of Python Tools for Scraping Dynamic Content

You can pick from different tools for scraping dynamic websites, depending on your needs. Some tools use browser automation libraries, while others aim for simplicity and speed. Using headless browsers can help run scraping tasks without opening a visible window.

Each tool has some trade-offs, be it ease of use, speed, support for JavaScript rendering, or more.

Handling JavaScript-Heavy Sites

If a page injects data into HTML code via scripts, you may grab embedded JSON format values instead of reading visible page text. Sometimes, you need to click tabs or dropdowns so new sections can load, or scroll if the page uses infinite scroll to show more content.

Instead of rendering the page, check the Network tab for hidden API calls. If the site delivers data via JSON, you can bypass the browser entirely and use a standard HTTP library like requests or httpx for much faster results. You can create a new browser instance with browser automation to wait until elements are dynamically loaded or detect when client-side rendering ends.

You can also inspect network calls in browser DevTools to sniff real API endpoints behind the scenes. That often saves time and resources compared with rendering full pages.

Conclusion

Dynamic web scraping with Python involves more work than static web pages. You need tools like Playwright or Selenium to handle JavaScript rendering, whereas web scraping static pages can be done using only requests.

Now you know how to fetch, parse, and export data safely. You've also learned about tips that could help you dodge IP bans and utilize web scraping on dynamic websites more consistently.

Dynamic web scraping is not that difficult once you get the hang of it. While it’s surely more challenging to scrape dynamic pages than static ones, it’s still highly doable once you get the right tools and information.

##FAQ