How to Run Selenium Headless in Python: Step-by-Step Guide

Selenium is an umbrella project that provides tools and libraries for browser automation across many programming languages, including Python. In simple terms, Selenium lets you run browsers without a graphical interface and test web applications. Selenium’s headless browser is also widely used for web scraping purposes as they are much faster and less resource intensive than headful browsers. Running Selenium headless in Python also provides more room for optimization and better programmatic control.

Preparation for Running Selenium

Before you can run a headless browser with Selenium, you’ll need to get Python and an IDE. You can download Python from their official website . There are many IDEs available with various different features, but a good starting point for Python browser emulation is PyCharm Community Edition . It’s a free IDE that works perfectly fine for all Python programming.

Once both are installed, open PyCharm and start a new project.

Then, you’ll need to install the Selenium library. You can do that by opening up the Terminal in PyCharm and typing in:

pip install selenium

PyCharm will then automatically download, extract, and install the Selenium library. It'll only be available for the current project. While you can install them permanently, doing so is not necessary for smaller projects.

Finally, you may have to find a headless webdriver. Headless Selenium runs by using existing browser webdrivers. Current versions should come with Webdriver versions for all popular browsers.

Alternative: Install Selenium Using WebDrive Manager

Instead of manually downloading the Chrome WebDriver, you can use the WebDriver Manager package. It saves time and avoids version mismatch issues. Install it by running this:

pip install webdriver-manager

Now, modify your code like this:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager

And use it with:

browser = webdriver.Chrome(service=Service(ChromeDriverManager().install()))

You can then use browser.get() and any other methods as per usual. The installation only runs once (if the driver does not exist), otherwise it checks if the appropriate version is present, so there’s not a lot of additional overhead.

Webdriver-manager is great because it handles everything behind the scenes. It’s great for smoother headless browser testing and quick setups. You don’t have to direct your code to specific paths or put the driver in your project folder.

Running Selenium in Headful Mode

We'll first test Selenium in headful mode to ensure everything is running correctly.

Start by importing webdriver from Selenium:

from selenium import webdriver

Then, we'll define a function that takes an argument for the URL:

def open_browser(URL: str):
    browser = webdriver.Chrome()
    browser.get(URL)
    browser.quit()

open_browser('https://iproyal\.com/')

Our first line defines the function and its argument (URL). Although adding “: str” is not necessary, it limits the acceptable inputs to only strings, which reduces the likelihood of human error. Make sure to remove it, if you plan on using other data formats.

“Browser” creates an object with the value of “webdriver.Chrome()”, which essentially creates a webdriver object we'll later use to perform certain actions.

Then, the “browser.get(URL)” uses the browser object's method “get” to go to a URL. After the page loads fully, the browser should exit automatically due to “browser.quit()”.

Finally, we call the function “open_browser” with our desired URL in between parentheses and quotation marks.

Let's execute the code by clicking the green arrow at the top right. If everything works correctly, a browser should open up automatically and automatically go to our homepage .

If you get a missing webdriver error when executing code, you'll have to download the webdriver manually. Since we'll be using the Chrome webdriver, you can download it from the testing repository . Make sure to download “chromedriver”, appropriate for your OS.

Switching to Selenium Headless Mode

In our previous example, a browser opened up and visited our homepage with the full GUI. For testing and web scraping purposes, most opt to run Selenium headless.

We'll have to add another import that will allow us to adjust the Selenium Chrome options for headless Chrome.

from selenium import webdriver
from selenium.webdriver import ChromeOptions

Now three new lines of code will have to be added to our function:

def open_browser(URL: str):
    options = ChromeOptions()
    options.add_argument("--headless=new")
    browser = webdriver.Chrome(options=options)
    browser.get(URL)
    browser.quit()

open_browser('https://iproyal\.com/')

“Options” creates an object to store our Chrome headless mode options. We'll then use the “.add_argument” method to add a string that tells the browser to run headless Chrome.

Finally, we need to adjust the “browser” object to include our Chrome headless Selenium commands.

We can now execute our code again by clicking the same green arrow.

Note that “nothing” will happen as we have removed the UI by running a headless browser. Python gives you a few more options that make Selenium's headless mode even better.

Running Through a List of URLs

In almost all cases, you'll be using Selenium to run through a larger list of URLs instead of a single one. We can do that by changing our function:

def open_browser(URL):
    options = ChromeOptions()
    options.add_argument("--headless=new")
    browser = webdriver.Chrome(options=options)
    for i in URL:
        browser.get(i)
    browser.quit()

Our function will now accept sequence objects (lists, tuples, dictionaries, etc) as we have used the “for” loop. When calling the function, we'll have to input one of those objects and then the browser will visit each URL in sequence.

We also need to add a dictionary object:

list_of_urls = ['https://iproyal.com/', 'https://iproyal.com/pricing/residential-proxies/']
open_browser(list_of_urls)

Each object in the dictionary is separated by a comma and you can add as many as you like. However, the current function will not accept a single string that's not in a sequence object.

Before you execute the code, it's recommended to comment out (by using # to the left of a line of code) the “.add_argument” method. Otherwise, Python's Selenium headless mode will be active and you won't be able to see if everything is happening as it should.

Here's the full code block:

from selenium import webdriver
from selenium.webdriver import ChromeOptions

def open_browser(URL):
    options = ChromeOptions()
    options.add_argument("--headless=new")
    browser = webdriver.Chrome(options=options)
    for i in URL:
        browser.get(i)
    browser.quit()

list_of_urls = ['https://iproyal.com/', 'https://iproyal.com/pricing/residential-proxies/']
open_browser(list_of_urls)

Headless Web Scraping Tutorial With Selenium

Going one step further, you can perform headless web scraping with Python using Selenium. For example’s sake, we’ll scrape the title of a webpage, simulate a wait for AJAX requests, and save the output.

Start with:

pip install selenium webdriver-manager

from selenium import webdriver
from selenium.webdriver import ChromeOptions
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
import time

def scrape_title(url: str) -> None:
    """
    Scrapes the title of a given webpage using Selenium with a headless Chrome browser,
    saves it to a file named 'page_title.txt', and closes the browser.

    Args:
        url (str): The full URL of the webpage to scrape.

    Returns:
        None: The function does not return a value. The extracted page title is written
              to 'page_title.txt'.
    """

    options = ChromeOptions()
    options.add_argument("--headless=new")
    browser = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)
    
    browser.get(url)
    
    time.sleep(2)  
    
    title = browser.title
    
    with open("page_title.txt", "w") as file:
        file.write(title)
    
    browser.quit()

scrape_title('https://iproyal.com/')

Here’s what’s happening:

• We’re using find_element to grab the page title.
• time.sleep() gives time for AJAX requests to finish.
• The title is saved to a file, which is useful for keeping static data.

It’s perfect for beginners doing Python web scraping without a browser. Even though it runs without a UI, you can still get the content you need. Handling a headless browser with Python this way is clean and fast.

Selenium Headless Chrome Options in Python

Selenium’s Chrome options allow customization for headless browser testing, scraping, and automation.

Here’s a breakdown of common options:

options = ChromeOptions()

options.add_argument("--headless=new")             
options.add_argument("--disable-gpu")              
options.add_argument("--no-sandbox")               
options.add_argument("--window-size=1920,1080")    
options.add_argument("user-agent=MyScraperBot") 

Each flag does its own thing:

• --headless=new turns on headless Chrome browser.
• --disable-gpu disables GPU hardware acceleration, rarely needed except for specific rendering issues.
• --no-sandbox lets headless script execution in Selenium work in containers.
• --window-size sets window size for rendering and screenshots.
• User-agent sets custom user-agent string for HTTP requests.

Here’s how to use all of them together:

options = ChromeOptions()
options.add_argument("--headless=new")
options.add_argument("--disable-gpu")
options.add_argument("--no-sandbox")
options.add_argument("--window-size=1920,1080")
options.add_argument("user-agent=MyScraperBot")

You now have full control over your Selenium headless browser. These settings make it easier to execute (or not) JavaScript, scrape static data, and do script execution without a full Chrome browser interface.

It’s perfect for Selenium headless browser automation. There’s even more settings you can find in the official documentation .

Conclusion You only need a few lines of code to run Selenium webdriver headless. Python, however, gives you many more options and, when automating, you'll be doing more than simply opening up a browser in headless mode. It gives you reliable headless testing in Selenium with Python, from importing Selenium webdrivers to configuring Chrome options. You’ve learned the ins and outs of headless browser testing and fast script execution. With these tools, working with headless Chrome becomes simple, fast, and useful when you’re scraping data, doing headless browser testing, or automating tasks.