How to Run Selenium Headless in Python: Step-by-Step Guide

Selenium is an umbrella project that provides tools and libraries for browser automation across many programming languages, including Python. In simple terms, Selenium lets you run browsers without the UI and test web applications.

Selenium’s headless browser is also widely used for web scraping purposes as they are much faster and less resource intensive than headful browsers. Running Selenium headless in Python also provides more room for optimization and better programmatic control.

Preparation for Running Selenium

Before you can run a headless browser with Selenium, you’ll need to get Python and an IDE. You can download Python from their official website .

There are many IDEs available with various different features, but a good starting point for Python browser emulation is PyCharm Community Edition . It’s a free IDE that works perfectly fine for all Python programming.

Once both are installed, open PyCharm and start a new project.

Then, you’ll need to install the Selenium library . You can do that by opening up the Terminal in PyCharm and typing in:

pip install selenium

PyCharm will then automatically download, extract, and install the Selenium library. It’ll only be available for the current project. While you can install them permanently, doing so is not necessary for smaller projects.

Finally, you may have to find a headless webdriver. Headless Selenium runs by using existing browser webdrivers. Current versions should come with Webdriver versions for all popular browsers.

Running Selenium in Headful Mode

We’ll first test Selenium in headful mode to ensure everything is running correctly.

Start by importing webdriver from Selenium:

from selenium import webdriver

Then, we’ll define a function that takes an argument for the URL:

def open_browser(URL: str):
    browser = webdriver.Chrome()
    browser.get(URL)
    browser.quit()

open_browser('https://iproyal.com/')

Our first line defines the function and its argument (URL). Although adding “: str” is not necessary, it limits the acceptable inputs to only strings, which reduces the likelihood of human error. Make sure to remove it, if you plan on using other data formats.

“Browser” creates an object with the value of “webdriver.Chrome()”, which essentially creates a webdriver object we’ll later use to perform certain actions.

Then, the “browser.get(URL)” uses the browser object’s method “get” to go to a URL. After the page loads fully, the browser should exit automatically due to “browser.quit()”.

Finally, we call the function “open_browser” with our desired URL in between parentheses and quotation marks.

Let’s execute the code by clicking the green arrow at the top right. If everything works correctly, a browser should open up automatically and automatically go to our homepage .

If you get a missing webdriver error when executing code, you’ll have to download the webdriver manually. Since we’ll be using the Chrome webdriver, you can download it from the testing repository . Make sure to download “chromedriver”, appropriate for your OS.

Switching to Selenium Headless Mode

In our previous example, a browser opened up and visited our homepage with the full GUI. For testing and web scraping purposes, most opt to run Selenium headless.

We’ll have to add another import that will allow us to adjust the Selenium chromedriver headless options.

from selenium import webdriver
from selenium.webdriver import ChromeOptions

Now three new lines of code will have to be added to our function:

def open_browser(URL: str):
    options = ChromeOptions()
    options.add_argument("--headless=new")
    browser = webdriver.Chrome(options=options)
    browser.get(URL)
    browser.quit()

open_browser('https://iproyal.com/')

“Options” creates an object to store our Chrome headless browser options. We’ll then use the “.add_argument” method to add a string that tells the browser to run in headless mode.

Finally, we need to adjust the “browser” object to include our Chrome headless Selenium commands.

We can now execute our code again by clicking the same green arrow.

Note that “nothing” will happen as we have removed the UI by running a headless browser. Python gives you a few more options that make Selenium’s headless mode even better.

Running Through a List of URLs

In almost all cases, you’ll be using Selenium to run through a larger list of URLs instead of a single one. We can do that by changing our function:

def open_browser(URL):
    options = ChromeOptions()
    options.add_argument("--headless=new")
    browser = webdriver.Chrome(options=options)
    for i in URL:
        browser.get(i)
    browser.quit()

Our function will now accept sequence objects (lists, tuples, dictionaries, etc) as we have used the “for” loop. When calling the function, we’ll have to input one of those objects and then the browser will visit each URL in sequence.

We also need to add a dictionary object:

list_of_urls = ['https://iproyal.com/', 'https://iproyal.com/pricing/residential-proxies/']
open_browser(list_of_urls)

Each object in the dictionary is separated by a comma and you can add as many as you like. However, the current function will not accept a single string that’s not in a sequence object.

Before you execute the code, it’s recommended to comment out (by using # to the left of a line of code) the “.add_argument” method. Otherwise, Python’s Selenium headless mode will be active and you won’t be able to see if everything is happening as it should.

Here’s the full code block:

from selenium import webdriver
from selenium.webdriver import ChromeOptions

def open_browser(URL):
    options = ChromeOptions()
    options.add_argument("--headless=new")
    browser = webdriver.Chrome(options=options)
    for i in URL:
        browser.get(i)
    browser.quit()

list_of_urls = ['https://iproyal.com/', 'https://iproyal.com/pricing/residential-proxies/']
open_browser(list_of_urls)

Conclusion

You only need a few lines of code to run Selenium webdriver headless. Python, however, gives you many more options and, when automating, you’ll be doing more than simply opening up a browser in headless mode.