50% OFF Residential Proxies for 9 months — use code IPR50 at checkout

Get The Deal
Back to blog

How to Use Chromedp for Web Scraping, Testing, and Automation

Justas Vitaitis

Last updated -
How to

Key Takeaways

  • Chromedp is ideal for Go developers who need quick web scraping or UI test automation on dynamic web pages.

  • You can use Chromedp in a headless browser in headless mode, making it faster and simpler than Selenium in some cases.

  • The Chromedp package enables you to script a full browser context, making tasks such as element node matching and data collection easier.

Ready to get started?

Register now

If you need a tool to automate your Chrome browser using Go, that’s where Chromedp comes in. It’s great for web scraping, running tests, or even filling out forms without clicking anything yourself.

Here, you will learn how to use the Chromedp package from the ground up.

What Is Chromedp?

Chromedp is a tool that helps you control Chrome using Go code. It communicates with a browser instance without needing to see anything on screen. In other words, Chromedp runs in the background using headless mode.

Chromedp is perfect for tasks like data extraction or automated testing working with a headless browser. However, if you wish, you can change the context options to run in non-headless mode.

Behind the scenes, Chromedp works by connecting to Chrome’s DevTools. You get full control of what happens in the browser context. Once you get the hang of it, it’s a handy tool for browser automation.

Also, you can even use Chromedp proxy settings if you want to stay private while scraping and minimize the chances of being blocked.

Installing Chromedp in Your Go Project

We’ll be using Visual Studio Code to run our Go code. First, make sure you have Go installed in the extension library. Then, open up the Terminal and run:

go mod init example.com/chromedp-local

You can use a real Github profile if you plan on sharing your code. After the initialization is completed, run:

go get github.com/chromedp/chromedp@latest

That pulls in the Chromedp package. Then, make a simple Go file:

package main

import (
    "context"
    "github.com/chromedp/chromedp"
)

func main() {
    ctx, cancel := chromedp.NewContext(context.Background())
    defer cancel()

    var title string
    chromedp.Run(ctx,
        chromedp.Navigate("https://iproyal.com"),
        chromedp.Title(&title),
    )

    println("Page title:", title)
}

This creates a new Chromedp context, opens a browser instance, and grabs the title. You’ll find the printed page title in the debug console menu.

Chromedp for Web Scraping

Using Chromedp for web scraping is a great idea. You can get text, images, or anything else. Let’s say you want to scrape the headline from a blog:

    var headline string
    err := chromedp.Run(ctx,
        chromedp.Navigate(`https://iproyal.com/blog/what-is-a-headless-browser/`),
        chromedp.Text(`h1`, &headline),
    )
    if err != nil {
        println("Error:", err.Error())
        return
    }
    println("Headline:", headline)

Now headline holds the page title. You can do web scraping on dynamic web pages, too. Since Chromedp runs in a headless browser, it waits for things to fully load.

If you want to scrape an image with Chromedp, here’s how:

    var imageSrc string
    err = chromedp.Run(ctx,
        chromedp.Navigate("https://iproyal.com/blog/what-is-a-headless-browser/"),
        chromedp.AttributeValue(`img[src*="What_are_Headless_Browsers_and_How_They_Work"]`,
            "src", &imageSrc, nil),
    )
    if err != nil {
        println("Error fetching image src:", err.Error())
        return
    }
    println("Image src:", imageSrc)

This is where the element node matching helps. Use CSS selectors to find exactly what you want. If the page uses JavaScript to build things, that’s not a problem. The headless shell handles dynamic web pages with no issues.

You can use this same Chromedp approach for data collection tasks across the web. Just load the page, wait a bit, and then you’ll be able to get what you want.

Automating UI Testing With Chromedp

If you want to test a login form or some other UI element, Chromedp can do that as well. All you need to do is interact with elements using selectors. Here’s how you can test a simple login:

err := chromedp.Run(ctx,
    chromedp.Navigate(`https://example.com/login`),
    chromedp.SendKeys(`#username`, "myuser"),
    chromedp.SendKeys(`#password`, "mypassword"),
    chromedp.Click(`#login-btn`),
)

With this, your code types into the form and clicks the button. It works great inside a CI pipeline. You don’t need a full browser running, just a headless shell with headless mode enabled.

If you’re using Chromedp proxy support, you can even run tests through different locations. It’s especially useful for geo-based features.

Also, your browser context stays clean for each test if you start fresh, which makes your results more reliable.

Chromedp vs Selenium: Which One Should You Use?

Here’s how Chromedp compares against Selenium:

Feature Chromedp Selenium
Language support Go only Many (Python, Java, etc.)
Browser instance Chrome only Chrome, Firefox, Edge
Speed Fast Slower
Setup complexity Simple with Go More complex
Use for web scraping Great with Go Great with many languages
Headless browser Built-in Needs setup

So, if you write Go and need a fast tool for web scraping or testing dynamic web pages, go with Chromedp. If you need multi-language support, Selenium would fit a lot better.

Also, make sure you top up or refresh your knowledge about headless browsers before you jump in.

Conclusion

Chromedp is a lightweight Go tool that handles web scraping, UI testing, and browser automation. It works silently in the background using headless browser mode, runs fast, and is easy to set up.

You should use Chromedp when you’re dealing with dynamic web pages, forms, or any other type of data extraction. Plus, the Chromedp package gives you full control over a browser instance. Combine that with smart element node matching, and you’ve got a solid tool.

And if you need to use other languages, check out how to scrape with Python and Selenium .

Create Account

Author

Justas Vitaitis

Senior Software Engineer

Justas is a Senior Software Engineer with over a decade of proven expertise. He currently holds a crucial role in IPRoyal’s development team, regularly demonstrating his profound expertise in the Go programming language, contributing significantly to the company’s technological evolution. Justas is pivotal in maintaining our proxy network, serving as the authority on all aspects of proxies. Beyond coding, Justas is a passionate travel enthusiast and automotive aficionado, seamlessly blending his tech finesse with a passion for exploration.

Learn More About Justas Vitaitis
Share on

Related articles