How to Set or Change User Agent with cURL

cURL or client URL is a command-line tool intended to send information over various internet protocols (HTTP, HTTPS, FTP, etc.). Client URL is widely used as a quick way to connect to websites, send or retrieve files, and for many other internet-based communications.

Since cURL follows internet protocol rules, any HTTP(S) request sends a lot of additional information. One important part of any HTTP request is the HTTP header, which includes a user agent.

HTTP headers are intended to send metadata about your machine to ensure the response is correctly formulated. A user agent includes data about your browser, OS, and some additional information.

User agents, however, can also be used for tracking purposes. As such, a user agent is often manually changed when engaging in web scraping or any other automation. In this guide, you’ll learn how to: cURL, set user agent(s), and what are the best strategies to minimize blocks.

cURL User Agent Overview

How to Use cURL

cURL is available by default on most operating systems, such as Windows 10, macOS, and some Linux distributions. You can access it through a command-line tool such as Terminal. Or you can read how to download curl file in our blog post.

For the purpose of this tutorial, we’ll be restricting ourselves to sending a cURL request through HTTP(S).

A default cURL request is considered to be “GET”. So, if you want to retrieve information from a URL, all you need to do is type “curl” and some domain:

curl https://iproyal.com

Running the cURL request should provide you with a long, somewhat messy output.

It’s important to note that cURL strictly follows the URL syntax – by default, it will only attempt to communicate with that particular URL. If there’s a redirect in place, you’ll receive a message that the document has been moved. To make cURL follow a redirect, use “-L”:

curl -L https://www.iproyal.com

If you attempted to cURL “ https://www.iproyal.com ” without the “-L” command-line option, you’d receive the aforementioned message.

Finally, you can send custom headers to any domain. You can do so by using “-H”:

curl -L -H "Accept: application/json" https://www.iproyal.com

Note that the command syntax allows you to mix and match most options in any order. There’s no difference between these two functions:

curl -L -H "Accept: application/json" https://www.iproyal.com

curl -H "Accept: application/json" -L https://www.iproyal.com

What Is a User Agent?

A user agent string is a short piece of text that’s interpreted as providing metadata about the machine that’s sending a connection request. The destination server then interprets the user agent string to provide an appropriate response. Here’s a user agent example:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36

While a user agent has been primarily used as a way to optimize server responses when delivering content, it’s also used for analytical and tracking purposes . Not only can a user agent be used to follow the same machine (even if it changes IPs), but it can also be used to forbid specific devices from accessing content.

As such, when using an automated script, such as during data collection, rotating user agents becomes a necessity as a web server may eventually completely restrict access to content.

How to Change the User Agent with cURL

You can use the cURL set user agent option (“-A”) to manually change the user agent included in your request:

curl -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36" https://iproyal.com

Note that the cURL set user agent option places no syntax restrictions. You can set any user agent string you like. However, if it’s completely faked, some websites may block your request outright.

The user agent string has a specific ordering that you may manipulate to achieve your desired result. Let’s use the agent string above as our example:

1. Mozilla/5.0 : A legacy component primarily used for compatibility. It’s included in almost every user agent string because some (very) old servers only recognize the Netscape browser.

2. (Windows NT 10.0; Win64; x64) : The second part of the user agent string provides information about the operating system. In this case, the cURL user agent states that the operating system is Windows 10, 64-bit, x64 CPU architecture.

3. AppleWebKit/537.36 : States the browser rendering engine. In this case, the cURL user agent showcases that it’s using AppleWebKit (used by Safari and Chrome) and the version number.

4. (KHTML, like Gecko) : This part of the cURL user agent string is another compatibility message. KHTML is a different browser engine (Konqueror), and signifies that it behaves like the Gecko engine.

5. Chrome/90.0.4430.93 : One of the more obvious parts of the cURL user agent string – states the browser name (Chrome) and version number.

6. Safari/537.36 : Since the same WebKit is used for the Safari browser, the cURL user agent string can be used to ensure Safari compatibility. The version number matches the AppleWebKit listed in a previous part of the cURL user agent string.

You can modify any part of the cURL user agent string to build something completely unique. Building a custom user agent string, however, isn’t always the best idea. First, the ordering matters for older servers, so a custom user agent string may be misinterpreted.

Additionally, if you create a specific user agent string that’s extremely unique, it’ll make it easier to track your machine. Browser fingerprinting partly relies on the uniqueness of the user agent, so your best bet is to pick something generic that everyone is using.

Setting a Permanent User Agent with cURL

If you don’t want to send a user agent cURL option with each request, there’s a possibility of creating a permanent one. For that you’ll need to create (or find) a file called “_curlrc” (for Windows) or “.curlrc” (for Unix and macOS):

• Windows : You can find the file in “C:\Users_curlrc”.
• Unix-based systems : You can find the file in “~/.curlrc”.
• macOS : You can find the file in the Home directory.

If the file doesn’t exist, use your OS text editor to create a file and add a single line:

user-agent = "[YOUR USER AGENT]"

Once that’s saved, run the cURL “-v” command-line option to any domain to verify that the user agent was set correctly:

curl -v http://iproyal.com

You’ll get a much larger output, which will take some scrolling, but your user agent will be near the very top of the message.