How to Set or Change User Agent with cURL

cURL, or Client URL, is a command-line tool designed to send information over various internet protocols, such as HTTP, HTTPS, or FTP. Since cURL must follow standard Internet Protocol rules, all its HTTP(S) and HTTP requests include HTTP headers with metadata to ensure the server’s response.

A User-Agent header includes data about your browser, OS, and other information. User agents are often used for tracking purposes, which is problematic when using cURL for web scraping or online automation. Learning how to use cURL to set a user agent manually, permanently, and in different programming languages is crucial for these tasks.

cURL User Agent Overview

How to Use cURL

cURL is available by default on most operating systems, such as Windows 10, macOS, and various Linux distributions (Ubuntu, Debian, Mint, and others). You can download curl and access it through a command-line tool such as Terminal.

Before you attempt to change user agents, verify your cURL installation. The same command works for macOS, Linux, and Windows.

curl --version

For the purpose of this tutorial, we’ll be sending cURL requests through HTTP(S).

cURL GET is the default cURL request method. So, you only need to type curl and specify the domain for retrieving information from a URL.

curl https://iproyal.com

Running the cURL request should provide you with a long, somewhat messy output.

It’s important to note that cURL strictly follows the URL syntax. By default, it will only attempt to communicate with that particular URL. If there’s a redirect in place, you’ll receive a message that the document has been moved. To make cURL follow a redirect, use -L:

curl -L https://www.iproyal.com

If you attempted to cURL access a URL without the -L command-line option, you’d receive the aforementioned HTTP redirect response.

You can send custom headers to any domain by using the*-H* flag that sends headers in quotes. Note that Windows requires headers to be written in double quotes, while Linux and macOS can use single and double quotes interchangeably.

curl -L -H "Accept: application/json" https://www.iproyal.com

The command syntax allows you to mix and match most options in any order. There’s no difference between these two functions:

curl -L -H "Accept: application/json" https://www.iproyal.com
curl -H "Accept: application/json" -L https://www.iproyal.com

What Is a User Agent?

A user agent is a piece of text in a header that provides metadata about the machine sending a connection request. The destination server then interprets the user agent to provide an appropriate response. Here’s a user agent example:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36

While a user agent is primarily used as a way to optimize server responses when delivering content, it’s also used for analytical and tracking purposes . Not only can a user agent be used to track the same machine (even if it changes IPs), but also be used to restrict specific devices from accessing content.

As such, when using an automated script, such as during data collection, rotating user agents becomes a necessity, as a web server may eventually completely restrict access to content.

How to Change the User Agent With cURL

You can use the cURL set-user-agent option (-A) to change the user agent included in your request manually.

curl -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36" https://iproyal.com

Note that this option places no syntax restrictions. You can set any User-Agent string you like. However, if it’s completely faked, some websites may block your request outright.

All User-Agent strings have a specific ordering that you may manipulate to achieve your desired result. Let’s use the agent string above as our example:

• Mozilla/5.0: A legacy component primarily used for compatibility. It’s included in almost every User-Agent string because some (very) old servers only recognize the Netscape browser.
• (Windows NT 10.0; Win64; x64): The second part of the User-Agent string provides information about the operating system. In this case, the cURL user agent states that the operating system is Windows 10, 64-bit, x64 CPU architecture.
• AppleWebKit/537.36: States the browser rendering engine. In this case, the cURL user agent showcases that it’s using AppleWebKit (used by Safari and Chrome) and the version number.
• (KHTML, like Gecko): This part of the cURL User-Agent string is another compatibility message. KHTML is a different browser engine (Konqueror), and signifies that it behaves like the Gecko engine.
• Chrome/90.0.4430.93: One of the more obvious parts of the cURL User-Agent string that states the browser's name (Chrome) and version number.
• Safari/537.36: Since the same WebKit is used for the Safari browser, the cURL user agent can be used to ensure Safari compatibility. The version number matches the AppleWebKit listed in a previous part of the string.

You can change User-Agent strings to build something unique, but building a custom user agent isn’t always the best idea. The ordering matters, especially for older servers, so that a custom User-Agent string may be misinterpreted.

Additionally, if you create a specific User-Agent string that’s extremely unique, it’ll make it easier to track your machine. Browser fingerprinting partly relies on the uniqueness of the user agent, so your best bet is to pick something generic that everyone is using.

Difference Between -A and -H for User-Agent

Both -A and -H can set the User-Agent header, but our example above uses -A (equivalent to --user-agent), since it's a dedicated shortcut for setting the User-Agent header specifically.

curl -A "INSERT USER-AGENT-NAME HERE insert user-agent-name-here" [INSERT URL HERE]

-H is a general-purpose header flag that can set any header you want, including the user agent. So, you can change the user agent with -H just the same as with -A. Note that in both cases, user agent names, like all header names, are case-insensitive.

One difference is that you can remove the User-Agent header entirely while using -H with an empty value after the colon.

curl -H "user-agent:" https://iproyal.com

-H is also preferred when you need finer control over your headers, as it allows setting multiple headers with the same command. In practice, the -A flag is often more convenient and more readable when scraping or automating tasks.

Setting a Permanent User Agent With cURL

If you don’t want to send a user agent cURL option with each request, you can set a permanent one. You’ll need to create (or find) a file called “_curlrc” (for Windows) or “.curlrc” (for Linux and macOS):

• Windows: You can find the file in “C:\Users_curlrc”.
• Unix-based systems: You can find the file in “~/.curlrc”.
• macOS: You can find the file in the Home directory.
• If the file doesn’t exist, use your OS text editor to create a file and add a single line:

user-agent = "[INSERT USER-Agent-NAME HERE]"

Once that’s saved, run the cURL “-v” command-line option to any domain to verify that the user agent was set correctly:

curl -v http://iproyal.com

You’ll get a much larger output, which will take some scrolling, but your user agent will be near the very top of the message.

Using cURL User-Agent in Popular Languages

Since cURL is a command-line tool, most programming languages don't use it directly. Instead, it's invoked as a shell command. The cURL syntax itself remains identical when you change user agents. Only the language-specific wrapper around it differs.

Python uses the subprocess module to run external commands.

import subprocess
import subprocess

result = subprocess.run(
    ["curl", "-L", "-A", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36", "http://iproyal.com"],
    capture_output=True,
    text=True
)
print(result.stdout)

PHP uses the built-in exec() function to execute external programs or shell commands directly from the script.

<?php

$userAgent = escapeshellarg("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36");
$url = escapeshellarg("https://iproyal.com");

exec("curl -sL -A {$userAgent} -H 'Accept: text/html,application/xhtml+xml' -H 'Accept-Language: en-US,en;q=0.9' {$url}", $output, $returnCode);

echo implode("\n", $output);
?>

JavaScript's equivalents are child_process, which can run processes separately from your main application, allowing you to run cURL commands and send HTTP requests.

const { exec } = require("child_process");
exec(
  'curl -sL -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36" -H "Accept: text/html,application/xhtml+xml" -H "Accept-Language: en-US,en;q=0.9" https://iproyal.com',
  (error, stdout, stderr) => {
    if (error) {
      console.error(`Error: ${error.message}`);
      return;
    }
    console.log(stdout);
  }
);

FAQ