How to Use Headers with Python Requests: A Beginner-Friendly Tutorial

Using correct request headers is one of the most important parts of a Python web scraping project. Sending incorrect or malformed headers may bring error status codes or trigger anti-bot measures. Avoid such hurdles by learning how to send Python request headers correctly.

You’ll find that using the Python requests library is quite straightforward when you’re already experienced. If you’re only taking your first steps in Python, there are some crucial basics to cover.

What Are HTTP Headers and Why They Matter

The Hypertext Transfer Protocol (HTTP) defines that every request and response must include HTTP headers, showing the web server how to process it. We can think of them as shipping labels for physical deliveries specifying how to handle the package, where to send it, and other crucial information.

When a Python script or a browser sends a HTTP request, headers tell the web server who you are, what you’re sending, and what kind of response is expected. There’s more to HTTP headers , but such is the basic role of request headers we’re concerned here.

The Python requests library is one of the most popular Python libraries, allowing users to send automated HTTP requests easily. With requests, headers like User-Agent are sent all at once in a single synchronous request, and the script waits for the server’s response before continuing.

For example, the User-Agent header informs that you’re using Chrome and waits for whether such a browser type is accepted. All other request headers are sent at the same time and receive a response synchronously.

More advanced web scraping or automation projects might require HTTPX or AIOHTTP request libraries. They can send Python request headers asynchronously or mix synchronous and asynchronous methods.

Common Header Fields and What They Look Like

Header fields are key:value pairs sent and received by the client and server with every HTTP response and request. They contain the metadata needed for the communication. When we speak of using Python requests headers correctly, usually we mean common header fields.

Some proxy or bot detection methods go beyond User-Agent or, in the case of API requests, Authorization headers. Uncommon or special headers like Forwarded, X-Forwarded-For, or Via are sometimes checked as well.

If they are present or misconfigured, they allow websites to track users, revealing their proxy usage or other identifying information. In such cases, proxy headers tests become handy as they allow you to test whether your proxy server can be identified by such header values.

Differences Between GET and POST Headers

GET and POST are two of the most commonly used HTTP methods, especially when building web scrapers or automating other tasks.

• A GET request is used to retrieve data from a server. It typically sends data through the URL as a query parameter.
• A POST request is used to send data to the server. The JSON data payloads are commonly placed in the body of the request.

Since GET and POST methods differ in purpose, the Python requests headers they use differ as well. GET requests can use only User-Agent and Accept headers. POST requests need to pass more information, with Content-Type request headers often mandatory.

Some Accept headers, such as Accept-Language, are irrelevant for sending structured data via POST. In API requests, both methods might require an authorization header since the server needs to know whether to allow access.

Installing and Importing the Requests Library

To start using Python requests headers, you’ll need to install the Python requests library. First, open a terminal (you might need admin permissions) to install it.

• On Windows, press Windows button + X. Select terminal (or press the letter I).
• On macOS, press Command (⌘) + Spacebar and enter the terminal.
• On Linux (Ubuntu), Press CTRL+Alt+T.

Alternatively, most developers use IDEs and virtual environments. For Python development Visual Studio Code or PyCharm work wonders. If you choose these options (recommended), use the terminal within those IDEs to get libraries:

Type in the following command in the terminal.

pip install requests

Alternatively, you could install a Python requests library for only one user.

pip install --user requests

Or make a system-wide installation.

sudo pip install requests

To confirm that everything is working, we can make a simple GET request to test a URL.

import requests

response = requests.get('https://httpbin.org/get')
print(response.status_code)
print(response.text)

If the Status code is 200, your request was successful, and the Python requests library is set up properly. The URL we provided in this snippet is made for such tests and will return JSON data about your request. Here's how the response might look like.

200
{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Accept-Encoding": "gzip, deflate",
    "Host": "httpbin.org",
    "User-Agent": "python-requests/2.25.1",
    "X-Amzn-Trace-Id": "Root=1-12345678-123456789012345678901234"
  },
  "origin": "123.45.67.89",
  "url": "https://httpbin.org/get"
}

In a real-world scenario, you are likely to encounter many more possible problems with Python requests. Be sure to implement a failed requests retry strategy to avoid issues like connection timeouts, server downtimes, or blocks.

How to View and Print Response Headers

The previous code example used a very basic way of displaying HTTP request headers. When building a Python scraper, integrating APIs, or doing other tasks, it's best to label the headers and print each on a separate line.

They can include content type headers, server details, rate limits, caching rules, and other details. The Python requests library allows us to access them using the .headers attribute on a response object. Here's how you can do it.

import requests

response = requests.get('https://httpbin.org/get')

print("Status code:", response.status_code)

print("\nResponse headers:")
for header, value in response.headers.items():
    print(f"{header}: {value}")

The output now will be each header in a separate line. It's useful in real-world scenarios to check whether the server is returning JSON data, finding rate-limiting headers, or other cases.

Status code: 200

Response headers:
Content-Type: application/json
Date: Mon, 05 Aug 2025 12:34:56 GMT
Server: gunicorn/19.9.0
Content-Length: 234
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
Connection: keep-alive

Since headers functions as a dictionary, it allows access to specific headers separately.

import requests

response = requests.get('https://httpbin.org/get')
content_type = response.headers['Content-Type']

print("Content-Type:", content_type)

After running such code, you would see output similar to the following.

Content-Type: application/json

How to Add Headers to a GET and POST Request

Now that we know how to view headers, another practical problem to solve is sending custom headers to outgoing requests. This is important when calling APIs or accessing websites with a web scraper or automation tool.

In the Python's requests library, custom headers are passed as a dictionary using the headers= parameter in functions like requests.get() and requests.post(). Here's how to set up a custom User-Agent in a GET request.

import requests

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)'
}

response = requests.get('https://httpbin.org/headers', headers=headers)
print(response.json())

This code snippet sends a custom User-Agent header with a specifically defined identity. It's important when you want your User-Agent to mimic a specific setup to avoid bans or other restrictions.

Similarly, POST requests can be customized as well by specifying the Content-Type: application/json.

import requests
import json

headers = {
    'Content-Type': 'application/json'
}

data = {
    'username': 'testuser',
    'password': 'secure123'
}

response = requests.post('https://httpbin.org/post', headers=headers, data=json.dumps(data))
print(response.json())

This custom Python requests header informs the web server that you are sending JSON data, which is required in many instances. Adding custom headers is crucial for communicating with the server and gaining access.

Conclusion

The examples we covered here should give you a head start when using Python requests headers and customizing them. The next step is to start using proxies with the Python requests library before integrating the code into your automation or web scraping project.