Mastering HTTPX Client for Python: A Comprehensive Guide

Choosing a Python HTTP client comes down to whether you need synchronous or asynchronous requests and HTTP/2 capabilities. The Requests library is often the default choice, but you are being limited by synchronous HTTP requests and the lack of HTTP/2.

HTTPX is a flexible choice that builds upon Requests to give you modern features with familiar control. The learning curve of HTTPX isn’t steep if you only need synchronous code. It gets more complicated as you also adopt async code and need to work with the error handling or more complex JSON data.

What Is HTTPX and Why Use It?

HTTPX is a next-generation HTTP client for Python 3 that builds upon the foundational principles of the Requests library while introducing advanced features. HTTPX works with both synchronous and asynchronous requests, while natively supporting HTTP/1.1 and HTTP/2 protocols.

At the same time, HTTPX maintains an API and commands, such as for GET and POST requests, familiar to users of the Requests library. Comparing HTTPX with AIOHTTP and Requests libraries highlights its versatility for many modern Python projects, such as Web scraping or API integration.

Requests remains a popular choice for simple synchronous use cases, and AIOHTTP is specialized for purely asynchronous, real-time applications with WebSocket needs. HTTPX is the superior choice for applications requiring concurrent request handling, forward-looking architecture, or HTTP/2 performance benefits.

HTTPX is faster than Requests even in synchronous mode with simple GET or POST requests. Asynchronous code with HTTPX can run twice as fast, which is crucial for projects where you send thousands of requests.

Installing HTTPX and Setting Up Your Environment

HTTPX requires Python 3.8 or later, so it’s recommended to start by installing the newest version or double-checking the version you currently have. Enter the command below in the terminal:

python --version

To avoid conflicts with system packages and isolate your project dependencies, set up a virtual environment. We'll use Windows as an example for terminal commands:

python -m venv venv
venv\Scripts\activate

If the terminal prompt starts with (venv), the virtual environment is active, and we can set up HTTPX. The single pip command below will install it with the optional h2 library for HTTP/2 support (the [http2] can be removed if you don't need it).

pip install httpx[http2]

Additionally, the square brackets might have other extras, such as a CLI tool or SOCKS proxy support. They all can be written in the same pip command by adding extras to the brackets, as such [http2,cli,socks].

Making Your First HTTP Requests

Now that the installation is complete, we can test it by sending our first HTTP request. Create a file named test_httpx.py and use a code editor, such as VS Code , to paste the code below:

import httpx

response = httpx.get('https://httpbin.org/get')
print(f"Status: {response.status_code}")
print(f"HTTP Version: {response.http_version}")

We're using a public HTTP testing service that echoes back our request headers, URL, and other details. After saving the file, we can run the code in our terminal or whatever our IDE offers as a run command.

python test_httpx.py

If all is well, you'll see the following message in your terminal confirming HTTPX is installed correctly and your httpx.get() was successful.

Status: 200
HTTP Version: HTTP/1.1

Now we can make a POST request that sends JSON data from the client to the server. It can be done with the following code:

import httpx

response = httpx.post(
    "https://httpbin.org/post",
    json={"name": "IPRoyal", "role": "tester"},
)

print(f"Status: {response.status_code}")
print(response.json())

After running it in your terminal, a successful POST request will return an output as shown below:

Status: 200
{'args': {}, 'data': '{"name":"IPRoyal","role":"tester"}', 'files': {}, 'form': {}, 'headers': {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Content-Length': '34', 'Content-Type': 'application/json', 'Host': 'httpbin.org', 'User-Agent': 'python-httpx/0.28.1', 'X-Amzn-Trace-Id': 'Root=1-6974bf9f-586154f75e102a63714cf780'}, 'json': {'name': 'IPRoyal', 'role': 'tester'}, 'origin': 'YOUR_IP_ADDRESS', 'url': 'https://httpbin.org/post'}

Making the same POST request multiple times can repeat the action or duplicate JSON data. In many cases, PUT requests are more fitting as sending the same PUT leaves the data in an unchanged final state. Here's how such a request might look:

import httpx

response = httpx.put(
    "https://httpbin.org/put",
    json={"id": 123, "name": "IPRoyal", "role": "tester"},
)

print(f"Status: {response.status_code}")
print(response.json())

Understanding HTTPX Request and Response Handling

Exploring the Response Object

HTTPX response objects are conceptually the same thing as those in Requests. Response objects bundle the status line, response body, headers, and other attributes in the same manner as Requests.

• status_code tells you what happened with the request by showing HTTP status codes, like 200, 404, 500, and others.
• headers describe the metadata about the body using response headers, such as Content-Type and Set-Cookie.
• content (bytes) is the raw response body as bytes.
• text uses the body decoded from the content, transformed into a string.
• json() assumes that the body contains JSON data and returns Python objects.

HTTPX also has a response.raise_for_status() is used for error handling. It inspects response.status_code, and if the status indicates an error, it carries extra context of the request and response for debugging. Unlike the Requests HTTPError, the HTTPX exception allows you to code in a simpler pattern.

import httpx

response = httpx.get("https://httpbin.org/get")
response.raise_for_status()  # error if 4xx/5xx

data = response.json()      # only reached on “successful” HTTP status

Customizing Requests

HTTPX customizes requests in almost the same way as Requests. You use keyword arguments or use an HTTP client. The main difference is that HTTPX adds first-class auth classes and lets you reuse them after setting.

You can add query parameters by using params= to add string values.

import httpx

response = httpx.get(
    "https://httpbin.org/get",
    params={"page": 2, "q": "httpx"},
)

Use headers= for things like custom auth, content types, or user agents:

import httpx

response = httpx.get(
    "https://httpbin.org/headers",
    headers={"X-Request-Source": "httpx-demo", "User-Agent": "my-client/1.0"},
)

And the cookies= argument is used to send cookies:

import httpx

response = httpx.get(
    "https://httpbin.org/cookies",
    cookies={"session_id": "abc123"},
)

HTTP requests can also be customized by adding authentication. HTTPX supports basic authentication via the auth parameter:

import httpx

auth = httpx.BasicAuth("username", "password")

response = httpx.get(
    "https://httpbin.org",
    auth=auth,
)

Token-based authentication (bearer token) can be used by setting the header manually:

import httpx

token = "YOUR_ACCESS_TOKEN"

response = httpx.get(
    "https://api.example.com/data",
    headers={"Authorization": f"Bearer {token}"},
)

More advanced authentication flows can be used by setting third-party libraries like httpx-auth that help to reduce manual inputs.

Handling Errors and Exceptions

HTTPX uses an exception hierarchy so you can distinguish when requests fail to reach the server or when some HTTP errors occur.

• RequestError is a base class for errors that happen while sending the GET or POST request or receiving the response. Always includes a .request attribute so you can inspect the URL, method, and headers that were used.
• HTTPStatusError happens when you call response.raise_for_status() on a response with 4xx or 5xx status code. The error carries both .request and .response, so you could inspect data.
• TimeoutException is used for all timeout-related errors, such as ConnectTimeout, ReadTimeout, or WriteTimeout. It allows you to handle these errors separately from other HTTP errors.

All of the exception types are derived from the HTTPError base class. So, you can use it as a catch-all error handling method, most convenient for simple POST request scripts and CLI tools:

import httpx

try:
    response = httpx.get("https://api.example.com/data", timeout=10.0)
    response.raise_for_status()
    data = response.json()
except httpx.HTTPError as exc:
    print(f"HTTP error while requesting {getattr(exc, 'request', None).url!r}: {exc}")

Another option is to list each exception separately, which is most useful when you need different behavior with each error.

import httpx

try:
    response = httpx.get("https://api.example.com/data", timeout=10.0)
    response.raise_for_status()
    data = response.json()
except httpx.TimeoutException as exc:
    # Request took too long: maybe retry or degrade gracefully
    print(f"Request timed out for {exc.request.url!r}")
except httpx.RequestError as exc:
    # DNS failure, refused connection, TLS issue, etc.
    print(f"Network error while requesting {exc.request.url!r}: {exc}")
except httpx.HTTPStatusError as exc:
    # Server responded, but with 4xx/5xx
    print(
        f"Error response {exc.response.status_code} while requesting {exc.request.url!r}"
    )

Sending Data: Form, JSON, Multipart File Uploads

HTTPX provides flexible ways to send different types of data with POST requests. You need to choose the right parameters, such as data= or json=, and know how HTTPX sets the content type automatically or, when needed, manually.

Form-encoded data is the traditional format used by HTML. The data= parameter is used to form data into a dictionary and convert it to a string format of key-value pairs for sending a POST request.

import httpx

# Form-encoded POST request
data = {
    "username": "alice",
    "password": "secret123",
    "remember": "true"
}

response = httpx.post("https://httpbin.org/post", data=data)
print(response.json())

JSON data is preferred when making structured POST requests, such as arrays. We can use the json=data parameter for HTTPX to automatically serialize the Python dictionary or list using json.dumps() when making a POST request.

import httpx

# JSON POST request
data = {
    "user": {
        "name": "Alice",
        "email": "[email protected]"
    },
    "preferences": ["email_notifications", "dark_mode"],
    "active": True,
    "count": 42
}

response = httpx.post("https://httpbin.org/post", json=data)
print(response.json())

If you need to manually control the JSON data encoding or Content-Type in a POST request, you can do so with the content= parameter.

import httpx
import json

data = {"key": "value"}
json_bytes = json.dumps(data).encode('utf-8')

response = httpx.post(
    "https://httpbin.org/post",
    content=json_bytes,
    headers={"Content-Type": "application/json; charset=utf-8"}
)

You don't need multiple requests for multipart file uploads with HTTPX. It allows sending a single request using the multipart/form-data format for files and forms. A simple file upload in HTTPX automatically sets content-Type: multipart/form-data with the POST request, and the file is sent and read in binary mode.

import httpx

files = {"upload-file": open("report.pdf", "rb")}

request = httpx.Request("POST", "https://httpbin.org/post", files=files)

print(request.headers['Content-Type'])

response = httpx.post("https://httpbin.org/post", files=files)
# Output: multipart/form-data; boundary=...

Streaming Large Responses

For large files or long responses, HTTPX provides a httpx.stream() context manager and iteration methods to process data in chunks. It's a way to download or process HTTP responses, including PUT and POST requests, without loading the entire response into memory at once.

• httpx.stream() can avoid buffering the entire response.

import httpx

with httpx.stream("GET", "https://httpbin.org/bytes/1000000") as response:
    for chunk in response.iter_bytes():
        print(f"Received {len(chunk)} bytes")

• iterbytes() allows us to define response size as binary chunks.

import httpx

with httpx.stream("GET", "https://httpbin.org/stream/10") as response:
    with open("downloaded.zip", "wb") as f:
        for chunk in response.iter_bytes(chunk_size=8192):
            f.write(chunk)

• itertext() converts responses into text chunks.

import httpx

with httpx.stream("GET", "https://httpbin.org/stream/10") as response:
    for text_chunk in response.iter_text():
        print(text_chunk, end="")

• iter_lines() streams responses line-by-line (mostly used for newline JSON data).

import httpx

with httpx.stream("GET", "https://httpbin.org/stream/5") as response:
    for line in response.iter_lines():
        print(line)

Timeouts and Retries

HTTPX uses a default timeout of five seconds of network inactivity to prevent hanging requests. You can configure timeouts manually for a single GET, PUT, or POST request.

import httpx
response = httpx.get("https://httpbin.org/delay/3", timeout=10.0)

It's also possible to set a default timeout for all requests, such as GET, PUT, and POST requests, on a client.

import httpx

client = httpx.Client(timeout=10.0)  # All requests use 10s timeout
response = client.get("https://httpbin.org/get")

You can even disable timeouts entirely, but it's generally not recommended and should be used cautiously.

import httpx

# For a single request
response = httpx.get("https://httpbin.org/delay/10", timeout=None)

# For all requests on client
client = httpx.Client(timeout=None)

Using httpx.Client lets you set default timeouts once and reuse the same connection pool across multiple GET, PUT, or POST requests. Httpx.Client is most useful for building a retry logic that implements exponential backoff for use cases like web scraping JSON data.

import httpx
import time

def fetch_with_client_retry(url, max_retries=3):
    # Create client with 10-second timeout for all requests
    with httpx.Client(timeout=10.0) as client:
        for attempt in range(max_retries):
            try:
                response = client.get(url)
                response.raise_for_status()
                return response
            except (httpx.TimeoutException, httpx.RequestError) as exc:
                wait_time = 2 ** attempt  # 1s, 2s, 4s
                print(f"Attempt {attempt + 1} failed: {exc}")
                
                if attempt < max_retries - 1:
                    print(f"Retrying in {wait_time}s...")
                    time.sleep(wait_time)
                else:
                    print("Max retries reached.")
                    raise

# Usage
response = fetch_with_client_retry("https://httpbin.org/delay/3")
print(response.status_code)

Redirects and History

HTTP status codes, like 301 or 302, tell the client to request a different URL. HTTPX handles them differently than Requests by also providing tools to track these errors, but not following redirects by default. You must enable them with follow_redirects=True, which can be done for each GET, PUT, or POST request separately:

import httpx

response = httpx.get("https://httpbin.org/redirect/3", follow_redirects=True)
print(f"Final URL: {response.url}")
print(f"Status: {response.status_code}")

Or for all GET, PUT, or POST requests on a client:

import httpx

client = httpx.Client(follow_redirects=True)

response = client.get("https://httpbin.org/redirect/3")
print(response.url)

To see which URLs were visited during the redirect chain, you can use response.history, which will list the response client status, number of redirects, and final URLs.

Asynchronous HTTP With HTTPX

In Requests, a synchronous flow sends a GET, PUT, or POST request and waits for its response before sending another one. An asynchronous flow, enabled by HTTPX, allows your program to perform other tasks, including sending other GET, PUT, or POST requests, while waiting for a response.

Asyncio is Python's built-in library for writing asynchronous code, and HTTPX has a special AsyncClient built from the ground up. Technically, asyncio can be used with Requests as well, but the asynchronous event loop will still be blocked.

Making Async Requests

Using the async with statement, you can automatically handle setup and cleanup of resources, such as opening and closing connections for GET and POST requests. Such context management ensures that resources are properly released even if an error occurs.

A simple async GET request uses Async and await. It follows the same pattern as synchronous counterparts by adding query parameters via the params= argument.

import httpx
import asyncio

async def search_users(query):
    async with httpx.AsyncClient() as client:
        response = await client .get(
            "https://httpbin.org/get",
            params={"q": query, "page": 1}
        )
        return response.json()

result = asyncio.run(search_users("alice"))
print(result)

Sending JSON data with a POST request asynchronously follows the same syntax as synchronous requests, as HTTPX handles all the complexity internally.

import httpx
import asyncio

async def create_user():
    user_data = {
        "name": "IPRoyal",
        "email": "[email protected]",
        "role": "admin"
    }
    
    async with httpx.AsyncClient() as client:
        response = await client.post(
            "https://httpbin.org/post",
            json=user_data
        )
        print(f"Status: {response.status_code}")
        print(response.json())

asyncio.run(create_user())

Concurrent Async Requests

HTTPX can run multiple requests and other tasks in parallel with asyncio.gather() function. It uses tasks as a list of async functions that haven't run yet, then starts them all at the same time and waits for all to finish before returning a list of results in order of completion.

import httpx
import asyncio
import time

async def fetch_url(client, url):
    response = await client.get(url)
    return response.json()

async def fetch_multiple_urls():
    urls = [
        "https://httpbin.org/delay/1",
        "https://httpbin.org/delay/1",
        "https://httpbin.org/delay/1",
        "https://httpbin.org/delay/1",
        "https://httpbin.org/delay/1",
    ]
    
    async with httpx.AsyncClient() as client:
        tasks = [fetch_url(client, url) for url in urls]
        results = await asyncio.gather(*tasks)
        return results

start = time.time()
results = asyncio.run(fetch_multiple_urls())
end = time.time()

print(f"Fetched {len(results)} URLs")
print(f"Total time: {end - start:.2f} seconds")

In our example, all five requests will complete in about one to two seconds since they run in parallel. Compare that to a synchronous version, which would take at least five seconds. Each request takes a second, and the program waits for the previous one to finish before starting the next.

The performance difference becomes significant in real-world tasks like scraping large targets with lots of JSON data.

Real-World Usage and Tips

HTTPX excels in production scenarios where developers need higher performance of HTTP/2 and more flexibility when choosing between async or sync code. Since its logic is also quite similar to Requests, it's quickly gaining popularity for various complex projects.

• API automation. HTTPX's timeout configuration and retry logic prevent failures from breaking the entire automation infrastructure.
• Web scraping. HTTPX's async and HTTP/2 support with asyncio.The gather() function enables scraping hundreds of pages or lots of JSON data in seconds instead of minutes.
• Microservices communication. HTTPX's HTTP/2 support and connection reuse reduce latency and resource consumption in service-to-service communications.

Getting the most out of HTTPX depends on your use case. For example, using the same AsyncClient with connection pooling is beneficial when extracting thousands of pages from a single JSON data source, but it may be counterproductive when implementing APIs. Yet, there are some general tips for using HTTPX.

• Set explicit timeouts as they prevent requests from hanging indefinitely when there are server, network, or other issues.
• Handle specific exceptions separately, so you can retry some while logging permanent ones for later debugging.
• Start synchronous and migrate to asynchronous code only when needed, as asynchronous code will only add unnecessary complexity when not fully used.
• Use single client instances for related requests to enable connection pooling automatically without requiring you to manage it.

Conclusion

We covered the basics to start using HTTPX with GET, PUT, and POST requests in synchronous and asynchronous projects. Some details, such as DELETE requests, can be learned later. For beginners, it's best to start with practicing error handling and timeouts in simpler projects.