Node-Fetch Proxy Tutorial for 2025

Node-Fetch is a popular library that brings the Fetch API to Node.js. With it, you can connect to pages, send post data, and request contents, which makes it a suitable tool for many tasks, including Node.js web scraping . In addition, Node.js includes default support for the fetch function since v18.

But there’s a problem. Out of the box, there’s no node-fetch proxy option. Therefore, you can get blocked quickly when using only your own IP address.

How Do I Use a Node-Fetch Proxy?

There are two main ways to use a node-fetch proxy. You can use a code library that creates a custom user agent, or you can use a reverse proxy .

How to Use Https-Proxy-Agent

One of the options to implement a node fetch proxy is to use a code library for a custom user agent. A popular library to implement this is https-proxy-agent by Nathan Rajlich .

In terms of implementation, it’s quite simple. You just need to install node fetch (you don’t need it if you are running NodeJS v18 or higher) and https-proxy-agent:

npm install node-fetch
npm install https-proxy-agent

Then, in your code, use the HttpsProxyAgent as the agent parameter of your fetch request. Like this:

import fetch from 'node-fetch';
import { HttpsProxyAgent } from 'https-proxy-agent';

(async () => {
    const proxyAgent = new HttpsProxyAgent('http://geo.iproyal.com:12321');
    const scrape = await fetch('https://ipv4.icanhazip.com/', { agent: proxyAgent });
    const html = await scrape.text();
    console.log(html);
})();

Notice that this code doesn't use any proxy server authentication. So if you want to use it this way, you need to whitelist your own IP address.

You can pass the authentication details to HttpsProxyAgent using a few methods. The simplest one is using the username and password as plain text in your request, like this:

async () => {
    const proxyData = new HttpsProxyAgent('https://username:[email protected]:12321');
    const scrape = await fetch('https://ipv4.icanhazip.com/', { agent: proxyData });
    const html = await scrape.text();
    console.log(html);
};

Other options for node-fetch proxy authentication with https-proxy-agent also exist, such as using auth or custom headers.

Web Scraping Without Getting Blocked With a Node-Fetch Proxy

Although web scraping is legal and very useful, many sites try to block it. They use a few tools to do it, but two of the biggest points are the IP address and the headers.

They check if a single IP address is performing a large number of requests or if they are done around the same time. This is a telling sign that it's a bot and not a real user.

Regarding the headers, they check if the request metadata looks like a request made from a real browser. If you don't use any parameters, a bot will just request the URL without sending any information.

However, a real user sends a lot of data about themselves, such as browser, version, language and more. Therefore, requests without this metadata are quite suspicious.

You can fix the IP detection by using IPRoyal's residential proxy service . With it, you connect using different IP addresses from real residential users around the world. Website owners won't be able to tell that two different requests with different IP addresses are from the same user, so you'll be left alone.

In addition to authenticated requests, you can whitelist an IP address if you want. This feature allows you to use your proxy server without sending a username and password.

To improve your scraping efforts, follow these best web scraping practices :

• Rotate IPs regularly. Use services that offer multiple residential IPs to reduce the chance of bans.
• Handle CAPTCHAs. Use headless browsers or CAPTCHA-solving services.
• Implement retry logic. If a request fails, set up a retry logic that waits for a bit and then tries again.
• Respect robots.txt. It reduces your risk of getting blocked or blacklisted.

Node-Fetch Proxy Authentication Methods

When using a proxy, it’s important to use the right authentication method depending on your proxy server setup. Here’s an overview for basic, digest, and SOCKS5 authentication with credentials in a node-fetch module context.

1. Basic Authentication (Username/Password)

const { HttpsProxyAgent } = require('https-proxy-agent');

(async () => {
  const proxyAgent = new HttpsProxyAgent('http://username:[email protected]:8080');
  const response = await fetch('https://ipv4.icanhazip.com/', { agent: proxyAgent });
  const html = await response.text();
  console.log(html);
})();

2. Digest Authentication

Digest needs extra handling. You can use digest-fetch to help:

const DigestFetch = require('digest-fetch');
const { HttpsProxyAgent } = require('https-proxy-agent');

const client = new DigestFetch('username', 'password');

(async () => {
  const proxyAgent = new HttpsProxyAgent('http://proxy.example.com:8080'); // optional proxy
  const response = await client.fetch('https://protected.example.com', { agent: proxyAgent });
  const data = await response.text();
  console.log(data);
})();

3. SOCKS5 Proxy Authentication

To use SOCKS5, you can use socks-proxy-agent:

const { SocksProxyAgent } = require('socks-proxy-agent');

(async () => {
  const proxyAgent = new SocksProxyAgent('socks5://username:[email protected]:1080');
  const response = await fetch('https://ipv4.icanhazip.com/', { agent: proxyAgent });
  const html = await response.text();
  console.log(html);
})();

How Do I Use Node-Fetch Proxy User Agents?

You can set user agents via the options argument in your node fetch proxy call. So instead of the simple fetch request:

const scrape  await fetch'https://ipv4.icanhazip.com/';

You can do it using the options argument after the URL:

const scrape  await fetch'https://ipv4.icanhazip.com/' { headers: { /** request headers here **/ }} ;

Therefore, in addition to user agents, you can use other header arguments. Here is a code sample using node-fetch proxy and custom user agents at the same time:

async  => {
    const proxyData  new HttpsProxyAgent'https://username:[email protected]:12321';
    const options = {
        agent: proxyData
        headers: {
            'User-Agent' 'Mozilla/5.0 (Macintosh; Intel Mac OS X 13_0_1) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.1 Safari/605.1.15'
        }
    };

    const scrape  await fetch'https://ipv4.icanhazip.com/' options ;
    const html  await scrapetext;
    consoleloghtml;
};

In this case, we are using just the user-agent, but you can pass any headers you want.

Conclusion

Now you have a better understanding of how to use node fetch proxies to scrape websites without getting blocked. In addition, you saw how you could use custom headers to make your scraping functions even better.

Now you can connect your scraper with a parser and extract data from any site you want.

FAQ

const fetch = require('node-fetch');

npm install -g npmnpm cache cleannpm update

npm i node-fetch

import fetch from 'node-fetch';

const fetch = (...args) => import('node-fetch').then(({default: fetch}) => fetch(...args));