How to Use Fetch with Custom Headers in Node.js (GET & POST)

Fetch API is a modern JavaScript interface for making HTTP requests in Node.js . Learning to use custom GET and POST request headers with the Fetch API is crucial for building a working Node.js scraper that can extract data from the responses.

Fetch API replaced the older XMLHttpRequest method and uses Promises to handle asynchronous requests. Setting up custom headers (like User-Agent, Accept, and Authorization) is much easier with working examples to guide you.

Setting Up Fetch in Node.js

Install the newest version of Node.js for your system or update it if you have an older one. It’s essential to use the 18+ version of Node.js since you won’t need to install Node Fetch API separately.

After installation, open Command Prompt (Windows) or Terminal (Mac/Linux) and verify it with a command.

node -v

Then, check the npm (Node Package Manager) installation with a similar command:

npm -v

If version numbers are visible and there are no errors, Node.js was successfully installed. You can proceed to creating your first project that uses Node Fetch API.

Let's start by creating and opening a project folder in your preferred IDE. If you prefer using the terminal, run the commands below:

mkdir my-scraper
cd my-scraper

Now, let's create a package.json file that tracks your Fetch API project details.

npm init -y

Now that our project is set up, we can create our first JavaScript file and see how Node Fetch API calls look in action. Simply open the folder with an Integrated Development Environment (IDE), such as VS Code , and create a new file called scraper.js there.

The code below uses JSONPlaceholder as a testing environment to send a GET request with Node Fetch API, convert the response body to JSON, log the data, and cache errors if any appear.

// Simple GET request to fetch data from an API
fetch('https://jsonplaceholder.typicode.com/posts/1')
  .then(response => response.json())  
  .then(data => {
    console.log('Data received:');
    console.log(data);
  })
  .catch(error => {
    console.error('Error:', error);
  });

After saving the file, use the following command in the terminal to run the Fetch API scraper script:

node scraper.js

Making a GET Request With Custom Headers

The basic code example we've provided above doesn't use custom Node Fetch headers, which work fine for a test website, but won’t in a real-world scenario. If you don't add custom headers, your Node Fetch request will automatically add the default HTTP headers of your device.

This is problematic when web scraping, as your HTTP requests might be a way to identify you and detect that you're using Node Fetch API to connect. Fetch requests can use a custom HTTP headers object to avoid such issues:

fetch(url, options)

Generally, User-Agent, Referer, and Accept headers are the most important when web scraping. A GET request with a custom agent HTTP header in Node Fetch API might look like this:

fetch('https://httpbin.org/headers', {
  headers: {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36'
  }
})
  .then(response => response.json())
  .then(data => console.log(data.headers))
  .catch(error => console.error('Error:', error));

Often, a website might check your HTTP request to see where you’re coming from. So, we need to add a custom Referer header to the Node Fetch API header:

fetch('https://httpbin.org/headers', {
  headers: {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36',
    'Referer': 'https://www.google.com/'
  }
})
  .then(response => {
    console.log('Status:', response.status);
    return response.text();
  })
  .then(text => console.log(text))
  .catch(error => console.error('Error:', error));

Lastly, add the Accept header that will specify what content type you need. It's important to add content types that an ordinary browser might request to avoid detection of your Fetch API scraper.

fetch('https://httpbin.org/headers', {
  headers: {
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'
  }
})

HTTP headers, such as Accept-Language, Authorization, and others, can be added in the same fashion in an object literal. In some cases, the website might require you to use POST request headers with the Fetch API to access the needed data.

Making a POST Request With Headers and Body

Custom POST requests are most useful when scraping requires interaction with the website. Filling out forms and logins, clicking buttons, or dealing with infinite scroll are common cases when you need POST requests in Node Fetch API.

Sending POST requests is a bit more difficult compared to GET requests. We start by defining a JavaScript headers object using an object literal:

const postData = {
  title: 'My POST Request',
  body: 'My content',
  userId: 1
};

We need to convert the JavaScript object into a JSON string. Without the conversion, the server gets a string representation from the Fetch API in the wrong format. We can do it by using the stringify function:

const jsonBody = JSON.stringify(postData);

Now we can add custom headers to our HTTP request that Node Fetch API uses. For example, the Content-type header, which tells the server we're sending JSON data that needs to be stored.

const postData = {
  title: 'My POST Request',
  body: 'My content',
  userId: 1
};

const jsonBody = JSON.stringify(postData);


fetch('https://jsonplaceholder.typicode.com/posts/', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json; charset=UTF-8'
  },
  body: JSON.stringify({
    title: 'My POST Request',
    body: 'My content',
    userId: 1
  })
})
  .then(response => {
    console.log('Status:', response.status);
    return response.json();
  })
  .then(data => console.log(data))
  .catch(error => console.error('Error:', error));

A similar process goes for other POST requests, such as Cookie, X-Requested-With, Authorization, and others. However, POST requests with Node Fetch API might raise different response statuses and errors. Error handling is needed to avoid crashing your scraper when dealing with response headers.

Node Fetch API treats network errors and HTTP errors differently, so a two-layer approach is required. Try/catch layer detects network errors:

try {
  const response = await fetch('https://jsonplaceholder.typicode.com/posts/1');
  // If network fails, jumps to catch immediately
  
} catch (error) {
  // Catches: network timeout, DNS failure, no internet, CORS
  console.log('Network error:', error.message);
}

An HTTP response headers status layer is needed to catch HTTP errors and give you information about them.

try {
  const response = await fetch('https://jsonplaceholder.typicode.com/posts/1');
  
  if (!response.ok) {
    // HTTP error detected - manually throw to trigger catch
    throw new Error(`HTTP error! Status: ${response.status}`);
  }
  
  // Only reaches here if both network AND HTTP succeeded
  const data = await response.json();
  
}

Error handling is also crucial when working with some methods in the Node Fetch API, such as the HTTP Delete request. Arguably, the consequences of failure with the Delete request are much worse than with GET or POST errors.

Full Working Example

const headers = {
  'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36',
  'Referer': 'https://www.google.com/',
  'Accept': 'application/json, text/plain, */*'
};

async function fetchWithErrorHandling(url, options = {}) {
  const response = await fetch(url, {
    headers,
    signal: AbortSignal.timeout(10000),
    ...options
  });

  if (!response.ok) {
    throw new Error(`HTTP error! Status: ${response.status} ${response.statusText}`);
  }

  return response.json();
}

async function scrapeWithGet() {
  console.log('=== GET Request ===');

  try {
    const data = await fetchWithErrorHandling('https://jsonplaceholder.typicode.com/posts/1');
    console.log(`Title: ${data.title}`);
    console.log(`Body: ${data.body.substring(0, 50)}...`);
    return data;
  } catch (error) {
    handleError(error);
    return null;
  }
}

async function scrapeWithPost() {
  console.log('\n=== POST Request ===');

  const postData = {
    title: 'My Test POST',
    body: 'This is the content of my post sent via Fetch API',
    userId: 1
  };

  try {
    const data = await fetchWithErrorHandling('https://jsonplaceholder.typicode.com/posts', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json; charset=UTF-8' },
      body: JSON.stringify(postData)
    });
    console.log(`ID: ${data.id}`);
    console.log(`Title: ${data.title}`);
    return data;
  } catch (error) {
    handleError(error);
    return null;
  }
}

function handleError(error) {
  if (error.name === 'TimeoutError') {
    console.error('Request timed out');
  } else if (error.cause) {
    console.error('Network Error:', error.cause.message);
  } else if (error.message.includes('HTTP error')) {
    console.error('HTTP Error:', error.message);
  } else {
    console.error('Error:', error.message);
  }
}

async function runScraper() {
  console.log('Starting scraper...\n');

  const getResult = await scrapeWithGet();
  const postResult = await scrapeWithPost();

  console.log('\n=== Complete ===');
  console.log(`GET: ${getResult ? 'Success' : 'Failed'}`);
  console.log(`POST: ${postResult ? 'Success' : 'Failed'}`);
}

runScraper().catch(console.error);

We used const headers as this is better for code reusability, but the fundamentals are the same as in previous steps. It's a complete request-response cycle with a const response variable storing the response body object with the Fetch API. Both GET and POST response bodies are accounted for.

Yet, our example is still meant for learning and testing. It omits certain crucial parts required for scraping a real-world website on a large scale. Here are some of the most crucial things to add to your Fetch API code.

• Proxies. Websites block repeated requests from the same IP. Correctly set up Node Fetch proxies to rotate IPs to avoid detection. Without proxies, most commercial websites will block your Fetch API requests, especially POST operations, that trigger anti-bot systems.
• User-agent rotation. While you'll blend in better using standard settings, large-scale scraping will require multiple browser profiles. It's recommended to use multiple desktop or mobile agents on rotation with Fetch API requests.
• Request delays and rate limits. Sending Fetch API requests without implementing delays is a sure way to ban your IP address. Implement rate limits and respect the website's crawl delay in robots.txt.
• Cookie management. Many sites track visitors for authentication and tracking. A Fetch API web scraper that wants to blend in needs to store Set-Cookie response headers and use them in future requests.
• HTML parsing. Our example website returns clean JSON when the Fetch API is used. A real-world website will require return const response headers that require additional parsing steps, depending on the data you want to extract.
• Timeout handling. Fetch API requests can hang indefinitely if the server is slow or unresponsive. In a real-world scenario, you should adjust the AbortSignal.timeout() based on expected response times of the target server.
• Logging & monitoring. Production scrapers also log and track the success rate of HTTP requests and the performance of proxies. Future versions of the scraper are adjusted accordingly to these logs.

Conclusion

While there are a lot of ways to send POST and GET requests, such as Curl commands, Node Fetch API is one of the best choices for web scraping. With the foundations covered here and some knowledge in JavaScript, you can start building efficient scrapers fit for real-world scenarios.