Axios Retry: Handling Failed Requests With Examples

Axios is a promise‑based HTTP client library that enables sending requests and, by extension, performing web scraping with Node.js. Failed requests and various server errors are one of the most common hurdles in such use cases. Writing manual retry logic is complicated, so solutions like the Axios Retry library can be a lifesaver.

It’s a separate plugin that hooks into Axios’s request-response logic and adds configurable automatic retry attempts for failed requests. It works seamlessly with Axios requests, allowing easy integration into modern JavaScript workflows. All you need is to learn how to set it up and implement the retry logic.

What Is Axios Retry and Why Use It?

Axios has no built-in retry logic for failed requests, so when a request returns an error, it’s simply rejected, and there’s no automatic retry attempt. You could use JavaScript’s built-in try…catch statement, but this would require writing lengthy manual retry logic.

The Axios Retry library is a much more efficient alternative. It’s an NPM package that intercepts failed requests and provides a pre-built retry logic. Axios Retry is highly useful for web scraping and similar use cases where errors like 503 service unavailable aren’t permanent and might be solved by simply trying again.

One of the main benefits of Axios Retry is the number of settings for configuring retries. You can set up automatic custom retry logic based on HTTP or server error codes, the number of retries, a backoff delay, and configure retries in many other ways.

While web scraping, you send requests to thousands of URLs over a short timespan. Network issues will be frequent, whether it’s due to the client or the server. A simple while loop cannot be used with Axios interceptors, and you rarely want to retry immediately, delaying retries with exponential backoff instead.

Using Axios with the Axios Retry plugin can make web scraping more efficient while keeping the code simple. Network, HTTP, and server errors that meet the retry conditions will be retried automatically. As such, failed requests are reduced while the scraper continues processing other URLs.

How to Install and Set Up Axios Retry

First, make sure you have Node.js installed on your device. If you don’t, follow the official instructions to download and install it. If you already have NVM installed, you can just install the latest Node.js version:

nvm install 22.20

Create a new project folder, initialize it with npm init, and create an index.js file. If you prefer the terminal:

mkdir axios_retry
cd axios_retry
npm init
touch index.js

After that, install Axios and the Axios Retry plugin with the following terminal command. Axios will be used for making web requests, while Axios Retry will be used to help you with automatic retries on network failure:

npm i axios axios-retry --save

Here's an example package.json file:

  "name": "axios-retry-requests",
  "version": "1.0.0",
  "description": "Use axios retry requests to make your app more resilient to network errors"
  "main": "index.js"
  "scripts": {
    "test" "echo \"Error: no test specified\" && exit 1"
  }
  "author": "",
  "license": "ISC",
  "dependencies": {
    "axios": "^1.12.2",
    "axios-retry": "^4.5.0"
  }
}

Basic Retry Example

We can then set up a basic retry system:

const axios = require('axios');
const axiosRetry = require('axios-retry').default;


axiosRetry(axios, {
  retries: 3,
  retryCondition: axiosRetry.isRetryableError, 
});

axios.get('https://quotes.toscrape.com')
  .then(response => console.log(response.data))
  .catch(error => console.error('Failed after retries:', error));

Copy-paste the code and run node index.js in your (IDE) terminal.

The axiosRetry function attaches retry configurations to an existing Axios instance using key-value pairs.

The retries property sets the number of retry attempts, and retryCondition determines when to retry based on network errors.

If a network error occurs, retry the Axios request up to three times before failing. The axios.get method sends a GET request, logging the response if successful or an error message if all retries fail. Request Aborted errors, caused by user cancellations, are intentionally excluded from retries.

Axios Retry With Exponential Backoff

An exponential backoff strategy is essential for web scraping projects. It continually increases the delay before sending the next retry. Exponential backoff strategies are helpful if a server is temporarily overloaded:

const axios = require('axios');
const axiosRetry = require('axios-retry').default;

// Configure axios-retry with exponential backoff
axiosRetry(axios, {
  retries: 3,
  retryDelay: axiosRetry.exponentialDelay,
  retryCondition: axiosRetry.isRetryableError,
});

axios.get('https://quotes.toscrape.com')
  .then(response => console.log(response.data))
  .catch(error => console.error('Failed after retries with exponential backoff:', error));

We only adjusted our axiosRetry object to include a new property, retryDelay. It uses the default Axios requests retry backoff delay, which is 2^n * 100 ms, where n is the number of retry attempts initiated.

If you need a custom backoff strategy, the value axiosRetry.exponentialDelay in Axios interceptors can be modified to refer to a custom function:

const axios = require('axios');
const axiosRetry = require('axios-retry').default;

// Custom exponential backoff function
function customExponentialBackoff(retryNumber) {
  // Start with 200ms delay, double each retry
  return 100 * Math.pow(2, retryNumber);
}

// Configure axios-retry
axiosRetry(axios, {
  retries: 3,
  retryDelay: customExponentialBackoff, // Use custom backoff function
  retryCondition: axiosRetry.isRetryableError,
});

// Make a GET request
axios.get('https://quotes.toscrape.com')
  .then(response => console.log(response.data))
  .catch(error => console.error('Failed after retries with custom exponential backoff:', error));

Customizing Retry Conditions

Axios Retry provides various ways to modify the retry conditions. One of the most useful is to modify the retry logic to apply only to specific error codes.

const axios = require('axios');
const axiosRetry = require('axios-retry').default;

// Custom retry condition
axiosRetry(axios, {
  retries: 4,
  retryCondition: (error) => {
    return error.response ? error.response.status === 503 : false;
  },
});

axios.get('https://quotes.toscrape.com')
  .then(response => console.log(response.data))
  .catch(error => console.error('Failed after retries on 503:', error));

In the example above, we check whether the error status code equals 503 (Temporarily Unavailable). If so, we retry. Some errors, such as 429 (Too Many Requests), come with a hint from the server on how long you should wait before retrying, allowing you to build a more customized logic.

When a server sends a 429 error, it includes a retry-after header informing about when a new request should be sent again. As such, we can modify our retry logic to reflect what the server responded to in the retry-after header.

const axios = require('axios');
const axiosRetry = require('axios-retry').default;

axiosRetry(axios, {
  retries: 3,
  retryCondition: (error) => {
    return error.response && error.response.status === 429;
  },
  retryDelay: (retryCount, error) => {
    const retryAfter = error.response.headers['retry-after'];
    
    if (retryAfter) {
      // If the header is a number (seconds), convert to milliseconds
      return parseInt(retryAfter) * 1000;
    }

    // If no 'Retry-After' header exists, we default to 0 (immediate retry)
    // or you could set a fixed number like 1000 (1 second).
    return 0; 
  }
});

axios.get('https://quotes.toscrape.com')
  .then(response => console.log('Success:', response.data))
  .catch(error => console.error('Stopped after retries:', error.message));

Another useful way to retry Axios requests is by modifying the logic to apply to specific HTTP methods that you suspect might fail. Here's an example of how it might be implemented to POST.

const axios = require('axios');
const axiosRetry = require('axios-retry').default;

// Configure axios-retry to apply only to POST requests
axiosRetry(axios, {
  retries: 3,
  shouldRetry: (config) => {
    return config.method === 'post';
  }
});

axios.post('https://quotes.toscrape.com', { data: 'example' })
  .then(response => console.log('Data submitted:', response.data))
  .catch(error => console.error('Failed after retries for POST:', error));

Axios Retry for Web Scraping

Unlike sending requests to your own database or an API, web scraping targets might want to block your access. As such, HTTP and server errors are more often, but also more temporary. Some of the most common failures while web scraping can be solved with proper Axios retry logic implementation.

• Rate limiting. Websites limit how many requests you can make per minute. If you hit this limit, they might send a 429 (Too Many Requests). Sometimes they may ban you without returning 429. In any case, implementing a retry delay logic allows you to wait out the penalty and collect the data.
• Server and network errors. Scrapers send thousands of requests at a much faster rate than a human visitor could. Even without intentional rate limits, you'll face 5xx or timeout errors that can be solved with later retry attempts.
• Proxy instability. Cheap, unreliable, or improperly set up proxies might crash mid-request, causing a 502 connection reset or other errors. Setting up proxies with Axios properly and using quality residential IPs solves such problems.
• Anti-bot systems. Systems like Cloudflare are known to block your connection request to check browser credentials briefly. Another attempt might pass if you rotate proxies or solve the CAPTCHA challenge.

Without a retry logic, a strict rate limit, or network issues could crash your scraper or leave you with an incomplete dataset. It's recommended to use exponential backoff logic, rotate proxies and user agents, with libraries like fake-useragent, as well as handle each case of HTTP errors differently.

Conclusion

Axios Retry is an excellent plugin if you need a simple and intuitive way to retry failed requests automatically. The best part is that it's not that difficult to use, as Axios Retry is well-supported by the community, with plenty of examples and discussions available online.

FAQ