How to Set Up Proxies with Screaming Frog SEO Spider

Screaming Frog is one of the best free web crawling tools, known for its SEO-related features. Whether you'll be using it to optimize your website or for some other crawling tasks, you'll need proxy servers to make it efficient. Luckily, Screaming Frog's proxy integration is built into the tool.

What Is Screaming Frog SEO Spider?

Screaming Frog SEO Spider is one of the most powerful website crawling and auditing tools. Although it's mainly targeted for SEO professionals, webmasters, and digital marketers in general find it useful for analyzing and optimizing websites.

It helps to create an effective SEO strategy by detecting broken links and redirects, duplicate and missing content analysis. Screaming Frog can also be easily used with third-party tools like Google Analytics for more comprehensive SEO assessments.

Screaming Frog performs some custom data extraction, which allows it to function as a basic web scraper in some simpler use cases. For large-scale data collection, however, you might need the paid version as the free one has a crawl limit of up to 500 URLs.

Why Use Screaming Frog with Proxies?

A comprehensive SEO strategy requires all-encompassing data, and collecting it with tools like Screaming Frog is bound to create various challenges. Many of them can be avoided by using residential proxies. These proxy servers will supply you with quality IP addresses for avoiding blocks, CAPTCHAs, and rate limits competitor sites might impose.

The data you collect while using proxies is often more accurate, since you can target specific geographical regions. Collecting data with a few IPs from different countries and then comparing it is crucial for many SEO audits, especially those where localized content can vary by region.

Residential proxies are best for blending in with other visitors and staying anonymous while crawling sites since they originate from ordinary home connections. For some websites, other proxy types, such as datacenter ones, can also work while providing faster speeds and lower costs.

How to Set Up IPRoyal Proxies with Screaming Frog

1. Download and Install Screaming Frog

Start by downloading Screaming Frog's SEO spider tool if you haven't already. Choose the version for your operating system - Windows, Linux, or macOS. Then, simply follow the installation steps.

2. Activating Paid License (Optional)

After installing and launching the software, you can activate your paid license by clicking on License > Enter License. You'll need to type in your License Username and License Key in the fields shown below.

3. Screaming Frog Proxy Integration Settings

Navigate to File> Settings> Proxy and tick the checkbox Use Proxy Server and Use Proxy Credentials as shown below.

The next step requires entering your Proxy address, port, username, and password. These proxy details can be easily found in IPRoyal's dashboard.

4. Entering Your IPRoyal Proxy Details

We'll be using residential proxies as an example, but any other proxies follow the same general steps. See our quick start guide for purchasing and configuring your proxies for the first time. In short, the needed proxy credentials (port, username, and password) can be found in the dashboard.

Proceed to enter your IPRoyal proxy details in the relevant fields and click the button OK and Restart in the lower right corner. Screaming Frog will restart, and you can test your proxy IPs by crawling.

5. Testing your Proxies

To test whether your proxies work, enter a URL for the Screaming Frog to crawl, for example, https://www.screamingfrog.co.uk/. If you're using other websites, be sure to check the third party's policies and robots.txt rules for crawling it. Once a URL is entered, click Start.

If your proxy is working and the crawl was successful:

• You'll find a list of URLs in the Internal tab in the main window (up to 500 for the free version).
• The status code column will mostly show 200 OK statuses.
• The bottom right status bar will take some time to complete and increase steadily.

If the proxy isn't working:

• No URLs are displayed in the Internal tab.
• Crawl might not start at all, or the bottom right status bar will complete instantly.
• You'll find a high number of blocked requests with various status codes, such as 403 Forbidden.

If you're still unsure whether Screaming Frog uses your proxy IP address, you can run another test crawl on our IP lookup tool or a third-party website that displays your IP address. If your proxy is working, you’ll see the proxy IP inside the crawled page’s content as a rendered page text.