The Web Unblocker Cost Benchmark
Price comparison between the most well-known web unblockers on the market
As you surely know, web scraping is becoming harder and expensive, with more websites protected with anti-bots. To bypass them, we usually have two paths: create our own custom web scraping solution, dedicated to the website, or delegate to an external tool for this extraction.
Nowadays there are several "super APIs” available on the market, with a set of features like changing the browser fingerprint, proxy management, HTML rendering, and so on, with some different approaches on the backend.
Their success rate is similar, as we have seen in the Hands On series, with some differences based also on the maturity of the products. These tests were made in the past months and things may have changed in this period of time since anti-bots are in continuous development and the results may have changed from the initial test (SPOILER: that’s why I’m thinking for 2024 about a recurring edition of the tests).
Besides the success rate against a given target website, another criterion for choosing one of the solutions is the price for a given web scraping project. But comparing these costs across different vendors is not that simple: vendors use two different metrics: pay-per-GB and pay-per-request.
Both metrics are difficult to estimate before starting a web scraping project, while if you’re already scraping a website and have a good logging system, you should be able to have clearer ideas on the total costs. But before comparing the costs on some selected target websites, let’s have a brief explanation of the pricing models of different “super APIs”, in alphabetical order.
Disclaimer: The following prices were collected on the 8th of December 2023, when the article was written. Since the pricing structure is not always so easy to understand, please refer to the customer support of the companies if you want to have more details.
Bright Data Web Unlocker
The Bright Data Web Unlocker I think it’s the oldest and most mature tool in this selection, and we’ve seen it in its review where it could bypass every anti-bot of the test.
The pricing model is Pay-per-request, with different plans and a “premium features” plan, which is triggered when we ask to scrape a certain list of websites available here. I’ve not found a common rule for these websites, since I see some of them are protected by Kasada, and others with PerimeterX or Akamai even if they’re quite easy to bypass.
Anyway, for this price comparison, we’ll use the Pay as You Go plan, without advanced features since the websites we’re going to estimate the costs of are not in the Premium list. We’ll use for the estimates the 3$ per 1k request price.
As the holiday season approaches, Bright Data prepared for you this offer: using this link, if you deposit 50$ or more you’ll get the same for free.
Infatica web scraper
Infatica web scraper just entered the arena of “super API” but its performance was quite good from the first moment, as shown by our test.
One of its strengths is the affordable pricing strategy to enter the market, even if it’s not that plain to understand.
We’re buying credits with every plan we purchase but a request could cost a different amount of credits depending on some parameters, as we see from the documentation.
In order to benchmark the costs of the solutions using the same features, we’ll suppose our requests will always use the render_js option, in order to get the fully rendered website’s pages.
Using the smallest plan available means we’ll have to pay 25$ per 25K requests with HTML rendering.
Also, Infatica wants to wish you a great holiday season with the following discount codes:
Webscrapingclub20 - 20% discount on the first purchase of the web scraper plans.
Nimble Unblocker Proxy
We’ve already met Nimble during our AI month, where we understood more about what’s happening in the web scraping industry with the advent of AI. While in the past we’ve tested their E-commerce API with great results, in this benchmark we’re assessing the costs of their Unblocker Proxy.
Also in this case we’re buying credits that can be used for every Nimble service but this time the cost of the Unblocker Proxy is fixed at 2.5$ per 1k requests.
Nimble wants you to celebrate this holiday season with this great offer: By using the code "nimblexmas555", you can get $500 worth of quota when you sign up! This offer is valid until the 5th of January!
Oxylabs Web Unblocker
The Oxylabs Web Unblocker is the Oxylabs “super API” for advanced web scraping and we’ve tested it in our Hands-On article with great results.
The pricing plan is Pay per GB, starting with 5 GB per month, and for this test, we’ll use the Micro plan, with a cost of 15$ per GB.
Oxylabs is happy to share with you a discount code twsc35, which offers a 35% discount and is valid for Micro, Starter, and Advanced Web Unblocker plans.
Smartproxy Site Unblocker
Also, Smartproxy entered the “super API” arena recently with their unblocker, with great results in bypassing the well-known anti-bots, as we have seen in our previous review.
Also in this case the pricing model is per GB, in particular 14$ per GB.
Smartproxy for this holiday season wants to share with you the following discount code WEBSCRAPINGCLUB10 with a 10% discount on the plans.
Zyte API
At the latest Extract Summit Zyte presented the updated version of their Zyte API, which we tested in its earlier release some months before.
As you can see from the previous table, prices are variable according to the difficulty of the target websites, so I asked Daniel Cave, head of product marketing at Zyte, for a bit of help with the price to use in this benchmark.
He shared with me some insights about a study they made internally at Zyte, based on the top 250000 websites on the web.
Their distribution over the five clusters is the following:
85.60% fall into tier 1-3. (average $1.20)
9.05% fall into tier 4
5.34% fall into tier 5
These stats lead to a weighted average price of 1.974 USD per 1k request.
Benchmark #1: E-commerce product list page, HTML download
The first example for this cost benchmark is Zalando, one of the largest European e-commerce websites, with approximately 1 Million items on sale in Germany.
According to our internal stats, it’s one of the most demanding websites in terms of GB we’ll scrape. Using the product list page to gather all the data, without entering the single product detail page, our scrapers extract 31GB of data and 30k requests.
Here’s how much it would cost to scrape the whole website with every “super API”.
Of course, this large volume of GB requested makes the unblockers billing per request more convenient, until we stop at the product list page level.
Benchmark #2: E-commerce detail page, HTML download
In this second example, we’re targeting the Zara USA website where we need to crawl each page and download data from there.
According to our internal stats, each full website scrape requires 1.6 GB and 44k requests.
The proportion between GB and requests in this case is more favorable to the Pay-per-GB models. As a side note, Zara is on the “premium” list of Bright Data websites but since it’s an illustrative example I’m keeping the standard price.
Final remarks
As we have seen from the previous benchmarks, the cost efficiency of each solution depends on the context and the single project. There are some use cases with a high volume of requests which make solutions billed per GB convenient and vice-versa.
Some of the variables to consider are difficult to know before starting a scraping project. While the number of requests can be deducted after studying the website, the estimate of the number of GB is much more difficult to calculate.
What is clear from this test is that a large web scraping project probably needs more than one provider to minimize the costs of web scraping and have more tools available to bypass the anti-bots.
To celebrate together this holiday season, also The Web Scraping Club has prepared for you two offers:
Group Annual subscriptions: 30% off for groups of more than 2 people, forever. Read The Web Scraping Club together with your friends and colleagues and save money! Valid only until the 6th of January.
Annual subscriptions: 20% off, forever, on your The Web Scraping Club annual subscription. Redeem by the 6th of January.