What's a proxy server?
Hi, this is Pierluigi from The Web Scraping Club, a newsletter where you can find news, insights, and tutorials with real-world examples about web scraping.
Being a paying user gives:
Access to Paid Content, like the post series called “The LAB”, where we’ll go deep diving with code real-world cases (view here as an example).
Access to the GitHub repository with the code seen on ‘The LAB”
Access to private channels on our Discord server
But in case you want to read this newsletter for free, you will always get a post per week about:
News about web scraping
Anti-bot software and techniques insights
Interviews with key people in the industry
And you can always join the Web Scraping Club Discord server
Today we'll cover in detail one brick of a modern web scraping architecture we've seen in this previous post, the proxies.
What is a proxy server?
Straight from Wikipedia, "In computer networking, a proxy server is a server application that acts as an intermediary between a client requesting a resource and the server providing that resource".
In other words, is a server application that stands between you and your target server: you can send a request to the target server through the proxy server, so the machine that sends the request to the target server is the proxy and not your client.
Why do we need proxies for web scraping?
There are several ways proxies can be useful when scraping.
The target website has a sort of geo-fencing, so it shows data depending on the location of the IP address. This is a common case and choosing a proxy with an address belonging to the country we need the data from is quite a common case.
The target website limits the requests from the same IP, in this case, we'll need a pool of proxies or a rotating proxy to split the requests into several different IPs
The target website uses anti-bot software that employs fingerprints to block traffic from servers. If our scraper is running in a cloud environment, we'll need a residential or mobile proxy to bypass it.
The target website blocks all the requests coming from the IP range of our cloud provider. In this case, we'll need an Elite Proxy to mask the origin of the requests.
Different types of proxies
As we have seen in the examples before, proxies can be categorized using the feature we want to highlight.
Usually, when comparing the different proxies vendors or when we find online a list of free proxies, the main features are anonymity, its location, and if it's static or rotating.
Proxy anonymity
Usually, this information is provided in the free online proxies lists like this one, where the proxies are divided in:
Transparents or No Anonymous. So-called because they do not change the starting request headers. On top it adds the "X-Forwarded-For" header containing the origin IP and the "Via" header saying that a proxy is being used.
Anonymous: Same as before but without the "X-Forwarded-For" header
Highly Anonymous or Elite: it replicates the original headers without adding anything.
Proxy location
Every IP can be traced back to a country of provenience, so as e-commerce websites can geo-fence any IP, also proxy providers usually sell proxies located in the main countries of the world.
On top, the most important players in the field allow their customers to choose the final location of the proxies, like a data center, a mobile phone, or a residential IP.
Data center IPs are the most common and cheap but also they are likely to get banned from websites that use strong fingerprinting techniques to block bots or IPS that are in a well-known range of data center addresses.
From the tests on https://incolumitas.com/, we can see how easy is to detect a data center IP.
I made a test, setting up a Highly Anonymous Proxy on an Ec2 AWS instance and this is the result of the IP API.
As you can see, combining data from several sources and public registries, it can be easily detected that the request is coming from an EC2 instance.
The TCP-IP Fingerprint test also gives interesting results.
These are the results when browsing from my Mac without using proxies. It can be detected with a fair degree of certainty that I'm using a Mac Os operating system and this is because every operating system uses different values inside the TCP Handshake packet sent when establishing a connection with a server.
In fact, when turning on the proxy on my Unix server, the results of the test change, and, with such an uptime value, it's a red flag signaling that the request is coming from a server and not a personal client.
To mimic a more realistic real user sending requests to a website, the most important proxy providers give the option to use proxies with residentials IP, so installed outside data centers, or mobile proxies.
Of course, due to scarcity and the cost of the bandwidth, these options are more expensive than the data center's one.
Static Vs Rotating
Using a static proxy means that your requests will be made all from the same IP of the proxy. This doesn't help in case the website limits the number of requests per IP in a certain timeframe.
The solution is a rotating proxy: it means that you set up your scraper as if you're using only one proxy but, in the background, every request sent to its address is routed to a pool of proxies.
The target website will see these requests coming from different IPs and machines and will not trigger any block.
Where to find the right proxy for me?
There ain't no such thing as a free lunch, even in this field.
For a list of free proxies, you can have a look here but tendentially they are unreliable and slow.
If you're willing to pay for proxies, there are many actors in this industry, the most famous are:
Key Takeaways
Sooner or later, you'll need to use proxies, as more and more websites are becoming harder to scrape.
Choosing the right one depends from website to website, there's no silver bullet (or maybe yes but it's too expensive).
For today it’s over, please let me know in the comments what you think about this post and feel free to join our community of web scrapers professionals on Discord if you want a more direct interaction.
Just as a reminder, for our readers, we have a special discount code 50% off for the Extract Summit happening the next 29th of September in London: use the code thewebscrapingclub when buying your tickets.
Is any of our you working on something spectacular in web scraping and want to share with us? please write to pier@thewebscraping.club and let’s talk about it! You could be in the next interview.