THE LAB #1: Scraping data from an app
How to inspect the network traffic of an app with Fiddler Everywhere and scrape the data from its servers.
This is the first post of “THE LAB”: in this series, we'll cover real-world use cases, with code and an explanation of the methodology used.
In the future, this kind of content will be available only to paying subscribers. Being the first of the series, this one will be available for free until Sunday 11th of Sept 2022, then will be behind a paywall.
The Web Scraping Club is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.
I usually write in this newsletter about how to extract data from websites but what if our target is an app with no web interface? Or if the website is too complex to scrape and, knowing the target has an app, we want to test if there's another door to access the data.
Since we cannot see the source code of the app, we must intercept the requests made from it to its servers and try to replicate them in our scraper.
To do so, we need a network traffic analysis tool. In this post, we'll use Fiddler Everywhere to support us, but there's plenty of similar software on the market, like Wireshark.
Keep reading with a 7-day free trial
Subscribe to The Web Scraping Club to keep reading this post and get 7 days of free access to the full post archives.