How to Scrape Data from Mobile Apps With HTTP Toolkit
Why doing it and what challenges we could face in the process
Scraping data from mobile apps can unlock valuable information otherwise difficult or impossible to obtain from websites alone.
This is especially useful for delivery apps (food delivery, courier, etc.), where the mobile app often provides rich, structured data via API calls. In this short guide, I’ll explain why app scraping is essential, how to deal with SSL pinning (a common obstacle) and walk through practical methods to intercept app traffic using HTTP Toolkit.
Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.
Why Scraping Data from Apps is Important
Being capable of scraping data from apps is key for a professional web scraper for mainly three reasons.
First, apps often have data or endpoints that websites don’t. Companies sometimes design their mobile apps to use different API endpoints than their public websites. This means certain information (restaurant details, delivery statuses, etc.) might only be accessible through the app’s network calls. In some cases, there isn’t even a full-featured website available – the service may be app-only, so scraping the app is the only option to get the data.
In the second instance, there are often fewer anti-scraping protections. Many mobile apps fetch content via streamlined JSON APIs without complex web interfaces. Unlike websites that may use heavy JavaScript, dynamic content, and CAPTCHA challenges to deter scrapers, mobile app APIs tend to be simpler and often lack those aggressive protections. In fact, a mobile app’s API calls are usually concise and efficient, and they typically only use basic measures like rate limiting (which can be handled with rotating proxies). This makes data collection from apps easier in many cases. During one of our consultancy tasks, it recently happened that the antibot applied on the website could be easily bypassed when using the app.
Missing data on the website. It’s common for a company’s app to show information not present on their website (or not in an easily accessible format). For example, a food delivery app might display real-time delivery driver locations, special in-app promotions, or detailed menu item data that the website might hide behind interactive elements. If you need this specific data, the app is your go-to source.
Thanks to the gold partners of the month: Smartproxy, Oxylabs, Massive and Scrapeless. They’re offering great deals to the community. Have a look yourself.
Understanding SSL Pinning
Of course, it isn’t all puppy dogs and rainbows, and scraping data from an app brings different challenges. One you’ll almost certainly encounter is SSL pinning. Apps use this security mechanism to prevent man-in-the-middle interception of their network traffic.
But what is SSL pinning? In simple terms, SSL (or certificate) pinning means the app is coded to only trust a specific certificate (or public key) for its server rather than any certificate trusted by the device’s OS. During an HTTPS connection, the app compares the server’s certificate with the one it expects (which is “pinned” or hardcoded in the app). If it doesn’t match, the app refuses the connection.
This remains true even if you install a custom certificate on the device – a pinned app won’t accept it because it’s not the exact one the app expects. In practice, SSL pinning is used to prevent man-in-the-middle attacks, ensuring that only the app’s own backend can decrypt the traffic. It’s an extra security layer beyond standard HTTPS. Apps that handle sensitive data (like banking or payment apps) or apps whose developers want to guard their APIs from prying eyes often implement pinning. (Notably, some high-profile social apps have used it, making their API usage harder to reverse-engineer).
Why do apps use pinning? The primary reason is to protect users and data from fake certificates or malicious Wi-Fi proxies. For example, without pinning, an attacker (or a researcher like us) could install a custom CA certificate on the phone and intercept the app’s HTTPS traffic. Pinning stops this by accepting only the real server’s cert (for example, the app might have the server’s public key embedded, so any other certificate – even if “trusted” by the OS – is rejected). This protects against scenarios where someone could otherwise snoop on or tamper with the data. Essentially, the app says, “I will only talk to exactly the server I expect, no exceptions.” The downside (for us and even for the app developers) is that it makes debugging or scraping the traffic much harder.
How can you tell if an app is using SSL pinning? The easiest way to detect pinning is by trying to intercept the app’s traffic with a proxy tool after installing the proxy’s certificate on the device. If everything is set up correctly, a non-pinned app will have its HTTPS calls intercepted (you’ll see the plaintext requests). If the app is pinned, the interception will fail – the app might not fetch any data, or you may see errors indicating SSL/TLS handshake failures. In other words, if you attempt to intercept and the app refuses to load data (despite trusting the proxy’s CA), it’s likely doing SSL pinning. For example, you might get an error in-app like “Cannot connect” or see in logs that certificate validation failed. At that point, you know the app is actively blocking your man-in-the-middle.
Scenarios for Scraping Mobile App Data
Not every setup is the same when it comes to intercepting app traffic. Here we outline a few scenarios and what’s required for each, especially focusing on Android vs iOS:
Scraping on a Real Android Device (with SSL pinning bypass)
If you want to run a delivery app on an Android phone and scrape its data, you’ll likely need to deal with SSL pinning. Many Android apps (especially modern ones targeting Android 7.0+ and above) won’t allow traffic interception by default, even if you install a custom certificate, due to pinning or network security settings. To get around this, rooting the Android device is often necessary. Rooting your phone gives you deeper access, allowing you to do two important things: install your proxy’s certificate as a system CA (trusted system-wide by all apps) and use tools to disable SSL pinning in the app.
Bypassing pinning on Android: In a previous article, we covered how to bypass SSL pinning using tools like Frida on a rooted Android device. In brief, this involves attaching to the running app and overriding the code that performs the certificate check (effectively tricking the app into accepting all certificates). There are also modules (for Xposed/Magisk) and custom-patched APKs that can disable pinning. Whichever method you use, the goal is to make the app trust your interception setup. Once pinning is bypassed, the app’s HTTPS requests can be intercepted as if it were any regular traffic.
Running the app on a rooted phone: After rooting and handling pinning, you can use a debugging proxy (like HTTP Toolkit, Burp Suite, or mitmproxy) on your computer and point the phone’s traffic to it. For instance, you would install the proxy’s CA cert to the system store (so the app trusts it) and configure the phone’s Wi-Fi to use the proxy. Then, as you use the delivery app (browse restaurants, etc.), you’ll see its API calls in the proxy’s log. These calls often return JSON data that you can examine and later use in your scraping scripts. If the app still doesn’t show any data via the proxy, that’s a sign the pinning bypass didn’t work. You may need to retry the bypass or use a different technique. (For some apps, you could alternatively run them in an Android emulator, which can be rooted or have its trust settings tweaked – more on emulators in the next section.)
Using a Physical iOS Device (without Jailbreak)
On iOS, things work a bit differently. iPhones do not allow user-level apps to install root certificates automatically into the trusted store – however, iOS lets users manually install and trust certificates in the system settings. The good news is that, unlike Android, iOS apps, by default, will trust user-installed CA certificates (there’s no separate network security config that ignores user certs). If the app isn’t employing SSL pinning, you can intercept its traffic on a stock iPhone or iPad without jailbreaking.
Here’s the scenario: you have a delivery app on your iPhone and want to capture its API calls. You can use a tool like HTTP Toolkit or any intercepting proxy in manual setup mode:
No jailbreak needed: Simply install the proxy’s certificate on the iPhone. For HTTP Toolkit, you might retrieve the certificate through a setup link or by exporting it from the app. On iOS, you typically import it as a profile and then go to Settings > General > About > Certificate Trust Settings to enable full trust for that cert. Once that’s done, your iPhone will trust the proxy acting as a fake “CA.”
Configure proxy: Connect your iPhone to the same network as your PC (running the proxy). Then, set the iPhone’s Wi-Fi HTTP Proxy setting to point to your computer’s IP and the proxy’s port (e.g., port 8000 for HTTP Toolkit’s interceptor). Now, all traffic from the phone (on that Wi-Fi) will route through your proxy.
Intercept the app: Open the target app on the iPhone and perform some actions (e.g., search for a restaurant). If the app is not pinned, its requests should flow through the proxy and be visible for you to inspect. For example, you might see a GET request to an endpoint and a JSON response containing a list of restaurants.
What if the iOS app is pinned? If nothing shows up, the app could also use SSL pinning. iOS apps can implement pinning like Android apps, which would block this interception. Bypassing pinning on iOS is more complex and usually requires a jailbroken device, so you can use tools like Frida or Objection to neutralize the pinning checks.
If you’re lucky, many less-sensitive iOS apps (such as some delivery or shopping apps) don't require pinning, so you won’t need to jailbreak.
In summary, using a physical iOS device can be the simplest approach if no pinning is present: just install the certificate and set up a proxy, and you’ll capture the traffic.
Intercepting Mobile App Traffic with HTTP Toolkit
Now that we have prepared our devices let’s see how to intercept traffic using HTTP Toolkit, a user-friendly tool for intercepting HTTP(S) traffic. It works on Windows, macOS, or Linux and can intercept traffic from browsers, scripts, and Android/iOS devices. We’ll cover the most straightforward case: scraping from a physical iOS device with no certificate pinning.
I wrote an extended version of this tutorial at this link if you want to get more details on how to bypass SSL pinning. In the article, I used Fiddler everywhere, but once you installed Frida on your mobile or virtual device, you could use HTTP toolkit.
Intercepting iOS App Traffic (Physical Device)
If you’re using an iPhone or iPad for scraping, the setup is a bit manual since HTTP Toolkit (and similar tools) can’t automatically configure iOS devices (iOS doesn’t allow third-party apps to set proxy settings or install certs for security reasons). But it’s still straightforward:
Start the HTTP Toolkit and click on IOS via Manual Setup. A new page with installation instructions will open. Just follow them to install the certificate and set up the proxy on your WIFI configuration. Remember that if you scan the QR code, the destination link should be opened with Safari, not other browsers. Otherwise, you cannot install the certificate.
To properly check if your installation ended correctly, you can visit the HTTP Toolkit test page (like amiusing.httptoolkit.tech), which can confirm interception.
Now, with the proxy set and cert trusted, launch the delivery app on the iPhone and perform some actions in the app. On HTTP Toolkit (PC), you should see the requests appear if all is well. You can then inspect them exactly as you’d normally do on your browser.
Sometimes, it may happen that the app keeps telling you it needs to be updated, probably as a form of protection. If an iOS app has pinning, and you can’t get around it (because you prefer not to jailbreak), you might consider switching to the Android method for that particular app (perhaps the Android version of the app has less strict security or can be run in an emulator with Frida). As an alternative, you can use research tools like objection (which uses Frida under the hood) that can dynamically bypass pinning on jailbroken devices.
Final remarks
Scraping data from mobile apps opens up a world of possibilities, especially for mobile-first services. Delivery apps are a prime example of how rich datasets (menus, prices, ratings, etc.) can be obtained by intercepting and replicating app network calls. This process has its own challenges, like SSL pinning and jailbreaking devices, but once it’s set up, thanks to tools like HTTP Toolkit, everyone can start gathering actionable data from mobile applications.