HTTP Toolkit, your best friend for network inspection
Discover the network traffic generated by an app or a website with this tool
During the past months at Re-Analytics we had some consultancy tasks that required app network inspections.
I needed to dive deep into this topic since we don’t perform it daily, and that’s the main reason I wrote articles about Frida and SSL certificate unpinning in past issues of The Lab.
Today, I want to share with you my latest findings on the subject, since I just recently discovered HTTP Toolkit.
Finding the most efficient way to scrape a website is one of the services we offer in our consulting tasks, in addition to projects aimed at boosting the cost efficiency and scalability of your scraping operations. Want to know more? Let’s get in touch.
How does HTTP Toolkit work
HTTP Toolkit is a tool for intercepting and monitoring HTTP and HTTPS traffic. Like Fiddler Everywhere, it acts as a man-in-the-middle between your client and your server, listening to everything passing between them. To decrypt HTTPS traffic will require installing an SSL certificate on the client, but the interesting part of HTTP Toolkit is that it will handle it by itself.
In The Lab 59 issue, we saw the complexity of this operation for an app: We needed to root the virtual device, install Frida on it, modify a script to use the Fiddler certificate, and finally run our tests.
With HTTP Toolkit, rooting the virtual device would be enough (even if we still could use Frida, as we’ll see later) to intercept the network traffic.
Even inspecting what happens on a browser is a piece of cake that requires only a few clicks.
Let’s see some examples together.
Network Inspection on a Browser Window
Inspecting network traffic with HTTP Toolkit is similar to using your browser’s developer tools tab but with steroids added.
First, some websites like bet365.it don’t allow you to browse them freely with the developer tools open, at least with Chrome. This is one measure to protect themselves from bots, but you can inspect the traffic using HTTP Toolkit without opening the dev tools.
One of the coolest features is the chance to export your browsers' requests in many different languages, in addition to the classic JS, Python, and cURL.
You can even mock and modify requests with ease by adding custom rules.
Intercepting traffic on a virtual Android device
This option greatly simplifies the network traffic interception from a mobile app.
With Frida, I needed to test several scripts from their codeshare platform, and in some cases, I could not make any of them work fully.
I encountered the main issues when the app I was targeting used external services for authentication or other stuff, and they were protected by certificate pinning. In these cases, the app could not execute these requests, making it unusable for me.
In my short experience with HTTP Toolkit, I haven’t had this problem yet, despite not having solved 100% of the cases. If you encounter an API endpoint protected by an anti-bot, you will probably be detected, and the requests will be blocked.
Once we launch our virtual Android device and then open HTTP Toolkit, we have the following options in the image to proceed.
We can use a non-rooted device connected to ADB, set up the device via a QR code, or use a rooted device with Frida installed.
In my short experience, I didn’t notice any difference in success rate between the first and third options, but my usage has been limited to a few apps, except for Google Services. With Frida, I could not intercept Google Maps calls, while with the first option, the app works a little better, but I still cannot see what happens under the hood. If you have different experiences, please share them in the comment section.
Once you click on “Android Device via ADB,” you’ll see this screen on the device informing you that HTTP Toolkit is running and installed trusted SSL certificates.
If you minimize this app and start playing with your target one, you’ll see the traffic flowing.
Isn’t that great?
The third option, the one that involves installing Frida, is slightly different at the beginning since you need to select a target app, which will open after you do so.
After that, the experience is identical, so it depends from case to case which way is the best for your needs.
Final remarks
HTTP Toolkit is a great user-friendly tool for inspecting network traffic in different environments: browsers, apps, containers, and more! These tools can usually be challenging to use and set up, as we’ve seen with Frida on mobile phones, so I was shocked by its simplicity.
Best of all, it’s free to use, but I choose to pay the basic plan to have all the export options for the requests, which simplifies my life when creating scripts.
If you have other tools to suggest, please leave them in the comment or in the new feedback form at the end of the article.
Like this article? Share it with your friends who might have missed it or leave feedback for me about it. It’s important to understand how to improve this newsletter.
You can also invite your friends to subscribe to the newsletter. The more you bring, the bigger prize you get.
Thanks a lot for the Great insights you share!