THE LAB #85: Bypass Akamai Bot Protection by Chaining Proxies

How JA3Proxy and residential proxies can help us bypass Akamai

Akamai Bot Manager is one of the most common anti-bot solutions on the market. It’s used by many high-profile websites, ranging from e-commerce sites to travel sites, and, depending on its configuration, can be challenging to bypass.

Based on my experience, the typical pattern I encounter when a website activates Akamai Bot Manager protection is that the scraper (usually a Scrapy one in my stack) hangs and times out from the first request.

But what is Akamai Bot Manager, and how can I see if a website is using it?

---

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

---

Akamai Bot Manager Overview

Akamai’s bot detection, like every other modern anti-bot protection software, works on multiple layers.

Network fingerprinting is one of the first: Akamai analyzes the TLS handshake and connection details (JA3 TLS fingerprint, cipher suites, TLS version, etc.) to see if they match a real browser or a known automation tool. Each browser (Chrome, Firefox, Safari, etc.) has a characteristic TLS fingerprint, and if your client’s TLS Client Hello doesn’t match any common browser, Akamai knows something is fishy.

It also inspects HTTP/2 usage and header order – real browsers almost always use HTTP/2+ these days, and they send HTTP headers in a particular order and format. If a client is still using HTTP/1.1 or has headers in a non-browser order, it’s a red flag. Additionally, Akamai looks for browser-specific headers or values that scrapers might omit; ensuring your headers (User-Agent, Accept-Language, etc.) and their order mirror a real browser’s is crucial.

Another layer is IP reputation and analysis. Akamai checks if the client IP is from a residential network, mobile network, or a datacenter. Residential and mobile IPs (the kind real users have) score high on trust, whereas known datacenter IP ranges are automatically suspect. High-volume requests from a single IP address or IP subnet will also lower the trust score. This is why successful scraping often requires rotating residential proxies – always to appear to come from different real-user locations, not a cloud server farm.

Finally, Akamai employs behavioral analysis using client-side scripts and AI models. A JavaScript sensor on the webpage collects a multitude of data points about the client’s environment and interactions (such as timing, mouse movements, or their absence, unusual properties in the browser object, etc.). Akamai’s AI models crunch this data to assign a bot likelihood score to each session. This aspect is the most challenging to bypass and often requires running a headless browser or replicating the sensor logic. (It’s beyond our scope here – our focus will be passing the network-level checks, which is the most common case for e-commerce websites, in my experience.)

But how can we detect that a website is using Akamai Bot Manager?

Apart from the usual Wappalyzer browser extension, if you notice the _abck and ak_bmsc cookies used on a website, this is the clearest sign that it’s using Akamai to protect itself.

Given these defenses, many Scrapy users have turned to the scrapy-impersonate download handler to bypass Akamai. This plugin integrates the curl_cffi library to impersonate real browsers’ network signatures.

In practice, Scrapy Impersonate makes your Scrapy spider’s requests “look” like a Chrome or Firefox: it offers TLS fingerprints (JA3) that match those browsers, uses HTTP/2, and even adjusts low-level HTTP/2 frame headers to mimic the browser’s patterns. By doing so, it addresses the TLS and HTTP/2 fingerprinting issue – a Scrapy spider with this handler can handshake with an Akamai-protected server in a way that’s almost indistinguishable from a normal Chrome browser.

---

Thanks to the gold partners of the month: Smartproxy, IPRoyal, Oxylabs, Massive, Scrapeless, Rayobyte, SOAX, ScraperAPI and Syphoon. They prepared great offers for you, have a look at the Club Deals page.

---

Limitations of Scrapy Impersonate

While Scrapy Impersonate is a powerful tool, it comes with certain limitations:

• Locked into Scrapy: Scrapy Impersonate is designed as a Scrapy download handler, which means it only works within Scrapy’s asynchronous framework. If your project doesn’t use Scrapy or you want to switch to a different framework (like a simple script with requests/httpx or an asyncio pipeline), you can’t directly carry over its capabilities. Migrating away from Scrapy often means a complete rewrite of your HTTP logic, and you’d lose the built-in TLS spoofing unless you implement a new solution from scratch.

• Proxy Rotation Challenges: Using Scrapy Impersonate alongside proxy rotation can be tricky. Under the hood, it replaces Scrapy’s default downloader with one based on curl_cffi, which doesn’t seamlessly integrate with Scrapy’s proxy middleware. Early adopters discovered that HTTPS proxy support was broken because the proxy handling code was bypassed. Although fixes and workarounds (like disabling Scrapy’s built-in proxy handling and configuring curl_cffi directly) exist, it’s harder to rotate proxies or handle proxy authentication with this setup. Robust error handling for proxy failures (e.g., detecting a dead proxy and retrying) is not as straightforward as with Scrapy’s standard downloader, because errors bubble up from the curl layer and may not trigger Scrapy’s usual retry logic.

• Maintenance and Flexibility: Scrapy Impersonate currently supports a finite list of browser fingerprints (Chrome, Edge, Safari, etc., up to certain versions). This list can lag behind the latest browser releases. You might be stuck impersonating an older browser version, which could be a problem if a target site specifically requires the nuances of a newer TLS handshake (some advanced WAFs actually check minor details that change between Chrome versions).

• Not a Silver Bullet: Perhaps most importantly, even with proper TLS and HTTP/2 impersonation, Akamai Bot Manager can still detect and block you. For websites that have implemented a higher level of protection, checking also the browser fingerprint, any browserless configuration, including Scrapy Impersonate, isn’t sufficient for Akamai or similar top-tier bot defenses. You might get past the TLS handshake, but fail on other signals (like the absence of the expected sensor data or subtle discrepancies in headers/cookies). In other words, it’s a piece of the puzzle, not a complete solution.

The solution we’ll see today helps in solving the first two points: we’ll chain together JA3Proxy, for an optimal TLS fingerprint and a rotating residential proxy to rotate our IPs and have higher reputation scores.

---

---

Understanding TLS Fingerprints and JA3

Before diving into the solution, it’s important to understand what exactly we are spoofing. Every HTTPS client presents a unique TLS fingerprint during the handshake. This fingerprint is a combination of the TLS protocol version and a bunch of options the client says it supports – think of it as the client’s “dialect” of speaking TLS. Key components include:

• Supported TLS Version: e.g. TLS 1.2 vs TLS 1.3. Modern browsers will offer 1.3 (while still allowing 1.2 for compatibility). Older clients or some libraries might only do 1.2.

• Cipher Suites: the list of cryptographic algorithms the client can use, in preferred order. Browsers tend to have long lists including ciphers like AES-GCM, ChaCha20, etc., plus some GREASE (randomized) values to prevent fingerprinting.

• Extensions: extra features in TLS, like Server Name Indication (SNI), supported groups (elliptic curves), ALPN (which is used for HTTP/2 negotiation), etc. Both the presence of certain extensions and their order matter.

The concept of JA3 fingerprinting is a standardized way to record these TLS Client Hello details. JA3, named after its creators’ initials, composes a fingerprint string by concatenating the above fields in a specific order:

JA3_string = TLSVersion,CipherSuiteIDs,ExtensionIDs,EllipticCurveIDs,EllipticCurveFormatIDs

Each list (ciphers, extensions, etc.) is joined by - and the sections by ,. For example, a Chrome browser might produce a JA3 string like:

771,4866-4867-4865-....-47-255,0-11-10-16-23-...-21,29-23-30-25-24,0-1-2

This represents TLS 1.2 (771 is 0x0303), a specific set of cipher suites, extensions, supported curves, and curve formats (the numbers are standardized IDs). The JA3 string is then MD5 hashed to create a compact 32-character fingerprint value. Security tools often log or compare the MD5 hash (since it’s easier to handle than a long string of numbers).

Why does this matter for bot detection? Because browser TLS stacks are fairly uniform. Chrome version X on Windows will always present the same JA3 fingerprint down to the list ordering. Firefox will have its own distinct JA3.

Python’s requests library (which uses OpenSSL under the hood) has a JA3 that’s totally different from any mainstream browser, so it’s easily detectable.

Anti-bot services like Akamai maintain databases of JA3 hashes: if your JA3 isn’t on the “known good” list (common browsers) or if it’s on a known automation list, you’ll get flagged. In fact, JA3 was originally created to track malware and bots by their TLS handshake, and it has become a handy tool for anti-scraping as well.

In summary, to pass Akamai’s TLS fingerprinting checks, we need our client’s JA3 to match a popular browser.

This usually means mimicking the latest Chrome or Firefox fingerprint (since those are the most common legit users on the web).

Simply changing the User-Agent string isn’t enough – we must modify the low-level TLS handshake. Scrapy Impersonate does this internally via curl_cffi (which itself leverages a special build of curl and TLS libraries to imitate browsers). But outside Scrapy, we need another way to achieve the same effect.

---

The scripts mentioned in this article are in the GitHub repository's folder 85.AKAMAI-JA3PROXY, available only to paying readers of The Web Scraping Club.

If you’re one of them and cannot access it, please use the following form to request access.

---

TLS Impersonation Proxy + Residential Proxy Chain

Our solution is to chain two proxies to make our scraper virtually indistinguishable from a real browser user:

1. JA3Proxy for TLS Impersonation: JA3Proxy is an open-source tool that acts as an HTTP(S) proxy that replays traffic with a chosen TLS fingerprint. In other words, you run JA3Proxy locally, configure it to imitate a specific browser’s TLS handshake, and then direct your scraper traffic through it. JA3Proxy will terminate your TLS connection and initiate a new TLS handshake to the target site using the impersonated fingerprint. From the target site’s perspective, it looks like, say, a Chrome browser connecting. The beauty of this approach is that it’s client-agnostic – you can use Python requests, httpx, cURL, or anything, by simply pointing it at JA3Proxy. You are no longer locked into Scrapy or any particular library to get browser-like TLS; the proxy takes care of it.

   Under the hood, JA3Proxy uses uTLS (an advanced TLS library in Go) to customize the Client Hello. It supports a variety of client profiles (Chrome, Firefox, Safari, etc., across different versions). You can, for example, configure it to mimic the latest browsers available in the library. For our needs, we’d choose the latest available Chrome fingerprint, Chrome 133. As for Scrapy-Impersonate, the integration of the latest browsers in the library can take some time, but until this gets regularly updated, it’s not an issue.
   

   One thing to note: JA3Proxy focuses on TLS fingerprints (the JA3 part). It doesn’t inherently modify HTTP headers (other than those that relate to TLS, like ALPN for HTTP/2) or handle higher-level browser behaviors.
   It gets us past the network fingerprinting, which is the hardest to change, but we must still ensure our HTTP headers and usage patterns are correct. Luckily, we can manually set headers in our HTTP client to mimic a browser (User-Agent, etc.), and HTTP/2 can be achieved as long as the TLS negotiation allows it (Chrome’s Client Hello will advertise ALPN support for h2, so if the site supports it, JA3Proxy will negotiate HTTP/2).

2. Residential Proxy for IP Rotation: The second part of the chain is an upstream residential proxy. This will take care of the IP reputation and distribution.

   The combined effect is powerful: to Akamai, your scraper now looks like Chrome 133 running on a residential IP. The TLS handshake matches Chrome’s JA3, the HTTP/2 and headers can be adjusted to match Chrome, and the source IP is a regular household. This addresses the major fingerprinting vectors at the network level.

   It doesn’t solve Akamai’s JavaScript-based challenges by itself, but this should be enough to bypass most of the websites you’ll encounter.

---

Before leaving with the code example, I wanted to let you know that I've started my community in Circle. It’s a place where we can share our experiences and knowledge, and it’s included in your subscription. Give it a try at this link.

---

Setup Guide for JA3Proxy