Bypassing Akamai Bot Manager for free
How a simple tool like Scrapy-Impersonate can help your scraping pipeline
If you work in the web ecosystem, not just in the web scraping niche, you should have heard the name Akamai several times.
Founded in 1998, Akamai Technologies is one of the oldest and largest players in the Content Delivery Network (CDN) and cloud services industry. Headquartered in Cambridge, Massachusetts, Akamai’s original mission was to speed up content delivery across the Internet, which is still core to its business today.
Over the years, Akamai has expanded far beyond CDNs. It now offers a full suite of web performance, cloud computing, and cybersecurity services to some of the biggest enterprises in the world—including media companies, e-commerce platforms, financial institutions, and governments.
Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.
Akamai Bot Manager: Their Anti-Automation Solution
One of Akamai’s most widely adopted products is its Bot Manager. This solution is designed to detect and mitigate automated traffic (bots) like our scrapers, that are trying to get some pricing data from their websites.
According to their official website, Akamai’s bot protection uses the usual techniques found in modern anti-bot solutions, such as behavioral pattern identification, fingerprinting,, and IP reputation.
In this article, we’ll focus on understanding how to detect Akamai Bot Manager and how to bypass its protections (always fairly and respectfully of the target website) using open-source tools.
Detecting Akamai Bot Protection on a Website
Before attempting to scrape a site, it’s important to detect if Akamai’s bot protection is in use. This knowledge lets you prepare the correct countermeasures. Here are two ways to identify Akamai on a target site:
Using Wappalyzer or Similar Extensions
One of the easiest methods is to use a technology detector browser extension like Wappalyzer.
Wappalyzer reveals the tech stack of a website, including security and anti-bot services in use. If the target site uses Akamai’s bot manager, Wappalyzer will list “Akamai Bot Manager” (often under a “Security” category) in its findings
In practice, you would install the Wappalyzer extension in your browser, visit the website (e.g. gucci.com), and click the extension. If you see Akamai Bot Manager in the report (as in the screenshot above), the site is using Akamai’s anti-bot service.
Thanks to the gold partners of the month: Smartproxy, Oxylabs, Massive and Scrapeless. They’re offering great deals to the community. Have a look yourself.
Inspecting Network Traffic and Cookies
Even without an extension, you can spot Akamai by examining the network requests and cookies in your browser’s developer tools. After loading the page, open the Storage or Cookies tab in dev tools and look at the cookies set by the domain. Akamai bot protection is known to set specific cookies to track your session and challenge status. Two common cookies are:
_abck – A cookie name used by Akamai Bot Manager (sometimes called “Akamai Abck cookie”).
bm_sz – Another cookie used by Akamai to distinguish between bots and humans.
You can find them on page 11 of this Cookie Policy of Net A Porter, a website protected by Akamai Bot Manager.
The presence of these cookies means the page likely ran an Akamai challenge script in the background.
Additionally, you might notice other signs in the network requests: for example, calls to URLs or endpoints containing strings like “akamai” or long alphanumeric payloads being sent back to the server (which can be the collected sensor data). While not always obvious, such patterns often accompany Akamai’s protection.
In summary, if you find any of these cookies or see Akamai mentioned in the site’s network activity, you should assume the site has Akamai bot protection enabled and adjust your scraping approach accordingly.
Example: Bypassing Akamai on Gucci.com with Scrapy-Impersonate
What happens when we try to scrape a website protected by Akamai with a browserless framework like Scrapy?
In the most common cases, we see our requests hanging and then being closed because of a timeout.
DEBUG: Retrying <GET https://www.gucci.com/it/it/> (failed 1 times): [<twisted.python.failure.Failure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion: Connection lost.>]Before choosing a browser automation tool for bypassing the bot protection, when I encounter Akamai, I usually give scrapy-impersonate a try.
In fact, in this case (and in 90% of the cases I encounter), adding it to the Scrapy Spider allows me to bypass Akamai Bot Protection.
2025-03-20 20:13:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.gucci.com/it/it/> (referer: None) ['impersonate']
https://www.gucci.com/it/it/c/productgrid?categoryCode=women&page=0
https://www.gucci.com/it/it/c/productgrid?categoryCode=men&page=0
https://www.gucci.com/it/it/c/productgrid?categoryCode=children&page=0
https://www.gucci.com/it/it/c/productgrid?categoryCode=jewelry-watches&page=0
https://www.gucci.com/it/it/c/productgrid?categoryCode=beauty&page=0
https://www.gucci.com/it/it/c/productgrid?categoryCode=decor&page=0How is this possible?
scrapy-impersonate is a plugin for Scrapy that replaces Scrapy's default HTTP request handler with a lower-level, browser-mimicking engine built on curl_cffi. This allows your scraper to send network requests that look and behave like those sent by Chrome, Firefox, Safari, etc. — not like those sent by a Python script.
Here’s what scrapy-impersonate improves over standard Scrapy:
It uses real browser TLS fingerprints (Chrome, Firefox, Edge, etc.)
It speaks HTTP/2, which modern browsers use (but Scrapy doesn't by default)
It sets headers, ALPN, JA3 hashes, and other TLS handshake details just like a real browser
That’s incredibly important because advanced bot protection tools like Akamai don’t just inspect the HTTP request headers — they analyze the way your client connects at the TLS level to see if you're a browser or a bot.
What is a TLS Fingerprint?
When a browser connects to a website using HTTPS, it goes through a process called the TLS handshake. During this handshake, the client says:
"Here are the encryption algorithms I support, my supported TLS versions, some extensions, and ALPN protocols."
These details include things like:
Supported cipher suites (e.g. TLS_AES_128_GCM_SHA256)
TLS version (e.g. TLS 1.3, 1.2)
Extensions (like SNI, supported groups, key shares)
ALPN protocols (used to negotiate HTTP/2 or HTTP/1.1)
JA3 hash — a popular way to fingerprint a client’s TLS configuration
All of these settings form what’s known as the TLS fingerprint — a unique signature of how a client connects.
Think of it like a browser’s accent: Chrome speaks TLS in a slightly different "tone" than Firefox or Safari, and Akamai is listening closely.
Of course, Python and Scrapy also have TLS fingerprints that have been classified as “bots,” which is why our Scrapy spiders are blocked without a Scrapy-Impersonate, who fakes the TLS fingerprint of a real browser.
How to use Scrapy-Impersonate in your Scrapy spider
The implementation of the Scrapy-Impersonate library in your scraper is quite immediate.
Once you have installed the package, it’s enough to add the following settings in your scraper.
custom_settings = {
"DOWNLOAD_HANDLERS": {
"http": "scrapy_impersonate.ImpersonateDownloadHandler",
"https": "scrapy_impersonate.ImpersonateDownloadHandler",
},
"TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
}Then, when making a request, you can choose the browser you want to imitate by passing the following variables in the meta fields of the request.
meta={'impersonate': 'chrome110'}One issue with this library is that when used in combination with proxies (which you must do if your scraper is running in a data center), sometimes you get network errors that are not handled correctly, so instead of retrying the request, the scraper stops.
On the other side, in some cases, like for Gucci.com, there’s no need to use Scrapy-Impersonate for every request. Once we get the “clearance cookie” from Akamai with the first request, we are allowed to scrape the website for the rest of the execution since the cookie expires after several days.
I hope this solution is helping you in your scraping activities. Do you have different experiences with Akamai? Please let me know in the comments below.
Can Akamai detect Playwright?