Give your AI a web data layer – Decodo’s Web Scraping API turns any site into clean, structured data your models can actually use.

Activate starter plan

I wanted a number I could actually trust, not just another marketing claim, and started to write this piece. Then I saw the great work on the Tyler Richards website (richards.foo/tools/stealth-benchmark) and took some inspiration from it. My idea was to take the open source Python automation stack, point each tool at the same signals a bot detector checks in JavaScript, and score them all the same way. In the past, I have written about anti-detect browsers and fingerprint test pages, but those were about commercial GUI browsers. This time, I focused on the libraries you actually pip install and use in code.

The whole test harness is in our GitHub repository reserved for paying users, inside the folder 111.STEALTH_BENCHMARK. You can run it yourself and get your own results.

The tools I put on the bench

I grouped the fifteen tools into families, so the scores are easier to compare.

My two baselines are plain Playwright and plain Selenium, with no stealth and no patches. These are the control group, and they are supposed to get caught. If a “stealth” tool scores the same as vanilla Playwright, that tells you a lot.

The Playwright stealth patches are patchright, rebrowser-playwright, and undetected-playwright. These keep the Playwright API and try to remove specific automation leaks underneath it.

The CDP-native group is pydoll, nodriver, and zendriver. They drop WebDriver entirely and talk to Chrome over the DevTools Protocol. nodriver is the official successor to undetected-chromedriver, written by the same author, and zendriver is a maintained fork of nodriver.

Undetectable browsers are getting impressively capable, but even the stealthiest one stalls when its connection lets it down. anyIP’s residential and mobile proxies pair real-user IPs with 99.9% uptime - so the network is never the weak link

Build with anyIP →

Then the browsers that rebuild the fingerprint instead of patching flags on top of a normal Chrome. camoufox and invisible_playwright are patched Firefox builds that rewrite the fingerprint at the C++ level. CloakBrowser does the same to Chromium, shipping its own patched binary with a drop-in Playwright API. RayoBrowse is a self-hosted stealth Chromium from Rayobyte that runs in a Docker container and serves a fingerprint picked from a database of real devices. Obscura is the odd one out, a headless browser engine written in Rust that embeds V8 and ships its own browser rather than driving Chrome at all. And scrapling through its StealthyFetcher wraps a stealth Chromium, while undetected-chromedriver is patched Selenium.

What the detector actually sees

A bot detector is not just one thing, so before testing, you have to decide which layer you are actually targeting. In practice, a real detector looks at four main areas: the JavaScript fingerprint surface, the transport (TLS and HTTP/2), request timing, and the proxy exit. For this benchmark, I focused only on the JavaScript fingerprint surface and ignored the rest on purpose.

This is not about being lazy. Scoping it this way is what keeps the results honest. Transport fingerprinting mostly depends on the browser’s own network stack, which is the same whether you use Chromium or Firefox, no matter which Python library you use. Timing and behavior are a different topic, more about mouse movement and dwell time. Proxy reputation is about your IP, not your tool. So the real question I can answer by swapping libraries is simple: when a detector’s JavaScript runs inside the page, what does it actually find?

The detector looks for a fixed list of tells. Is navigator.webdriver true. Does the user agent say HeadlessChrome. Is the WebGL renderer a software rasterizer like SwiftShader instead of a real GPU. Are there leftover automation globals on window, like the old chromedriver cdc_ variables or a framework’s init scripts. Does the chrome object have the right properties. Do the user agent, navigator.platform, and the Client Hints all agree on the OS. The knowledge base on richards.foo lists hundreds of these, but I pulled the highest severity ones into my own probe.

How I set up the test

At the core, I use a single HTML page called probe.html, served from a small local server. Each tool loads this page, the page reads its own fingerprint in JavaScript, and writes the result back into the DOM as base64 JSON. I use base64 so I do not run into any HTML-escaping issues when I pull the rendered page from each driver.

To keep the harness manageable, I wrap every tool, no matter how different its API, in a single method:

def render(url, wait_selector=None, timeout=30000, settle_ms=6000, screenshot_path=None) -> str:
    # launch in the tool's stealth config, goto(url), wait, return final HTML, close

The new anti-bot solution - Toughest walls, lowest price. Claim your free 10 000 requests with coupon code WSCLUB

Claim your credits

Whether it is Playwright sync, Selenium’s execute_script, pydoll and nodriver’s async CDP calls, scrapling’s StealthyFetcher.fetch, CloakBrowser’s launch(), RayoBrowse over a Docker CDP socket, or Obscura over its Rust CDP server, they all get wrapped by render(url) -> html. I use the same method for the probe and the two oracle pages, so I never have to special-case a driver. Here is the full vanilla Playwright adapter, and it does not get simpler than this:

 from playwright.sync_api import sync_playwright

def render(url, wait_selector=None, timeout=30000, settle_ms=6000, screenshot_path=None):
    with sync_playwright() as p:
        browser = p.chromium.launch(channel="chrome", headless=True)
        page = browser.new_page()
        try:
            page.goto(url, wait_until="load", timeout=timeout)
            if wait_selector:
                page.wait_for_selector(wait_selector, timeout=timeout)
            else:
                page.wait_for_timeout(settle_ms)
            if screenshot_path:
                page.screenshot(path=screenshot_path, full_page=True)
            return page.content()
        finally:
            browser.close()

Check the TWSC YouTube Channel

For every Chrome-based tool, I used channel="chrome" to drive the real installed Chrome, not Playwright’s bundled Chromium. This choice is important. Playwright’s bundled Chromium in headless mode reports a software WebGL renderer (SwiftShader), which is just a headless artifact, not something the tool did. When I point the same tools at real Chrome, the WebGL renderer shows the actual Apple M2 GPU. By using real Chrome everywhere, the tests I measure reflect the tool’s stealth, not which Chromium binary happened to ship. The engine-level browsers like Camoufox, CloakBrowser, and Obscura ship their own binary, so they use that instead.

I ran everything in headless mode on purpose, since that is both harder and more common for scraping. Headed mode hides some of these tells automatically, and I mention that where it is relevant.

I do not just trust my own score, so I use two outside referees. My probe gives a 0 to 100 score by subtracting a weight for each tell the tool leaks. Then I check two open bot-detection pages for an independent verdict: BrowserScan and deviceandbrowserinfo. deviceandbrowserinfo is more useful because it returns a JSON breakdown, an isBot flag, and about twenty individual detections like hasWebdriverTrue, isHeadlessChrome, and isAutomatedWithCDP. I tuned the weights in my probe so the ordering matches those verdicts, and that is the calibration step. The runner also saves a screenshot of the deviceandbrowserinfo verdict for every tool, so the human or bot call is on record, not just my parser’s output.

Running it, and what came back

Fifteen tools, three targets each, and here is the table the reporter produced.

My probe score and the deviceandbrowserinfo verdict matched on 12 out of 13 tools I could get a verdict for, so that is a 92 percent match. I saved the heatmap of every tell by every tool at 111.STEALTH_BENCHMARK/results/heatmap.png, and you can see the tiers at a glance.

Four tools passed the strict referee: camoufox, CloakBrowser, RayoBrowse, and scrapling. Three of them get there the same way, by rebuilding the browser itself instead of patching automation flags onto a normal Chrome, so there is no HeadlessChrome in the user agent and nothing that reads as CDP automation. scrapling is the fourth, and it passes for a reason worth its own section below, because it took a version bump to get there. Everything that still leaves navigator.webdriver true sits at the bottom of the table, and everything in the middle is one user agent string away from a clean pass.

Read more

Then you get access to a residential proxy pool, and the scraper works.

If you’ve been there, you probably moved on without asking why. The adage is simple: if it works, don’t touch it!

But that “why” matters more than you think because residential proxies aren’t a kind of black magic. They’re a specific technical architecture on top of which your scrapers work. And as with any architectural choice, you have to make decisions on purpose and based on specific parameters.

In this article, you’ll get a complete technical picture of residential proxies, starting from how IP addresses are classified, how the proxy infrastructure routes your traffic, what residential proxies are, and their common types. In the end, you’ll also discover the ethics behind residential IP sourcing and how to choose ethical providers with a practical decision framework.

Let’s get into it!

Give your AI a web data layer – Decodo’s Web Scraping API turns any site into clean, structured data your models can actually use.

Activate starter plan

---

What Is an IP Address?

An IP address (Internet Protocol address) is a numerical label assigned to every device connected to a network that uses the Internet Protocol for communication. On the practical side, the IP protocol serves two core functions:

• Host or network interface identification.

• Location addressing.

The current dominant versions are IPv4 and IPv6:

• IPv4 addresses are 32-bit integers, expressed in dotted-decimal notation as four octets ranging from 0 to 255 (for example, 203.0.113.47), yielding a theoretical address space of ~4.3 billion unique addresses.

• Due to exhaustion of this space, IPv6 was introduced, using 128-bit addresses expressed in colon-separated hexadecimal groups (for instance, 2001:0db8:85a3:0000:0000:8a2e:0370:7334), offering a space of ~3.4 × 10³⁸ addresses.

IP addresses aren’t handed out randomly by providers: there’s a strict hierarchy that they must respect. Specifically, a global authority called IANA divides the total pool of IP addresses into large blocks and distributes them to regional bodies known as RIRs (Regional Internet Registries). These RIRs then assign smaller blocks to ISPs (your home internet provider) and cloud providers (like AWS or Google Cloud), who finally assign individual IPs to end users and servers.

Why does this matter for scraping? Because this entire chain is publicly documented, and anyone can look up who owns a given IP address using tools like WHOIS (a directory of IP ownership records). This means a target website can instantly tell whether your request is coming from a residential IP, a datacenter IP, or a known proxy or VPN provider.

This is precisely why residential proxies are so valuable in web scraping: their IPs are indistinguishable from those of regular home users, making them much harder for anti-bot systems to flag.

Start your scraping journey with Byteful: 10GB New Customer Trial | Use TWSC for 15% OFF | $1.75/GB Residential Data | ISP Proxies in 15+ Countries

Claim your 10GB here

What Is a Proxy Server?

A proxy server is an intermediary host that sits between your client and the target web server, forwarding requests and responses on your behalf:

Without a proxy, the TCP connection to the target server is established directly from your machine. With a proxy, your client opens a connection to the proxy, which opens a separate connection to the target, and the target sees the proxy’s IP address as the request origin. This way, your real IP never appears in the target’s access logs or network layer.

This IP substitution is the fundamental value proposition of a proxy in a scraping context. Every request your scraper sends carries a source IP and, as covered above, target servers actively use that IP to make trust decisions. By routing through a proxy, you externalize your network identity. This way, the anti-bots on target servers evaluate the reputation, geolocation, and classification of the proxy’s IP, not yours.

At the network level, a proxy is simply a process listening on a TCP port, capable of accepting inbound connections and initiating outbound ones. It maintains two separate TCP connections for each proxied request: one with your client and one with the target. These connections are independent: they can use different protocols, different keep-alive settings, and the proxy can pool or reuse the outbound connection across multiple client requests.

The proxy may also perform additional functions depending on its implementation. For example, it can manage authentication, logging, IP rotation (cycling through a pool of IPs across requests), and header manipulation.

Proxy Protocols: HTTP/HTTPS vs. SOCKS5

When routing traffic through a proxy, the protocol determines how the proxy handles your traffic at the application and transport layers. This has real implications for scraping purposes. The main protocols are the following:

• HTTP/HTTPS: Proxies that use these protocols operate at the application layer (OSI Layer 7). When your scraping client sends a request through an HTTP proxy, it constructs a full HTTP request and sends it to the proxy server, which forwards it to the target on your behalf. For plain HTTP traffic, the proxy reads the request in full. For HTTPS traffic over an HTTP proxy, the client issues a CONNECT method request, instructing the proxy to establish a raw TCP tunnel to the destination. Once the tunnel is open, the TLS handshake occurs directly between your client and the target server. This means that the proxy becomes a blind pipe and cannot decrypt the payload. This is critical to understand: an HTTP proxy handling HTTPS via CONNECT does not perform SSL termination (unless it’s explicitly a man-in-the-middle proxy with a custom CA cert injected).

• SOCKS5: Is a lower-level protocol operating at the transport layer (OSI Layer 5). Unlike HTTP proxies, SOCKS5 is protocol-agnostic. This means that it simply relays raw TCP (and optionally UDP) streams without any awareness of the application-layer protocol being tunneled. After a brief handshake, the proxy establishes a TCP connection to the destination and shuttles bytes in both directions without inspection. This makes SOCKS5 suitable for any TCP-based protocol, not just web traffic.

One operationally significant detail to consider is DNS resolution. With SOCKS5, the domain name is forwarded to the proxy for resolution, meaning DNS queries originate from the proxy’s network, not yours. This prevents DNS leaks, which can expose your real network location even when your traffic routes through a proxy. HTTP proxies also resolve DNS on the proxy side for CONNECT-tunneled requests, but behavior can vary by implementation.

Now, here’s a question for you: for scraping purposes, which protocol is the best? Well, for most scraping use cases, either protocol is functionally equivalent from the target server’s perspective, since both can tunnel TLS. SOCKS5 is generally preferred for its flexibility and slightly lower overhead, but HTTP proxies are often easier to drop into existing HTTP client configurations without additional dependencies.

Tired of stitching together proxies, browsers, and parsers just to scrape one page? Gaffa handles the whole stack for you, with the control to fine-tune every step when you need it.

Start With Gaffa

What Are Residential Proxies?

A residential proxy is a proxy server whose outbound IP address is assigned by a consumer Internet Service Provider (ISP) to a real, physical household device like a home router, laptop, smartphone, smart TV, or similar. The word “residential” refers directly to the ISP classification of the IP: these are addresses allocated to residential customers, and they appear as such in public IP intelligence databases.

How Residential Proxies Work

The operational model of a residential proxy network involves three distinct parties: the proxy provider (the company offering proxy servers on the market), the exit node (the residential device), and the scraping client (you).

The exit node is a real consumer device whose owner has installed software that, in exchange for some compensation, allows the provider to route third-party traffic through their internet connection. This software runs as a background process and registers the device with the provider’s orchestration infrastructure. The device’s residential IP is thus made available in the provider’s pool.

The provider’s infrastructure is the backbone that ties everything together. It consists of:

• A gateway layer: A set of proxy endpoints that your scraping client connects to. These endpoints are hosted on standard data center infrastructure and serve as the entry point into the residential network.

• An orchestration/routing layer: The internal system responsible for selecting which residential exit node to use for a given outbound request, based on criteria like target geolocation, session persistence requirements, and node availability.

• The exit node pool: The fleet of residential devices registered with the provider, each maintaining a persistent connection back to the orchestration layer.

Thanks to this infrastructure, the scraping requests flow like that:

1. Your scraping client connects to the provider’s gateway endpoint and authenticates.

2. The gateway receives your request and passes it to the orchestration layer, which selects an available residential exit node from the pool based on your session’s parameters.

3. The orchestration layer forwards the request to the selected exit node over the persistent internal tunnel.

4. The exit node’s software initiates the actual outbound TCP connection to the target server from the residential device’s IP.

5. The target server receives a request from what appears to be a residential IP, processes it, and returns the response.

6. The response travels back through the same path in reverse: exit node → orchestration layer → gateway → your client.

This is why, from the target server’s perspective, the only visible connection is the one from the residential exit node. Your client’s IP, the provider’s gateway IP, and the internal routing infrastructure are entirely invisible.

Check the TWSC YouTube Channel

Types of Residential Proxies

Residential proxies are segmented into distinct types based on two primary axes: IP persistence (how long you hold the same IP across requests) and targeting granularity (how precisely you can select the exit node’s location).

This section reports and describes the common residential proxy categories that are based on those two axes.

Rotating Residential Proxies

This is the most common type. With rotating proxies, the provider’s orchestration layer assigns a new exit node—and, therefore, a new IP—for each outbound request, or at a configurable interval. From your client’s perspective, you connect to the same gateway endpoint every time, but each request exits from a different residential IP. This is the default behavior on most residential proxy platforms and is the correct choice for large-scale scraping where you need to distribute requests across a wide IP footprint to avoid per-IP rate limits. Belos is a schema that shows, at a high level, how this process works:

The rotation can be triggered in two ways, depending on the provider:

• Per-request rotation: A new IP is selected for every single HTTP request.

• Timed rotation: The same IP is held for a configurable duration before rotating.

In the context of web scraping, the primary tradeoff is that rotating proxies make stateful interactions difficult. If a target website tracks session state via server-side session IDs or cookies tied to an IP, switching IPs mid-session will break the session. So, for purely stateless scraping cases like fetching product pages, search results, and public data, rotation is ideal.

Sticky Residential Proxies

Sticky proxies allow you to hold the same residential IP for an extended, configurable period. This typically ranges from a few minutes up to 30 minutes, depending on the provider, and is bounded by the exit node’s availability online. The session is maintained by passing a session identifier in your proxy credentials, which instructs the orchestration layer to pin your requests to the same exit node for the duration of the session.

This type is essential for scraping workflows that require statefulness, like multi-step form submissions, login-based scraping, checkout flows, or any target that correlates sequential requests by IP and invalidates sessions on IP change. The risk is that if the exit node goes offline mid-session (the residential device disconnects from the network, which happens regularly on consumer connections), the session is broken, and the provider will typically reassign you to a new IP. This requires your scraper to handle session recovery; otherwise, your pipelines will break.

Static Residential Proxies

Static residential proxies (often known as ISP proxies) occupy a middle ground. These IPs are registered to an ISP (giving them a residential-looking ASN and WHOIS classification) but are hosted on stable data center infrastructure, rather than actual consumer devices. The provider procures these IPs directly from ISPs and hosts them on servers they control.

The key advantage over standard rotating residential proxies is stability and speed. Since the IP is hosted on a server rather than a consumer device, it doesn’t go offline unpredictably, latency is lower, and bandwidth is not constrained by a household connection. You get a fixed, persistent residential-classified IP that behaves like a data center proxy in terms of reliability.

The tradeoffs are:

• Cost: These are among the most expensive proxy types.

• Increasing detectability: As anti-bot vendors are building databases of known static residential/ISP proxy IP ranges, this kind of proxy works as a cat-and-mouse race to escape anti-bots.

Mobile Proxies

Mobile proxies are a specialized subtype where the exit node is a mobile device connected via a cellular network, and the IP belongs to a mobile carrier’s ASN. These are distinct from standard residential proxies because mobile IPs carry a specific characteristic: mobile carriers use large-scale carrier-grade NAT (CGNAT), meaning a single public IP is shared among potentially thousands of concurrent mobile users.

This has a profound implication for anti-bot systems: blocking a mobile carrier IP risks blocking thousands of legitimate users simultaneously, making mobile IPs extremely difficult for targets to block without significant collateral damage. As a result, mobile proxies are the hardest proxy type to block and command the highest price per GB in the market. They are most valuable for targets with aggressive anti-bot systems, typically heavily protected e-commerce sites, social platforms, and ad verification use cases.

The tradeoff is latency and bandwidth variability. The exit node is on a cellular connection, which is subject to signal quality, network congestion, and higher round-trip times compared to a wired residential or data center connection.

Residential Proxies Types: A Summary Comparison

The table below summarizes the subtypes of residential proxies, each with its own specific characteristics:

Need help with your scraping project?

How Residential Proxies Are Sourced and Why It Matters

The technical sophistication of residential proxy networks is surely interesting for techies. But what’s also interesting is where the exit nodes actually come from. This is because understanding the sourcing model is not just an ethical exercise. It has direct operational implications for scraping professionals because the legality and reputational standing of the proxies you use can affect your liability exposure and the long-term reliability of your infrastructure. It also has implications for companies working in highly regulated markets, where specific certifications matter more than anything else. We’ve seen recently how bad things can go from one day to another if your proxy provider is not fully compliant on both IP sourcing side and customer onboarding.

As introduced previously, providers build their residential IP pools by recruiting real consumer devices to act as exit nodes. The mechanisms for doing this fall into a spectrum that is described in the subparagraphs below.

SDK Monetization

The most widespread IP sourcing method is SDK embedding. In this model, a residential proxy provider operates a B2B program where mobile app developers and software vendors integrate the provider’s SDK into their applications. In exchange, the developer receives a revenue share or a flat fee per active device. When an end user installs the app, the SDK runs in the background and enrolls the device’s internet connection into the proxy pool.

The provider’s SDK typically does the following:

• Establishes a persistent outbound connection to the provider’s orchestration infrastructure.

• Monitors device state to only proxy traffic when the device is charging and connected to Wi-Fi (to avoid draining battery or mobile data, which would generate user complaints).

• Throttles bandwidth consumption to stay below the threshold of user noticeability.

• Handles NAT traversal to make the device reachable for outbound proxy sessions.

The apps embedding these SDKs span a wide range. Just to name a few:

• Free VPN clients.

• File utility apps.

• Weather apps.

• Games.

So, the SDK is not the app’s core function: it’s the monetization layer that the developer has grafted on.

Opt-In Peer Networks

Some providers operate a peer model, where users knowingly and explicitly enroll their devices in exchange for a direct benefit. This typically is free access to a premium VPN or bandwidth credits. The consent flow clearly explains that the user’s connection will be used to route third-party traffic.

This is the most ethically defensible sourcing model, as the value exchange is explicit and the user retains full awareness of what their device is being used for.

Ethical Considerations

The ethical landscape of residential proxy sourcing is genuinely complex, and it deserves some treatment to help you discern the right proxy provider to choose.

The Consent Problem

The foundational ethical issue is informed consent. When a residential device is used as an exit node, its owner’s internet connection carries traffic for a third party. This means that their IP address becomes associated with requests they did not initiate, to targets they may never have interacted with. If a scraping client uses that exit node to send requests that the target considers abusive, the device owner’s IP is the one that appears in server logs. The potential consequences for an unknowing participant range from having their IP blacklisted by services they use personally to, in extreme cases, being flagged by automated fraud or abuse detection systems.

This is why true informed consent in this context is very important. Also, for deep ethical reasons, informed consent would require a clear, plain-language explanation that the user’s internet connection will relay unknown third-party traffic, that their IP will be the visible origin of that traffic, and that this may affect their IP’s reputation.

Bandwidth and Resource Consumption

Enrolling a device in a proxy pool consumes the device owner’s bandwidth. In many markets, this resource is metered and has a real financial cost. In mobile-heavy markets where Wi-Fi and cellular data are tightly managed household resources, this is a non-trivial extraction of value from the device owner. This is another reason why proxy providers offer rewards to IP owners.

The Downstream Use Problem

Even when sourcing is fully and clearly consensual, the ethical chain does not end at the exit node: it extends to what the proxy traffic is used for. This is because a residential proxy provider cannot fully control what its clients scrape. This means that the same infrastructure used by a legitimate price monitoring service can be used for credential stuffing attacks, ad fraud, scalping bots, or scraping protected personal data.

For this reason, responsible providers implement terms of service enforcement, traffic monitoring, and client vetting. As a scraping professional, you are a link in this chain and carry some responsibility for the downstream ethics of the traffic you generate.

Legal Exposure

The legal status of residential proxy usage varies by jurisdiction and use case. In the United States, the Computer Fraud and Abuse Act (CFAA) has been interpreted broadly to potentially cover unauthorized access to computer systems. And whether bypassing a website’s IP-based blocks via residential proxies constitutes “unauthorized access” has been the subject of significant litigation.

In the European Union, the GDPR introduces additional complexity when scraped data includes personal information. So, note that using residential proxies does not insulate you from these legal frameworks. It only changes the network-layer attribution of the requests.

What to Look For in a Residential Proxy Provider

If you’re new to the scraping industry and need to get residential proxies, you may be overwhelmed by the offers that are on the proxy market. There are a lot of providers out there, and choosing the best one for you may be hard.

If you want to evaluate providers not only based on budget, the suggestion for you is to primarily make ethical considerations. If you don’t know where to start, indicators of a responsible residential IP sourcing model include the following:

• Published transparency reports detailing how exit nodes are recruited.

• Clear opt-in documentation and publicly accessible SDK disclosure pages.

• Active terms of service enforcement against prohibited use cases.

• Direct compensation models for exit node participants.

• Compliance certifications. Some of the most ethical providers have pursued SOC 2 or ISO 27001 certifications, which indicate a structured approach to operational governance.

Conclusion

Residential proxies are easy-to-use tools, but may be hard to understand. The gap between “it works” and “I know why it works” is where most engineers live, and it’s fine until it isn’t.

Understanding the full stack changes how you make decisions for your scraping projects. Why? Because it changes how you pick a provider, how you evaluate proxy types for a given target, and how you think about the ethical and legal weight of the infrastructure you’re routing traffic through.

So, now that you know how this infrastructure actually works, let us know: does it change how you’d pick proxies for your next scraping project?

Did you like this article? Share it with someone who might find it useful and get a discount on paid plans.

Share

Give your AI a web data layer – Decodo’s Web Scraping API turns any site into clean, structured data your models can actually use.

Activate starter plan

I’m picking up where I left off in a previous investigation, so I won’t repeat all the details here. Back then, I pulled libshopeeaegis.so (one of Shopee’s native crypto libraries) into Ghidra and rebuilt it in Python, byte for byte. That was a straightforward win: just an integer switch over standard AES, SHA, and HMAC, once you get past the intentionally confusing symbol names. But as I mentioned at the end of that article, Aegis is just a general-purpose toolbox, not the actual library that signs API requests. The real signer is a different beast, and that’s what I’m looking at now.

What actually gates the API

Capture any Shopee API request, strip the anti-fraud headers, replay it, and the backend answers with an HTTP error and code 90309999. That code is the whole story of this article. It is the anti-fraud rejection, and by the end you will have seen it come back from the app, from a plain HTTP client, and from two different real browsers.

Two headers carry the protection. On the Java side, two interceptors in okhttp, the HTTP client library the Android app is built on, attach them: com.shopee.app.network.antifraud.b and .d, which name themselves SecurityNewSapInterceptor and SecurityNewSapPostInterceptor. The first header is af-ac-enc-sz-token, a constant for the session:

af-ac-enc-sz-token: ghHLT8yNv0Xd8t9C2/t6Sw==|uJEC5q...JfDCw4ImZS5+gWyOO5Dr...8bd|rsGw3AWMR7Ryh7Jo|08|1

The right foundation to keep a smooth scraping workflow. anyIP helps teams collect web data more consistently with high-quality residential and mobile IPs worldwide.

Explore anyIP Proxies →

This token is a pipe-delimited string: an encrypted device fingerprint, a 16-byte wrapped key, an 81-byte ciphertext, a 12-byte nonce, and two version markers. It’s computed once per session and then reused. The second header, x-sap-ri, comes with a set of signatures keyed by short hex strings, and these change every request since they’re computed over the request itself. Both headers are generated by libshpssdk.so, a hefty 4.7 MB native library. The tricky part is the per-request signature, since that’s the bit you can’t just precompute and reuse.

Finding the signer, and deciding not to port it

libshpssdk.so exports exactly one symbol, JNI_OnLoad. Everything else is hidden, which tells you the native methods are registered at runtime with RegisterNatives, the Java Native Interface (JNI) call an app uses to bind native functions to Java methods, rather than exported by name. So we catch the registration as it happens. On arm64, RegisterNatives is a slot in the JNIEnv function table at byte offset 0x6b8. We hook that slot spawn-gated, before the app’s own code runs, and read out every method the library registers. The agent is in /shopee_sap_oracle/agent_oracle.js.

We are after one JNI signature. A request signer takes the request and returns headers, which in JNI terms is two byte arrays in and a string out, ([B[B)Ljava/lang/String;. One method matches. It is vuwuuwvw, registered at libshpssdk+0xc7ebc, on the obfuscated class com.shopee.shpssdk.wvvvuwwu, a name we recovered from the jclass argument of the registration call through java.lang.Class.getName().

When I opened that address in Ghidra, I found the function itself barely does anything. It just marshals its arguments and passes them to another function, along with a 440-byte blob and a table of function pointers. That’s the classic shape of a bytecode interpreter: the blob is the program, the pointer table is the opcode handlers, and the real logic lives in the bytecode, not in any native function I can easily read. This is where my earlier approach hits a wall. Last time, I could just read the primitive bodies and match them to AES and SHA. Here, there are no such primitives. To reimplement the signer, I’d have to reverse and interpret the whole virtual machine (VM), a project that would take weeks and is on the same level as breaking DroidGuard, Google’s own anti-abuse VM. So instead of porting it, I decided to just call it directly.

The one thing that works: the oracle

The signer is a Java method on a class we already resolved, so we drive the app’s own code. The agent exposes one RPC (remote procedure call) export that takes a URL and a body, turns them into the two byte arrays the method expects, and returns whatever the signer returns. The driver (/shopee_sap_oracle/frida_oracle.py) spawns the app, waits for the signer to register, and signs requests we choose. Re-running it against the current app, the signer registers in three seconds and answers:

[SIGN] url=https://mall.shopee.ph/api/v4/native/homepage body={"test":1}
   3c7b6059 = LR4te3nzTKb0RcbDqGs1bsXb0pB=
   71356940 = uMaRB6hpxBCq1BwZ2bFqGXjGFow=
   a0e0dcb3 = WFiutLJdM2C0t5Lr4yaoPr9CLn4=
   eaab58ec = 3BJOkkGhS53kj19RA2QqVLfC1m8sZFLSYm8W8S+Kl3yHU+vuzvb+...   (~2 KB)
   x-sap-ri = d9e7576a26efc4c9896e6716017c0e63cf0cae2c4c560ed7dd03

Start your scraping journey with Byteful: 10GB New Customer Trial | Use TWSC for 15% OFF | $1.75/GB Residential Data | ISP Proxies in 15+ Countries

Claim your 10GB here

That’s the win, and I want to be clear about what it actually means. I can now produce the exact headers the app would attach for any request I choose, without needing to understand how the signer works under the hood. But before I trust these headers, it’s worth looking closely at the output, because it gives you a sense of how tough it’ll be to reuse them. The structural checks I’m about to show run on the captured values using /shopee_sap_oracle/analyze_headers.py. No device needed.

The short values look like base64 of a 20-byte digest, which is the length of HMAC-SHA1, so the easy reading is that they are keyed hashes and that is where a first pass leaves them. The easy reading is wrong. Decode seven of them as standard base64, and five come back with non-zero bits in the final padded position, which a conforming base64 encoder never produces from raw bytes:

  6m6r94xGLKAjTMYNMepdMiLQpSB=  len=20  NON-canonical (low2=1)
  wLJO02yz2I+FAZETVwC0aTCflyB=  len=20  NON-canonical (low2=1)
  1iwMbNiaPma2XLT7ToRs4jgiasE=  len=20  canonical
  47WhCSbrTx1r5FdFFlgtpHPN0OO=  len=20  NON-canonical (low2=2)
  LzCPR2/LYpvqP/z1HBXtuzj+8dQ=  len=20  canonical
  vmPAmagMmOlffOGIWF8MwJEBuNT=  len=20  NON-canonical (low2=3)

Check the TWSC YouTube Channel

So these aren’t just plain base64-encoded digests. Either the alphabet is shuffled or there’s some transformation after encoding, and that’s exactly the kind of detail that trips up anyone trying to reproduce the signature with HMAC-SHA1 and a standard encoder. The keys themselves are interesting too: within a session, no two requests share a key, and those eight-hex identifiers aren’t the actual header names you see on the wire. Plus, endpoints with a body return four entries, while GETs without a body return three. There’s a whole mapping layer from the signer’s per-request keys to the real header names, and I never managed to reverse that part.

Then there is x-sap-ri itself. It is 26 bytes: a leading counter, a three-byte session ID, an eight-byte middle field, a 01 marker at byte twelve, and a thirteen-byte tail. The counter is the interesting part, because it is coarser than it looks. In the fresh capture, both requests came back with counter d9 and session e7576a in front, and only the middle and tail differed:

d9 e7576a 26efc4c9896e6716 01 7c0e63cf0cae2c4c560ed7dd03
d9 e7576a b4c3b40323380413 01 d56d3f5c7eaef45395425b9720

So the leading byte isn’t just a simple counter that increments with each request. What actually separates one request from another is hidden in the middle and tail fields, and the token is tightly bound to the specific request through those parts. That binding is the first reason why a replay fails, and it’s where the real obstacles begin.

App wall one: the replay is out of sequence

The obvious next step was to sign a real product request using the oracle and try replaying it from outside the app. I picked a phone case from Shopee’s public site (shop_id=20396998, item_id=5420033304), had the oracle sign its product detail URL, attached the session’s af-ac-enc-sz-token and a cookie I grabbed with a one-shot header hook, and replayed the request with curl-cffi. The driver for this is /shopee_sap_oracle/replay.py. Here’s what Shopee sent back:

[*] oracle signed 4 headers: ['5cfcca02', 'c382fc01', 'cf1e6149', 'x-sap-ri']
==== REPLAY RESULT ====
HTTP 403
{"5":false,"2":false,"0":2,"3":90309999,"error":90309999, ...}

The status code changes depending on the endpoint, but the error code is always the same. Having a valid signature is necessary, but it’s not enough. This replay was always going to fail for two reasons: first, the x-sap-ri binding ties the token to the device’s own request sequence, so any injected, out-of-sequence request gets rejected, even with a fresh signature. Second, I put the signer’s per-request-keyed values under those same eight-hex keys, but I never actually confirmed that’s how the app names them on the wire. For a clean external replay, you’d need both of those right, and I could only be sure about one.

App wall two: the RASP cancels everything

The way to remove both doubts is to stop replaying and make the request from inside the app, where the sequence, the device state, the TLS, and the real header names are already coherent. The app builds its API calls on okhttp, so we went looking for its OkHttpClient at runtime with Java.choose. The probe is in /shopee_sap_oracle/crash_trace.js. There are more than a dozen live instances, and you spot the API client by its interceptors. The antifraud layer is not among the application interceptors where we first looked. It sits lower, among the network interceptors, which run after okhttp has set up the real connection, so it signs the request that is actually about to go on the wire.

As always, the code that will be used for the python reimplementation we’re showing now can be found in our GitHub repository reserved for paying users, inside the folder 110.SHOPEE-PART2.

Read more

If you’ve run a scraper at any real scale recently, you’ve felt the shift. The block didn’t come from too many requests per minute, and rotating to a fresh IP didn’t fix it. The page just quietly refused to load the data you came for. That’s fingerprinting doing its job, usually before your first byte of useful content arrives.

We keep hitting the same pattern. A pipeline that ran fine for months starts returning empty pages or CAPTCHAs, the proxy pool looks healthy, the rate limits are nowhere near tripped, and nothing in the logs explains it. What changed is not how many requests you send. It is how much the target now knows about the client sending them.

What browser fingerprinting is, and why it matters for scraping

Fingerprinting identifies and tracks a client from the unique combination of characteristics its browser and device expose, with no cookies, no login, and no explicit identifier required. It doesn’t ask “who is this user?” It asks “does this look like the same client as before, and does this client look real?”

The signals are individually boring. Browser version, OS, screen resolution, installed fonts, timezone, locale, how the device renders graphics. None of them identify you alone. Put them together and you get a profile distinctive enough to survive across sessions, IP changes, and incognito mode. For the defense side that’s a bargain, because one collection step feeds fraud detection, bot detection, rate limiting, and access control at once.

From IP-based blocking to identity-based detection

The traditional model was simple. Send requests, rotate IPs, stay under the rate limit. As long as each request looked like it came from a different address, you stayed under the radar.

Fingerprinting breaks that model cleanly. Rotate all the IPs you want, and if your browser fingerprint stays identical across them, you’ve just told the anti-bot that one client is hiding behind a dozen addresses. No real user does that. It is more suspicious than a single sticky IP would have been. The rotation you relied on becomes the tell.

How fingerprinting works under the hood

Modern systems don’t lean on one identifier. They build a layered profile across browser properties, device capabilities, rendering behavior, and network characteristics, then score how coherent the whole thing looks. The more consistent and ordinary the profile, the more trust it earns. That last part is the one most scrapers get backwards, and we’ll come back to it.

Passive signals

These come for free, straight from the HTTP request or a basic JavaScript query, with no heavy lifting required:

• User-Agent string (browser, version, OS)

• Accept headers (language, encoding support)

• Screen resolution and color depth

• Timezone and locale

• Installed fonts and plugins

Active signals

Active fingerprinting runs JavaScript to pull things that are much harder to fake, because they depend on how the hardware actually behaves:

• Canvas fingerprinting renders hidden graphics. Tiny differences in font rasterization, anti-aliasing, and GPU instructions produce a distinctive output.

• WebGL fingerprinting probes the GPU and surfaces details about the graphics hardware and driver stack.

• AudioContext fingerprinting reads the audio processing pipeline, which varies slightly across systems and adds more entropy.

These matter because they measure behavior, not claims. A User-Agent is just a string you can set to anything. A Canvas hash is the device answering for itself. Spoof the User-Agent but leave the Canvas output screaming “headless Chromium on a Linux server,” and the two halves of your story don’t match.

Network-level signals

Detection doesn’t stop at the browser. The network layer sanity-checks the rest of the profile:

• IP address and ASN (who owns the network)

• Geolocation consistency, checking IP against timezone against language

• TLS fingerprinting (JA3, and increasingly JA4), the shape of your TLS handshake, which most HTTP clients get wrong in a recognizable way

• HTTP/2 fingerprinting, the frame and settings ordering that Akamai in particular leans on

• Connection reuse and timing patterns

A browser claiming to be mobile Safari in Spain, connecting from a datacenter ASN with a Python-default TLS handshake, has three independent signals contradicting each other. Any one is forgivable. Together they’re a verdict.

What makes a fingerprint suspicious

Most fingerprints aren’t obviously fake. They’re built from real values: common browsers, standard resolutions, familiar operating systems. What anti-bot systems hunt for is how those real values fit together.

The first giveaway is a mismatch between attributes. Real devices follow predictable patterns. Certain browsers cluster on certain operating systems, resolutions cluster around popular hardware, fonts and rendering align with the platform. A macOS Safari profile serving Windows fonts, or a mobile User-Agent reporting a 2560-wide desktop viewport, breaks that pattern. Each discrepancy is explainable on its own. Stacked together, they aren’t.

Then there’s rarity. A perfectly unique fingerprint feels like good privacy and is terrible for blending in. Real users fall into large, overlapping groups of near-identical configurations. A profile built from attributes that almost never co-occur is both easier to track and quicker to flag. This is the over-randomization trap, and it’s the most common self-inflicted wound we see. People randomize every parameter and end up with a device that has never existed on Earth.

Consistency over time counts just as much. Real users are stable, same device, same browser, broadly the same setup from one request to the next. When one fingerprint shows up on an IP in one region and then again on an IP in another, or when key attributes drift between sessions, it reads as synthetic. Several IPs sharing a single identity is a pattern normal traffic just doesn’t produce.

Automation leaves its own tells. Even a carefully configured environment leaks something: a missing or stubbed API, a navigator.webdriver flag left on, slightly different JS execution paths, rendering quirks that don’t match a real build. No single tell is fatal. Detectors collect the small ones until the picture tips toward automation.

Last comes behavior. A flawless fingerprint still falls apart under robotic behavior. Pages loaded instantly with no intermediate navigation, clicks landing with pixel precision, sessions that fire one request and vanish, all of it undercuts the identity. Online, your identity is as much how you move as how you look.

Why fingerprinting breaks the old playbook

For years the routine was simple. Rotate IPs, randomize headers, respect rate limits. It worked because defenses were built around request-level signals, mostly where traffic came from and how fast it arrived. Fingerprinting moves the decision from the isolated request to a persistent identity, and most of the old tricks stop paying off.

Stateless requests became stateful identities. The old model judged each request on its own. Block one IP, grab another, keep going. Fingerprinting adds continuity. Change the IP and the browser environment can still look identical, or inconsistent in ways that link the sessions together. Rotating IPs no longer resets who you are.

Header randomization stopped working too. Rotating User-Agents and tweaking Accept-Language used to add diversity. Fingerprinting doesn’t read those values in isolation. It checks whether they agree with each other. Spin up a new User-Agent on every request while your rendering and JS execution stay constant, and you don’t look like many users. You look like one unstable client lying about itself. Tools like Apify’s fingerprint-suite and BrowserForge exist because building a coherent set of headers and fingerprint values is harder than randomizing them.

Datacenter-only setups draw scrutiny. Datacenter proxies are fast, cheap, and easy to rotate, but once the network is part of the identity that stops being free. A datacenter ASN has a different shape than residential traffic, and pairing one with a fingerprint that claims to be a consumer laptop is a contradiction. It won’t always block you, but it raises the baseline scrutiny, and stacked with any other inconsistency it pushes you over the line. This is why residential and mobile proxies have become table stakes on harder targets, even though they only fix the network half of the problem.

Detection also moved earlier. Older systems often gave you partial access before a rate limit kicked in. Now the fingerprinting script usually runs the moment the page loads, sometimes before any real content is served. Fail the check and you never reach the data. You get a CAPTCHA, a soft block, or a clean-looking empty response.

How to build a setup that survives fingerprinting

The goal isn’t to be invisible. It’s to be unremarkable, an identity where every layer tells the same ordinary story.

Run a real browser engine

Lightweight HTTP clients can’t fake Canvas, WebGL, or AudioContext, because those depend on a real rendering pipeline. So most serious setups drive an actual browser with Playwright, Puppeteer, or Selenium.

Vanilla automation still leaks, which is where the patched ecosystem comes in. playwright-stealth and puppeteer-extra-plugin-stealth paper over the best-known tells. undetected-chromedriver and its async successor nodriver harden Selenium-style automation, and patchright is a drop-in patched Playwright aimed at the same goal. When you need full managed profiles, anti-detect browsers like Camoufox (open-source, Firefox-based, with fingerprint injection built in), Kameleo, GoLogin, Multilogin, AdsPower, and Dolphin Anty handle the consistency for you. None of these is a magic bullet, since anti-bots study them too, but they clear the low-level signals that plain Playwright fails on day one.

Keep each fingerprint internally consistent

This is where over-randomization kills more scrapers than under-randomization ever has. Don’t change every parameter on every request. Give each session one stable, coherent identity, where the User-Agent matches the OS, the resolution matches the device class, and fonts, language, and timezone all tell the same story. If you generate fingerprints programmatically, lean on something built for coherence, like BrowserForge or fingerprint-suite, rather than rolling your own random values.

Align the fingerprint with the network

A fingerprint is never judged alone. Claim to be a mobile user in Spain while connecting from a datacenter in another region, and the whole session weakens. Treat proxies as part of the identity layer, not just a source of IPs:

• Residential or mobile proxies to match real-user network conditions

• Geo-targeted IPs so the IP, timezone, and language reinforce each other

• Session-based (sticky) routing for continuity across a session

And don’t forget the handshake. If your browser fingerprint is flawless but your TLS fingerprint is a Python default, JA3/JA4 will give you away. curl_cffi, curl-impersonate, and tls-client exist to make that layer match the browser you’re pretending to be.

Treat sessions as isolated identities

Each session should behave like its own user, with its own fingerprint, its own IP, and its own cookies and local storage. Don’t reuse one fingerprint across several IPs at the same time, and don’t let session state bleed between identities. In practice that means pairing one fingerprint to one IP for the life of a session, then rotating both together, the way a real device and its connection move as a unit.

Make the behavior match the identity

A perfect fingerprint can’t save robotic behavior. Real users don’t teleport onto deep pages, strip the data, and disappear. They land somewhere shallow, navigate, pause, scroll. Adding delays that reflect reading time, and routing through intermediate pages instead of hammering endpoints directly, does more for credibility than another round of fingerprint tuning.

Know when to stop building it yourself

Maintaining all of this is real, ongoing work, and the targets keep moving. Browser hardening, fingerprint coherence, proxy alignment, session isolation, behavioral pacing, every layer needs upkeep. At some point the maintenance cost outweighs the control you get from it, and pushing these layers down into infrastructure starts to make sense.

That’s the niche a managed scraper API fills. Instead of hand-coordinating proxies, browser environments, and anti-detection logic, you hand off the identity layer and keep your code about the data. Infatica’s Scraper API is built for that hand-off, with fingerprinting, session management, and request routing handled underneath, so what you maintain is extraction logic rather than evasion logic. It isn’t the right call for every project, but when anti-detection upkeep starts eating your roadmap, it’s a trade worth weighing.

Where this actually bites

Fingerprinting decides what data you can reach, how trustworthy it is, and whether your pipeline survives at scale. Getting past IP filtering is step one. Holding a believable identity for the whole session is the real game, and it’s where most of the projects we see actually break.

Price intelligence and travel feel it first. Travel platforms are among the most aggressive adopters. Prices move by location, device, and browsing pattern, so a setup that trips the defenses sees prices real users never see, or different ones every session. A hotel-pricing comparison is only as reliable as the identities behind the requests.

Search results are another. Search engines read the context of a query, not only the query itself. Weak fingerprinting earns an instant CAPTCHA. To capture what real users in a given region actually see, the requests have to look like they came from real users in that region, on consistent devices.

E-commerce monitoring runs into the same wall, and we’ve watched it happen most often on the luxury retailers sitting behind Akamai. Retailers tune availability, pricing, and promotions by user profile, and they use fingerprinting to tell shoppers from bots. Get flagged and you see hidden or reordered listings, throttling, or temporary bans, small distortions that quietly poison competitor-pricing analysis over time.

Ad verification and market research depend on seeing exactly what a real user sees: ad placement, localized copy, device-specific variants. Fingerprinting interferes by serving fallback content, blocking certain placements, or masking geo-targeted campaigns. That’s fatal when you’re validating campaigns across regions.

Account-based platforms lean hardest on fingerprinting to spot unusual access. These setups tend to fail not on a single request but because the session itself stops looking trustworthy. A sudden device-profile shift, an IP-location jump, or off behavior can each trigger a security check or kill the session.

Conclusion

Fingerprinting changed the question. It used to be “where is this request coming from?” Now it’s “does this client hold together as a believable identity, request after request?” Browser attributes, network signals, and behavior all feed that judgment, and the old reflex of rotating IPs and randomizing headers actively works against you when the layers don’t agree.

The setups that hold up aren’t the ones that randomize the hardest. They’re the ones that look the most ordinary: one coherent fingerprint, matched to one plausible network, behaving like a real person, held steady for the length of a session. The closer your setup sits to how real users actually browse, the more reliable your data collection becomes, and the more boring, in the best way.

Fortunately, not all providers have gone in that direction (think of the simple one-click Cloudflare Turnstile). However, many others are designing more intricate mechanisms to counter AI-based automation.

Is this the end of CAPTCHA solvers? And does this impact your scraping script? And how should we interpret these changes overall? Let me introduce you to the new generation of CAPTCHA systems to address all of these questions!

Give your AI a web data layer – Decodo's Web Scraping API turns any site into clean, structured data your models can actually use.

Activate starter plan

Why CAPTCHAs Have Become More Sophisticated

Before the rise of modern AI, CAPTCHAs could already be solved automatically using traditional machine learning techniques and browser automation. The approach was based on three steps:

1. Detect the challenge.

2. Recognize its content.

3. Simulate the required user interaction.

For example, image classification models can identify the correct images in reCAPTCHA v2 (”Select all images containing hydrants”):

(For more details on reCAPTCHA solving, see our previous articles: Part I and Part II.)

Similarly, OCR models could extract text from distorted image-based CAPTCHAs.

Combined with browser automation, these models were often accurate enough to solve many CAPTCHA challenges without human intervention.

A good example was Botright, an open-source browser automation library that integrated machine learning models to solve CAPTCHAs. The project has since been abandoned (its last commit was over two years ago), but it demonstrated how effective this approach could be.

I’ve personally tested similar techniques. While they were far from 100% reliable, they were still capable of passing many CAPTCHA challenges.

CAPTCHA providers have long been aware of these techniques. As automated solvers became more capable, they responded by making CAPTCHAs steadily more intricate. This led to the introduction of puzzle-based challenges, such as those used by AWS WAF CAPTCHA, hCaptcha, and other providers.

Yet, the rapid advancement of multimodal AI models has raised the bar even further…

Instead of training specialized ML models for each CAPTCHA type, LLMs can understand images, reason about what they see, and let the controlled browser interact with the page much like a human would. That has forced CAPTCHA providers to develop more sophisticated techniques to distinguish humans from bots.

This is where we stand today!

Are CAPTCHAs Dead Because of AI?

The short answer is: No!

While multimodal AI models have fundamentally changed how CAPTCHA providers think about distinguishing human users from automated bots, traditional CAPTCHAs are far from obsolete. There are at least three reasons why:

1. Not every bot uses AI: Many scraping bots still rely on traditional browser automation or simple scripts. They don’t have access to AI tools capable of understanding and solving CAPTCHA challenges.

2. LLMs are not deterministic: AI models can solve the same CAPTCHA challenge most of the time, but fail spectacularly on others. No wonder you can find examples on Reddit and X of AI-powered browsers repeatedly failing at CAPTCHA challenges.

3. Solving the challenge is only part of the verification process: CAPTCHA systems usually evaluate much more than just the answer to the test. They also analyze behavioral signals such as mouse movements, cursor acceleration, click timing, scrolling behavior, touch interactions, and many other indicators. That’s why one-click, press-and-hold, and slider CAPTCHAs exist. An AI model may correctly identify the solution, yet still exhibit interaction patterns that look “robotic” rather than “human,” causing the verification to fail.

That said, AI is undeniably making life harder for CAPTCHA providers. Numerous research papers have shown that, given sufficient training data and realistic browser automation techniques (such as human-like mouse movements, scrolling, and timing), AI systems can successfully solve many traditional CAPTCHA schemes.

The real question is no longer whether AI can solve older CAPTCHAs, as it clearly can. The more interesting question is: Can it reliably solve the new generation of AI-resistant CAPTCHAs?

New Approaches to AI-Resistant CAPTCHAs

More and more CAPTCHA providers are experimenting with advanced challenges specifically engineered to resist AI-powered solvers.

In this section, I’ll focus on three approaches to newer CAPTCHAs that I’ve personally seen challenge even some of the most popular CAPTCHA-solving solutions.

hCaptcha’s New Puzzle-Based Challenges

hCaptcha has always been one of the more difficult CAPTCHA providers to bypass, thanks to its creative human verification tests.

As AI capabilities have advanced, hCaptcha has responded by introducing elaborate puzzles purpose-built to thwart automated solving:

Notice how many of these challenges require much more than simply clicking on the correct images. They can involve drag-and-drop interactions, spatial reasoning, and specific knowledge.

Instead of fighting every CAPTCHA, focus on avoiding them. anyIP gives you access to high-quality residential and mobile proxies that help your traffic blend in more naturally.

Start with AnyIP

Microsoft’s Multi-Option CAPTCHAs

Explore the Microsoft forums and Reddit, and you’ll find plenty of users complaining about Microsoft’s newer human verification challenges (example 1, example 2, example 3). Here are a few examples:

These challenges can be somewhat ambiguous. They tend to present multiple plausible answers, require you to search through numerous options, and may involve several consecutive steps before the verification is complete. As a result, they take considerably longer to solve and can be frustrating for legitimate users (especially if a mistake forces them to restart the entire challenge).

Temu’s Proprietary CAPTCHAs

Some platforms are moving away from standard CAPTCHA providers and building their own verification systems in-house. A notable example is Temu, which has recently introduced a restrictive, custom CAPTCHA mechanism:

These challenges go beyond simple image selection and require very specific actions. For instance, they can involve multi-step interactions such as drag-and-drop tasks or slider-based tests that must be completed in a precise way.

What they generally have in common is the heavy use of AI-generated images. This creates an interesting dynamic: AI is being used to produce the very content to stop AI-powered bots.

The Future of CAPTCHA: What Comes Next

While these new CAPTCHA challenges are definitely more difficult for AI to solve, it’s still unclear whether AI will evolve quickly enough to keep up in the ongoing arms race between scraping developers and CAPTCHA-protected websites.

What’s clear is that providers will continue experimenting…

Hand-Gesture CAPTCHAs

Hand-gesture CAPTCHAs represent a shift from traditional image-based verification toward real-world interaction signals.

Google has recently begun testing a new reCAPTCHA system that asks users to prove they’re human by performing simple hand gestures in front of their device camera.

When enabled, the browser requests camera access and records a short video of the user’s hand movement. The system then analyzes the footage by extracting around 21 hand-landmark points (measurements of finger and joint positions) to identify specific gestures.

This is an attempt to improve “liveness detection,” making it harder for automated systems to pass as real users. Still, it’s not impossible that AI-driven tools could eventually simulate these gestures using AI-generated videos via camera injection.

Check the TWSC YouTube Channel

Stronger CAPTCHAs, Worse User Experience?

Newer CAPTCHAs are significantly harder for AI to pass, but they also introduce a new layer of friction for us, the everyday users. That’s the main trade-off in the current wave of anti-AI verification systems.

As models become more capable, CAPTCHAs are forced to evolve in parallel, often becoming more demanding not only for bots but also for legitimate users.

As a result, many users have been complaining on Reddit and X about frustration, increasingly time-consuming challenges, and privacy concerns (such as having to use a camera just to complete a CAPTCHA).

This has led to growing criticism that could eventually push providers to relax some of these verification mechanisms, either by making them easier or less frequent (which would be a win-win scenario for web scraping).

It’s quite clear that the real objective is finding a balance between effectively blocking bots and avoiding unnecessary friction for legitimate users.

Are These New CAPTCHAs Effective Against AI?

These newer CAPTCHAs aren’t becoming more elaborate simply for the sake of increasing difficulty. Providers are well aware that the probability of AI reliably passing these checks at scale remains low, though not zero.

The idea is to design challenges that are intricate enough to deter automation, while still incorporating sufficient interaction signals and behavioral data to assess whether the user is actually human.

In essence, beneath the surface, the core principle hasn’t fundamentally changed. What we are seeing is an evolution in execution, rather than a reinvention of the underlying idea.

Need help with your scraping project?

Avoiding CAPTCHAs Remains More Effective Than Solving Them

The final question is: “Should these new CAPTCHAs scare us, web scraping experts/enthusiasts?” In a sense, they might, but ultimately this is not really a revolution…

The truth is that in web scraping, the most effective strategy is rarely about solving CAPTCHAs. Instead, it’s about avoiding them altogether.

CAPTCHA systems are just one layer of a broader anti-bot ecosystem, and triggering them is a signal that the traffic is already considered low-trust or detectable.

For this reason, the focus should be on prevention rather than resolution. Best practices involve:

1. Routing traffic through high-quality residential proxy networks with strong reputation IPs.

2. Using stealth browser environments such as Kameleo Docker, Camoufox, or Rayobrowse.

3. Managing and rotating browser sessions in automated scraping workflows.

The end goal is to consistently emulate real, human-like browsing behavior to minimize the chances of detection. If human verification tests still appear, CAPTCHA solvers will remain the go-to option, as they must evolve to keep up with new challenges in order to stay in business.

The Web Scraping Club is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Conclusion

In this post, you’ve understood why CAPTCHAs have become more sophisticated and the role AI has played in this evolution. You’ve also seen what providers like hCaptcha, platforms like Microsoft, and sites like Temu are doing to defend against AI-powered automated scrapers and bots, as well as what the future of CAPTCHAs might look like.

I hope you now have a clearer understanding of the current and future CAPTCHA landscape. If you have any questions, feel free to leave them in the comments below. Until next time!

Did you like this article? Share it with someone who might find it useful and get a discount on paid plans.

Share

On July 2nd, the FBI, together with several industry partners, seized hundreds of domains tied to NetNut. For those who don’t know, NetNut is a large residential proxy service run by Alarum Technologies, a company that’s even listed on NASDAQ. The seized domains included netnut.com, proxyjet.io, and divinetworks.com.

This didn’t come out of nowhere. Let me walk you through how we got here, because the timeline is important to understand what happened.

How the takedown happened

It all started on June 19th, when three security firms published reports showing that NetNut’s residential proxy network was actually powering a botnet called Popa. The software was running on everyday devices like smart TVs and streaming boxes. Qurium, one of the security firms, stumbled onto this while looking into some denial-of-service attacks and traced the control servers back to pirated streaming apps. Then Brian Krebs made the story public, and about two weeks later, the authorities stepped in.

The takedown was a team effort: Google, the FBI, Lumen Technologies, The Shadowserver Foundation, and others all played a part. This is where things get interesting from a technical perspective, because taking down a proxy network is nothing like just unplugging a server.

Residential proxy networks are built to be decentralized. The exit nodes are actually millions of consumer devices scattered all over the world. Google estimated that NetNut had at least 2 million devices in its network. Obviously, you can’t just go and seize someone’s smart TV. What you can do is target the things those devices rely on to function as part of a commercial proxy service.

First, there’s the command and control layer. Google shut down the accounts and services that NetNut was using to control the malware. What I found interesting is that this C2 infrastructure was actually running on Google’s own services. This technique, called ‘living off trusted services,’ lets malicious traffic blend in with normal cloud traffic. Without command and control, there’s no way to route customer traffic through those devices.

Next is the enrollment pipeline. Google Play Protect began warning users and disabling apps that included NetNut SDKs, and it will keep blocking any new installs. If a proxy network can’t add new devices, it starts to shrink over time, since devices get reset, replaced, or just go offline.

Then there’s the commercial front door. By seizing the main domains, the FBI basically shut down NetNut’s storefront and made it impossible to bring in new customers at scale.

All of this together, according to Google, cut down the number of available devices by millions and seriously hurt NetNut’s business.

What makes this case different

I’ve seen proxy networks get taken down before, but NetNut is different. This wasn’t some anonymous group hiding on the dark web. NetNut was run by a company listed on NASDAQ, with investor relations, quarterly reports, and even a legal team that put out a statement saying they were aware of the seizure and cooperating with investigators.

It must be said that Alarum has rejected the accusations, disputing their accuracy. On the commercial and official side, NetNut has always presented itself exclusively as a legitimate proxy provider for enterprise data collection, without ever declaring any kinship with the clandestine Popa botnet. In its communications following the seizure, the company stated that neither Alarum nor NetNut had been formally contacted by the FBI or any other governmental authority in connection with these matters, while temporarily suspending data traffic through the affected network services in order to investigate the incident and assess the infrastructure involved. Markets, however, delivered their own verdict quickly: Alarum’s stock crashed by more than 70% at the end of last week, bringing its market cap down roughly 96% from its 2024 peak, and the company itself warned that prolonged disruptions are expected to have a material adverse effect on its operations and financial results.

It’s important to be clear about which part of NetNut’s network is involved here, because NetNut wasn’t just selling one type of proxy. They offered rotating residential proxies, static ISP proxies, mobile proxies, and regular datacenter proxies. Datacenter IPs are the least controversial, since they come from cloud and hosting providers with clear ownership. The takedown focused on the residential side, where NetNut actually used two very different sourcing models.

On one side there are the ISP proxies, sourced through DiviNetworks, the company NetNut originally grew out of. This is an above-board arrangement: as Spur documented in its research on how proxy providers co-opt entire networks, DiviNetworks recruits ISP partners with a straightforward revenue-stream pitch, terminates a GRE tunnel on the partner’s edge router, installs software on a VM inside the partner network, and uses policy-based routing to steer a slice of traffic through that VM and back out through NetNut’s infrastructure. From the partner’s perspective, a small amount of traffic is diverted; from NetNut’s perspective, selected egress from the partner’s public IP space becomes a commercial proxy product. The ISPs sign an agreement and get paid. You can argue about whether their subscribers know, but at least the contractual chain is in daylight.

The other side is the residential proxy pool, and this is where the Popa botnet comes into play. Researchers found that NetNut built this network by distributing SDKs for devices like smart TVs and streaming boxes. Synthient looked at over 20 real Popa publishers and didn’t see any of them asking users for consent. This is the part of the network that led to the takedown. When you have two million devices enrolled without real consent, that’s not a legitimate proxy network; it’s a botnet with a billing system, at least from a law enforcement point of view.

But the fact that divinetworks.com was seized along with the other domains tells me that investigators aren’t making a clear distinction between the different parts of the business. Even providers who think their sourcing is clean should take note.

This isn’t just a one-off event; it’s part of a bigger pattern

If this all sounds familiar, it’s because we’ve seen it before, and not long ago. The NetNut takedown follows right after Google’s disruption of the IPIDEA proxy network back in January 2026. At the time, IPIDEA was one of the biggest residential proxy operations out there, and NetNut’s main competitor.

Google’s post-mortem on IPIDEA included what I think is the key point here: when their own botnet starts to fall apart, proxy operators just buy capacity from competitors and become resellers. The proxy industry is tightly connected, and Google is pretty confident that a lot of popular residential proxy brands were actually just reselling NetNut’s botnet.

If you buy residential proxies from a brand you haven’t really checked out, pay attention. The IP pool behind your provider’s logo might be NetNut’s, or IPIDEA’s, or whoever is next in line.

The timing here is no accident. Bot traffic is exploding, especially with AI agents and LLM-based crawlers on the rise. In all my years doing this work, I’ve never seen web scraping infrastructure get this much attention from regulators and platforms. We’re in a real bind: we need more scraping and automation than ever, which means more proxies, but at the same time, there’s a real push for transparency about how these networks are built and used, so we can tell the good actors from the bad.

The lesson from NetNut is that just being transparent about where your IPs come from isn’t enough. You also need to know who is actually using your network. Alarum said they did KYC and due diligence on NetNut’s customers, but Spur found that NetNut didn’t really require corporate verification or real know-your-customer checks before selling proxy access. Down the line, resellers and white labelers usually didn’t do any KYC at all. Anyone who knew where to look could buy access with just a burner email and a few dollars in crypto. The result? In just one week in June 2026, Google saw 316 different threat groups using NetNut exit nodes, including cybercriminals and espionage actors.

What this means if you were a customer

Let’s be honest: losing a proxy provider isn’t really a technical problem. If you’re running a serious scraping setup, switching providers just means changing your endpoint. And if you’ve set up any kind of automatic fallback system, which you really should have, your pipelines probably won’t even notice the downtime.

The real headache is legal and compliance risk, especially if you’re working in a highly regulated industry.

Take financial markets, for example: I’ve worked in this space myself. Web scraping is a key part of what’s called alternative data: things like scraped prices, app usage stats, satellite images, and transaction data. Investment funds use these to get an edge on listed companies, alongside the usual financial statements and market data.

Anyone who has tried to sell web-scraped data to an investment manager knows what comes next: the due diligence questionnaire. I’ve been through this process, and it’s not a formality. The standard DDQ used in the industry, published by FISD’s Alternative Data Council, dedicates an entire appendix to web scraping and explicitly asks vendors whether they use proxies, whether those proxies are anonymizing, whether the proxy providers include information sufficient for scraped websites to contact the company, whether data is collected from pages behind logins or CAPTCHAs, and whether the vendor or its service providers have received cease and desist orders, subpoenas, or been subject to investigations in the past five years.

Now imagine you’re a data vendor facing that last question, and your proxy provider’s homepage is showing an FBI seizure banner. Telling a hedge fund that your infrastructure supplier is under federal investigation is not going to help you close any deals. In this business, your compliance is only as strong as the weakest link in your supply chain. Proxies are definitely part of that chain, whether your legal team knows it or not.

How I Think About Choosing Proxy Providers After NetNut

The reality is that residential and mobile proxies have always been the most controversial part of the proxy world. Unlike datacenter IPs or ISP deals, residential proxies need real people to agree to run a proxy on their devices in exchange for something, maybe money if they’re in a bandwidth-sharing program, or a free app, since developers get paid by proxy providers for every install.

Most people don’t realize how widespread this is. Spur scanned over 6,000 LG and Samsung smart TV apps and found that more than 2,000 of them had residential proxy SDKs inside. That means about a third of smart TV apps are quietly acting as proxy infrastructure. The real difference between legitimate and abusive setups comes down to consent and disclosure. Some providers, like Bright Data, Massive, and Oxylabs, at least spell out the arrangement in their terms of service, as Spur pointed out. For example, you can check Bright Data's peer program directly on bright-sdk.com, where you can find the certifications and compliance that they obtained.

Some providers have shown they’re willing to take a hit to keep their networks clean. For example, Bright Data has publicly shared that they dropped SDK partners who wouldn’t digitally sign their apps or who served ads leading to shady websites. These partners made up over 10% of their residential peer network, so it wasn’t a small decision. On the You might be skeptical about companies promoting their own good behavior, but I think this is the right direction: sourcing based on real standards, not just growth at any cost. They’re also founding members of the Alliance for Responsible Data Collection, together with other players like Sequentum and Common Crawl.

So here’s what I take away from all this. After what happened with IPIDEA in January and NetNut in July, picking a proxy provider based only on price isn’t just about quality anymore; it’s about risk. You need to ask the same questions the security researchers asked NetNut: How do you source your IPs, and can you prove consent? Who can actually buy access to your network, and what does your KYC process look like, even for your resellers? And what happens, both in contracts and technically, when abuse is found?

If your provider can’t answer these questions, someone else will, maybe with a seizure banner.

You buy proxies from a provider with the biggest IP pool you can find. You rotate IPs for every request. You randomize User-Agent strings. You connect to proxy servers from various regions. You add delays between requests. You expect to smoothly scrape the necessary data in no time, as you have made everything so your scraper does not leave any traces behind and is impossible to get caught. The next thing you see usually happens around request 150-300 - this is blocks. CAPTCHAs, error 403, redirects to honeypot pages serving you wrong data. The whole pipeline crumples down.

If this situation is painfully familiar, you have probably let session stickiness out of sight while building a proxy rotation layer - a common mistake. This article explains in detail how to rotate proxies to mimic real users so your scraper remains a reliable data source even at scale.

Why careless IP rotation may speed up bot detection

When you realize a website has blocked you out, the first instinct is to add more IPs - maybe buy proxies that perform better and are more adapted for dealing with detection systems. You try to add more geo-diversity. In other words, you use the same approach as before, just scale it. Sometimes it helps a bit; however, sometimes, blocks are back even faster.

The mistake here is that you are looking for a problem in the wrong place. It is not the lack of IP diversity, but the lack of session consistency.

Modern anti-bot systems like Cloudflare, Akamai Bot Manager, DataDome, or PerimeterX do not rely on the IP address as a primary method of bot detection. Instead, they consider a lot of signals and model session behavior. In such circumstances, your rotating efforts, if not strategic, can only make things worse and produce behavioral signals that are bot-like, not user-like. More IPs are not a fix for that problem. Modeling behavioral signals of genuine users is.

The anatomy of bot detection

In order to build a truly stealthy scraper with a rotation layer that gets the job done, it is important to understand first how detection happens - what exactly can give you away.

So, there are four layers of different signals that anti-bot systems analyze simultaneously and make a decision whether to let you be or deny access.

• Layer 1 - Network

IP reputation, ASN classification, type of IP (residential, mobile, or datacenter), and provider fingerprints belong to that layer. Also, if a proxy or VPN IP is known, it would also be shown here.

This layer is the one developers usually optimize the most for. Ironically, this is the least important layer for security systems. That is why you fail when following scraping tutorials that offer to simply set as big an IP pool as possible and rotate addresses for every request. It is not because you have done something wrong or your proxies are bad, but because this approach does not work for scraping at scale.

• Layer 2 - TLS/HTTP fingerprints

HTTP clients’ fingerprints are nearly as unique as an IP address. TLS handshake leaves quite a trace with its cipher suite order, list of supported extensions, and elliptic curves. HTTP/2 creates another signal with header prioritization, window size, and frame ordering. The combination of TLS and HTTP/2 fingerprints is often enough for security systems to figure out whether it’s human-generated or bot-generated traffic, even without considering signals from other layers.

Widely used scraping libraries, for example, curl, requests, httpx, aiohttp produce distinct, detectable fingerprints. Python is a popular choice for building scrapers now; still, if you use its requests library, a calibrated WAF would spot it from TLS handshake alone, User-Agent headers be forgotten, whatever they say.

• Layer 3 - Behavioral

Requests cadence, referrer chain integrity, cookie handling, navigation patterns, and timing distributions are evaluated here. Even the delays between requests, if they are too precise and equal, can fuel suspicions, as humans do not make requests with a machine’s precision. Real users’ referrers have logic to them. Human users’ cookies are consistent and show that they came from a search engine page. They do not disappear mid-session or mid-navigation.

• Layer 4 - Session Graph

This layer evaluates whether the sequence of cookies, tokens, page flow, and state transitions within the session makes sense for a human user. That is a layer most scrapes fail to pass, as not so many developers pay attention to coherence. Cause even if stand-alone signals look normal, it is important that they make sense when evaluated together. For example, a user visits an e-commerce website. Their session token would show that. Their viewport size would stay the same throughout the visit. They will go between pages with different time intervals, reflecting reading and decision time.

Coherence is the key that opens or closes a door for your scraper. And one proxy rotation at the wrong time can reveal your scraper’s true identity.

Session stickiness: a true solution

In the context of a proxy architecture, session stickiness is a condition when all requests that belong to the same logical session are routed via the same IP address. At the very least, it must be the same subnet, geo location, and ASN cluster.

Now, let’s define what a session may mean, cause in terms of scraping, there are several options possible.

• Authentication session

The most obvious case. You log in, receive a session token, and proceed to the data. If your IP changes somewhere between the login request and the first data request after authentication, to the target server, it looks like a user logged in from Paris and then suddenly that same user is asking for data from Seoul. Session invalidation or re-verification is guaranteed.

• Stateful pagination

This type is subtler and catches even sophisticated scrapers off guard. The thing is that many modern applications track your pagination state on a server’s side - in other words, a server “remembers” a page you are currently at and ties it to your IP. If the IP address changes between pages 3 and 4, the server loses this track. It may either block you or return incorrect data.

• Multi-step forms

Cases when you need to simulate checkouts, survey scrapers, or lead generation cases are the most session-sensitive. The state that was established at the beginning is carried forward across the rest of the steps. A mid-process IP change is a red flag in such a scenario; detection systems will react immediately.

To sum up, it is not random IPs you need, but sessions and a rotation strategy that considers session coherence.

Building the rotation layer

Time to get to work. Remember: rotating session objects, not IP addresses.

The session object model

A session object, in simple words, is a set of signals that make a set of requests look like they originated from a real human user.

@dataclass
class ProxySession:
    session_id: str
    proxy_endpoint: str          # host:port or provider session token
    cookies: dict = field(default_factory=dict)
    headers_profile: dict = field(default_factory=dict)
    geo: str = ""                # e.g. "pl,us,en"
    asn: str = ""                # e.g. "12345"
    created_at: float = field(default_factory=time.time)
    last_used: float = field(default_factory=time.time)
    request_count: int = 0
    flagged: bool = False

    def is_expired(self, ttl_seconds: int = 600, max_requests: int = 200) -> bool:
        age = time.time() - self.created_at
        return self.flagged or age > ttl_seconds or self.request_count >= max_requests

The headers_profile is not just a User-Agent string. It is a set of HTTP headers that are used for each and every request of one session: User-Agent, Accept-Language, Accept-Encoding, Accept, sec-ch-ua, sec-ch-ua-platform, sec-ch-ua-mobile. They must stay the same for every request. If one request is coming as if from Chrome browser version 108, and another one from version 124, detection systems will notice a mismatch. Even without knowing exactly that you are a scraper, such a mismatch is enough to block you.

Sticky session assignment

Any scraper must use the same session object within one session - this is a rule. So the session assignment must not be random, but based on that rule to prevent any mix-ups.

import hashlib
import threading
from typing import Dict

class SessionPool:
    def __init__(self, proxy_provider):
        self.provider = proxy_provider
        self._sessions: Dict[str, ProxySession] = {}
        self._lock = threading.Lock()

    def _make_key(self, job_id: str, target_domain: str) -> str:
        raw = f"{job_id}:{target_domain}"
        return hashlib.md5(raw.encode()).hexdigest()[:12]

    def get_or_create(self, job_id: str, target_domain: str) -> ProxySession:
        key = self._make_key(job_id, target_domain)
        with self._lock:
            session = self._sessions.get(key)
            if session is None or session.is_expired():
                session = self.provider.new_session(geo=self._infer_geo(target_domain))
                self._sessions[key] = session
            session.last_used = time.time()
            session.request_count += 1
            return session

    def _infer_geo(self, domain: str) -> str:
        # Simplified — in production, map TLDs/domains to geo requirements
        tld_map = {".pl": "PL", ".de": "DE", ".co.uk": "GB", ".fr": "FR"}
        for tld, geo in tld_map.items():
            if domain.endswith(tld):
                return geo
        return "US"  # Default

Notice that the session key is determined by job_id + target_domain, so the scraper sticks to the same session until it expires. The session expiration is also based on 2 rules - TTL (sessions older than 10 minutes automatically get refreshed) and request count (after 200 requests, the session gets refreshed to prevent statistical fingerprinting from long-lived sessions).

Geo-tied rotation

Another life hack is to rotate IPs within the same geo or ASN cluster even when a session has expired, and you are spinning to the new one. Even between sessions, sudden changes (like jumping from Tokyo residential IP to Toronto residential IP) are noticeable and suspicious. When that happens, it looks more like a VPN switch, not a user returning.

Result-oriented proxy providers offer sticky sessions. In such a case, a session token has a location backed into it, and the provider guarantees that the traffic will originate as if from that location during a session’s lifetime.

Header profile coherence

Not only must IP remain the same throughout the session. If you rely on the same address, but the header profile changes from request to request, it is like a “Hello, I’m a scraper” sign for detection systems.

HEADER_PROFILES = [
    {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
        "Accept-Language": "en-US,en;q=0.9",
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
        "Accept-Encoding": "gzip, deflate, br",
        "sec-ch-ua": '"Chromium";v="124", "Google Chrome";v="124", "Not-A.Brand";v="99"',
        "sec-ch-ua-mobile": "?0",
        "sec-ch-ua-platform": '"Windows"',
    },
    # Add more profiles reflecting real browser market share distributions
]

One session - one profile. No changes mid-session.

Retry logic

Retry logic is another thing that can blow your cover. When a request fails due to some reason and the target server instantly receives another one from the same origin, it most certainly means bots are in the game, as humans do not act with such light speed. An even less obvious trap is retrying with a different session object when your current session simply needs a brief backoff.

The better approach is to retry while relying on the same session object and a slight delay. If you receive a clear block signal, like 407, 429 errors with specific headers, or CAPTCHA, switch to the new session.

def fetch_with_retry(
    url: str,
    pool: "SessionPool",
    job_id: str,
    target_domain: str,
    max_retries: int = 3,
) -> requests.Response:
    failures: list[str] = []

    for attempt in range(max_retries):
        session = pool.get_or_create(job_id, target_domain)
        try:
            response = make_request(url, session)
            if response.status_code in (407, 429) or is_captcha(response):
                session.flagged = True
                failures.append(
                    f"attempt {attempt}: session {session.session_id} "
                    f"blocked (status={response.status_code})"
                )
                if attempt < max_retries - 1:
                    base = min(2 ** attempt * 2, 30)
                    time.sleep(base * (0.8 + 0.4 * random.random()))
                continue
            return response
        except (requests.Timeout, requests.ConnectionError) as e:
            failures.append(f"attempt {attempt}: {type(e).__name__}: {e}")
            if attempt < max_retries - 1:
                base = min(2 ** attempt * 2, 30)
                time.sleep(base * (0.8 + 0.4 * random.random()))

    raise MaxRetriesExceeded("All attempts failed:\n  " + "\n  ".join(failures))

There is an important difference: TransientError (which includes network hiccup and timeout) retries with the current session. BlockedError marks the session as flagged, which in turn triggers is_expired() and forces a new session creation.

Scaling concurrent sessions

At a small scale, the session pool model works just fine. However, if your scraper runs on 50+ machines simultaneously, you need all of them not to create new sessions each, but to stick to the one.

A Redis-backed session pool is a solution. Basically, Redis is a shared notepad that lives on a server and that every machine can read and write to, so that there is no chaos in sessions. Store each ProxySession object (serialized as JSON) with a TTL key:

import redis
import json

r = redis.Redis(host='localhost', port=6379, db=0)

def store_session(session: ProxySession, ttl: int = 600):
    key = f"session:{session.session_id}"
    r.setex(key, ttl, json.dumps(session.__dict__))

def load_session(session_id: str) -> Optional[ProxySession]:
    data = r.get(f"session:{session_id}")
    if data:
        return ProxySession(**json.loads(data))
    return None

If a proxy provider offers sticky sessions, your Redis only needs to track the session token, not the exit node state. A provider handling IP consistency and you dealing with cookies and headers is the preferred architecture for effective scaling because building and maintaining a distributed session pool is good 2-4 weeks of work. You need developers to do it - and add ongoing maintenance as websites update their detection systems.

Testing rotation layer

To be sure everything works fine, verify your rotation layer. Send 10 requests to see whether all of them really go via the same IP address.

import requests

def verify_session_stickiness(session_token: str, n: int = 10):
    proxy = f"http://user__sessid.{session_token}:pass@gw.dataimpulse.com:823"
    seen_ips = set()

    for i in range(n):
        r = requests.get("https://httpbin.org/ip", proxies={"https": proxy})
        ip = r.json()["origin"]
        seen_ips.add(ip)
        print(f"Request {i+1}: {ip}")

    print(f"\nUnique IPs: {len(seen_ips)}")
    assert len(seen_ips) == 1, f"FAIL: Session leaked across {len(seen_ips)} IPs"
    print("PASS: Session sticky across all requests")

verify_session_stickiness("your_session_token_here")

To verify headers, use https://httpbin.org/headers and compare headers across requests. Any differences in User-Agent, Accept-Language, or sec-ch-ua between requests of the same session are a problem.

If you want a deeper test, use bot.sannysoft.com or pixelscan.net. Those sites show what detection systems see - WebDriver flags, inconsistent browser APIs, JA3 anomalies, timezone/language mismatches. You see what exactly is wrong and at which layer. This is an easier, faster, and cheaper way than constant tries and errors.

Final thoughts

Scrapers that remain undetected are not the ones with the largest IP pools. They are the ones that mimic real users’ behavior the most persuasively - when the IP, the TLS fingerprint, the header profile, the cookie state, and the navigation flow all together create traffic patterns that are typical for a genuine human visitor.

IP rotation is necessary, but it is a tactic, while session architecture is a strategy. When you adopt that approach, you feel the difference - in lower block rates and higher data accuracy, in how much less firefighting your team goes through every time security systems update. Scalability is another benefit.

Also, there is an undeniable tendency for detection systems to develop, and so must your scrapers do. Rotation that is based on sessions, not single IPs, allows for that development.

If you want to skip the “build infrastructure from scratch” part, DataImpulse is an ethical proxy provider that offers both rotating and sticky sessions. You can set the duration of a session manually - up to 120 minutes is possible. DataImpulse (dataimpulse.com) provides 90M+ of residential, mobile, and datacenter enterprise-grade addresses in 195 locations with advanced targeting options (extra costs) to help maintain session coherence. Web scraping, SERP tracking, price monitoring, and other sensitive scenarios are among its use cases. Prices start from $1/GB - for non-expiring traffic and 24/7 human support.

Among the many tools a scraping engineer should master are headless browsers. But if you want to be completely honest, in the scraping industry there’s some confusion about the concepts of “headless browsers” and “browser automation tools”.

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

---

In this article, you’ll get your ideas clear on headless browsers and browser automation tools. You’ll also discover the Oxylabs Headless Browser and learn how to use it in practice against common anti-bots. In particular, you’ll see the difference in servers’ responses when hitting the same target URLs with and without the Oxylabs solution.

Let’s get into it!

What is a Headless Browser?

A headless browser is a fully functional web browser that operates without a graphical user interface (GUI). To understand what this means in practice, it helps to discuss what a browser actually does under the hood.

When you open a browser like Chrome and navigate to a URL, the browser performs a precise sequence of operations:

• It sends an HTTP request to a server.

• It receives an HTML document.

• It parses that HTML into a DOM (Document Object Model) tree.

• It fetches and applies CSS stylesheets to compute the visual layout, downloads and executes JavaScript files, and processes any asynchronous network requests (XHR/Fetch API calls) triggered by that JavaScript.

• It renders everything into visible pixels on your screen.

Below is an image that graphically shows the whole process:

A headless browser performs every single one of those steps, except painting pixels to a physical display. It still builds the DOM, executes JavaScript, processes AJAX calls, and maintains a fully accurate representation of the page state in memory. But it does it without needing a graphical rendering environment.

It’s also important to understand the difference between a headless browser and a basic HTTP client. Tools like Python’s requests library or curl are pure HTTP clients. This means that they send a request and return whatever raw response the server provides. They have no concept of a DOM, no JavaScript engine, and no rendering pipeline.

The most widely used headless browsers are the following:

• PhantomJS: One of the first dedicated headless browsers, built on an older version of WebKit. It was widely used before 2017 but has since been officially abandoned and is no longer maintained. You may still encounter it in legacy codebases, but it’s better not to use it for new projects.

• Headless Chrome/Chromium: Google officially introduced headless mode in Chrome 59 (2017), effectively making PhantomJS obsolete overnight. It exposes the Chrome DevTools Protocol (CDP) as its programmatic interface, which automation frameworks connect to to send instructions and receive data.

• Headless Firefox: Mozilla introduced headless support in Firefox 55. Powered by the Gecko rendering engine and the SpiderMonkey JavaScript engine, it’s a strong alternative to Chromium, particularly useful when you need to scrape or test behavior tied to the Gecko engine specifically.

• WebKit: WebKit is the browser engine that powers Apple’s Safari. Playwright bundles a version of WebKit that can be run in headless mode on non-Apple platforms, giving you the ability to simulate Safari’s rendering behavior on Linux or Windows servers (something that would be otherwise impossible).

• Splash: A lightweight, scriptable headless browser built specifically for web scraping, developed by the Scrapy team. It is written in Python and Lua, exposes an HTTP API, and integrates natively with the Scrapy framework, making it a beginner-friendly option for Python-based scraping pipelines.

Your scraping workflows deserve a proxy infrastructure that just works. With Swiftproxy on your side, consistency is built-in.

Try Swiftproxy today

Headless Browsers vs. Automation Frameworks: A Critical Distinction

At this point, it’s essential to draw a clear architectural boundary between two categories that are frequently, but incorrectly, used interchangeably: headless browsers and browser automation frameworks.

As said in the previous paragraph, a headless browser is the actual browser engine itself. So, it’s the software responsible for all the core browser operations described above.

A browser automation framework, on the other hand, is a programming library or tool that sits on top of a headless (or full) browser and provides a developer-friendly API to control it programmatically. The framework itself does not render pages, execute JavaScript, or manage network traffic: it delegates all of that to the underlying browser engine. What it provides is an abstraction layer to navigate to URLs, click elements, fill in forms, wait for conditions, intercept network requests, take screenshots, and to perform any similar activity.

Think of it this way: the headless browser is the engine of a car, and the automation framework is the steering wheel, pedals, and dashboard. You need both to drive, but they are fundamentally different components with different responsibilities.

This distinction also has a practical implication. If a website employs bot detection based on browser fingerprinting, it’s the underlying browser engine’s characteristics that matter, not the automation framework on top of it. This means that switching from Puppeteer to Playwright doesn’t change your browser fingerprint if both are driving the same Chromium engine with the same configuration.

The list below reports the programming libraries and tools used to control headless (or full) browsers programmatically:

• Puppeteer: A Node.js library developed and maintained by Google that provides a high-level API to control Headless Chrome/Chromium over CDP. It was the first major modern automation framework to replace PhantomJS and remains widely used in the JavaScript/Node.js ecosystem.

• Playwright: Developed by Microsoft, Playwright is a more modern and feature-rich automation framework that supports Chromium, Firefox, and WebKit under a single unified API. It has become the preferred choice for many scraping engineers due to its superior handling of async operations, automatic waiting strategies, built-in network interception, and robust multi-browser support. On the side of programming languages, it’s available for Node.js, Python, Java, and .NET.

• Selenium: The oldest and most widely adopted browser automation framework in existence, originally released in 2004. Unlike Puppeteer and Playwright, Selenium communicates with browsers via the W3C WebDriver protocol. This means it supports a broader range of browsers and is the de facto standard in enterprise testing environments. Selenium supports multiple languages, including Python, Java, C#, Ruby, and JavaScript, which gives it an exceptionally large community and an extensive ecosystem of documentation and tooling. For engineers working in Python, Selenium with a headless Chrome or Firefox driver is still a very common scraping approach. Its main trade-off is that it can be slightly slower and less ergonomic for modern async-heavy scraping scenarios.

Why Do Engineers Need a Headless Browser?

When you start defining how to scrape a website, your first instinct is often to reach for a lightweight HTTP client. And, in many cases, that is the right call. Simple, server-rendered websites that return complete HTML in their initial response are perfectly scrapable with tools like requests coupled with BeautifulSoup.

However, as you begin working with modern websites, you will quickly encounter scenarios where an HTTP client falls flat. Understanding why this happens (and when a headless browser becomes necessary) is one of the most fundamental skills you need to master for your career.

This section discusses the most common scenarios when you need to use a headless browser for scraping target websites.

JavaScript-Rendered and SPA Content

The most common reason to turn to headless browsers is client-side JavaScript rendering. A large proportion of modern websites, in fact, are built as Single Page Applications (SPAs) using frameworks like React, Vue.js, Angular, or Next.js. In these architectures, the server does not send a fully populated HTML page. Instead, it sends a minimal HTML shell along with a bundle of JavaScript files. The browser then executes that JavaScript, which makes additional API calls to fetch the actual data and dynamically constructs the DOM.

If you scrape the raw HTML with an HTTP client, you will receive that empty shell and find none of the content you are looking for. A headless browser, instead, loads the page as a real user’s browser would, waits for the JavaScript to finish executing, and exposes the fully populated DOM for you to extract data from.

AJAX and Asynchronous Data Loading

Closely related to SPAs is the broader pattern of asynchronous data fetching. Even on websites that are not full SPAs, it’s extremely common for certain page sections to be populated via AJAX requests that fire after the initial page load. These requests hit internal or external APIs, and the responses are injected into the page dynamically.

With a headless browser and a tool like Playwright, you can intercept these network requests directly. Also, this is a powerful web scraping technique. Rather than scraping the rendered HTML, you capture the raw API response (usually as JSON), which gives you clean and structured data to work with.

Infinite Scroll and Pagination

Many websites like social media platforms, job boards, and e-commerce product listings implement infinite scroll. This means that new content is loaded dynamically as the user scrolls down the page. On the practical side, this means that there is no “Page 2” URL to follow. The content doesn’t exist in the DOM until a scroll event triggers a new batch of data to load.

In this scenario, a headless browser allows you to simulate scroll events, wait for new content to be injected, and continue extracting data until a stopping condition is met. This is fundamentally impossible to replicate with a plain HTTP client.

User Interaction Simulation

Some content is only accessible after specific user interactions, like clicking on a button, selecting a value from a dropdown menu, submitting a search form, switching between tabs, and similar. A headless browser gives you a full interaction API that allows you to click on elements, type into input fields, select options, trigger keyboard shortcuts, and simulate virtually any user action.

This is particularly important when scraping search-driven websites like travel booking sites, real estate portals, and any similar websites where results are generated after the user performs actions.

Bot Detection and Browser Fingerprinting Evasion

Many high-value websites deploy anti-bot systems that analyze incoming requests to determine whether they originate from a real browser or an automated script. These systems are trained using machine learning models to check for dozens of signals. A headless browser provides a much more convincing browser environment than a raw HTTP client.

In particular, with additional configurations and stealth patches, you can significantly reduce the likelihood of your scraper being detected and blocked. Of course, this is an arms race: anti-bot technology continues to evolve, and a headless browser alone is not a silver bullet.

Check the TWSC YouTube Channel

What Is Oxylabs Headless Browser?

Oxylabs Headless Browser is a cloud-based browser automation solution that allows engineers to run and control remote browser sessions without the overhead of managing browser infrastructure locally or on their own servers. Rather than spinning up a local Chromium or Firefox instance on your own machine or cloud VM, Oxylabs Headless Browser offloads all of that complexity to dedicated remote servers maintained by Oxylabs.

The result is a production-ready and fully managed browsing environment that you connect to via a standard WebSocket (WSS) connection, and control using any automation framework that supports the Chrome DevTools Protocol (CDP) exactly as you would a local browser instance.

The image below shows, at a high level, the architectural difference between traditional browser automation and using Oxylabs’ solution:

The currently available browser instance is a Chrome-based environment (ubc.oxylabs.io). This runs high-performance Chromium instances with advanced stealth configurations on dedicated servers with integrated residential proxies.

Key Features

What differentiates Oxylabs Headless Browser from a self-managed setup is the set of capabilities that are baked directly into the infrastructure. Rather than requiring you to implement them yourself, it provides you with the following:

• Integrated residential proxies: Every browser session is automatically routed through Oxylabs’ residential proxy network. This eliminates the need to manage a separate proxy integration and ensures that outgoing requests carry legitimate residential IP addresses, reducing the risk of IP-level blocks.

• Automatic CAPTCHA solving: The platform detects and solves CAPTCHAs by default on page load. For CAPTCHAs that appear mid-session, it exposes an event-based API via the window object that your script can subscribe to in order to pause and resume automation around the solving process.

• Geolocation targeting: You can pin your browser session to a specific country, state, or city by appending query parameters directly to the WebSocket connection URL. This is essential for scraping geo-targeted content, verifying localized pricing, or accessing region-restricted pages without maintaining a geographically distributed proxy infrastructure yourself.

• Device type emulation: The p_device parameter allows you to emulate desktop, mobile, or tablet environments. This gives you access to device-specific content, responsive layouts, and mobile-only UI elements or CAPTCHA variants.

• Session inspection: This is a debugging feature that uses VNC (Virtual Network Computing) to give you real-time visual access to a running browser session. This is invaluable when diagnosing issues that are not apparent from logs alone. For example, for unexpected page rendering, broken interaction flows, or timing-related problems in your automation script.

• MCP integration: Oxylabs Headless Browser supports connection via the Model Context Protocol (MCP), enabling AI systems such as Claude Desktop or Cursor to drive browser automation sessions autonomously, which opens the door to AI-driven scraping and web navigation workflows.

How Oxylabs Headless Compares to Self-Managed Headless Browsers

The practical significance of Oxylabs Headless Browser is best understood by contrasting it with the self-managed alternative. When you run Playwright or Puppeteer against a local Chromium instance, you are responsible for the entire stack:

• Installing and maintaining the browser binary.

• Managing proxy rotation and authentication.

• Implementing CAPTCHA-solving integrations.

• Applying stealth patches to avoid fingerprinting.

• Handling concurrency limits.

• Ensuring your infrastructure scales with your scraping workload.

Oxylabs Headless Browser, instead, abstracts all of that away behind a single WSS endpoint. In practice, this means that your code remains identical to what you would write against a local browser. The connection string is the only thing that changes.

This makes Oxylabs’ solution particularly suited for teams that need to scale browser-based scraping quickly without investing in dedicated infrastructure engineering.

As a final note, it’s worth considering that billing is based on traffic consumed (per GB). This means traffic optimization becomes an important cost management practice when operating at scale.

Need help with your scraping project?

Testing Common Anti-bots The Hard Way: Step-by-step Tutorial on How to Hit Them Headless Mode

In this section, you will learn how to hit common anti-bots in headless mode using Playwright. The idea is to compare the results obtained in headless mode with the ones obtained using Oxylabs Headless Browser.

The target websites with their anti-bots are:

• Ssense → Cloudflare

• Gucci → Akamai

• Loccitane → DataDome

• Empire-cat→ Kasada

Let’s get into the tutorial!

Prerequisites and Requirements

To replicate this tutorial, you need at least Python 3.8 installed on your machine.

Create a new folder for your project:

mkdir test-antibots

At the end of this step, the test-antibots/ folder will have the following structure:

test-antibots/
   |
   |-- tester.py
   |
   |__ venv/

Where:

• tester.py is the Python file that contains the scraping logic to test the antibots.

• venv/ is the folder that contains the virtual environment.

All the files can also be downloaded from our GitHub repository, available to anyone.

In the activated virtual environment, install the needed libraries:

pip install playwright chromium

Very well. You are ready to code your tester!

Step #1: Code The Scraper to Test the Anti-bots

To test the antibots, write the following in tester.py:

from playwright.sync_api import sync_playwright

# Target pages with their anti-bots
targets = [
    ("<https://www.ssense.com/en-us/men/designers/acne-studios>", "Cloudflare"),
    ("<https://www.gucci.com/us/en/pr/women/shoes-for-women/sandals-for-women/flat-sandals-for-women/womens-slide-sandal-with-horsebit-p-834427AAE046273>", "Akamai"),
    ("<https://loccitane.com/en-us/olio-jeunesse-divine-30ml-27DH030I22.html>", "DataDome"),
    ("<https://www.empire-cat.com/equipment/used/catalog/aa/j1s03815>", "Kasada")
]

results = []

# List of titles or substrings that indicate common blocks/challenges
BLOCK_INDICATORS = [
    "just a moment", "access denied", "attention required", 
    "cloudflare", "ddos protection", "captcha", "verify you are human"
]

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    context = browser.new_context()

    for url, antibot in targets:
        page = context.new_page()
        try:
            # We capture the response to check the status code
            response = page.goto(url, wait_until="domcontentloaded")
            page.wait_for_timeout(5000)

            title = page.title() or ""
            status_code = response.status if response else 0
            
            # Validation Logic
            is_blocked = False
            error_reason = ""

            # 1. Check HTTP Status (403 Forbidden is a classic block)
            if status_code >= 400:
                is_blocked = True
                error_reason = f"HTTP {status_code}"
            
            # 2. Check for "Waiting Room" or "Block Page" titles
            elif any(indicator in title.lower() for indicator in BLOCK_INDICATORS):
                is_blocked = True
                error_reason = "Challenge/Block Page detected"
            
            # 3. Check for empty titles (Common with Kasada/DataDome blocks)
            elif not title or title.strip() == "":
                is_blocked = True
                error_reason = "Empty/N/A Title (Hidden Block)"

            if is_blocked:
                results.append({"url": url, "antibot": antibot, "title": title, "status": "fail", "error": error_reason})
                print(f"❌ [{antibot}] Blocked: {error_reason}")
            else:
                results.append({"url": url, "antibot": antibot, "title": title, "status": "ok"})
                print(f"✅ [{antibot}] Success: {title}")

        except Exception as e:
            error_msg = str(e).split('\n')[0] # Keep it brief
            print(f"❌ [{antibot}] Exception: {error_msg}")
            results.append({"url": url, "antibot": antibot, "title": None, "status": "fail", "error": error_msg})

        finally:
            page.close()

    browser.close()

# Print summary table
passed = sum(1 for r in results if r.get("status") == "ok")
failed = len(results) - passed

print("\n" + "=" * 105)
print(f"{'STATUS':<10} {'ANTIBOT':<12} {'RESULT/REASON':<30} {'URL'}")
print("=" * 105)
for r in results:
    status = "✅ PASS" if r.get("status") == "ok" else "❌ FAIL"
    antibot = r.get("antibot", "N/A")
    # Show the reason if it failed, otherwise show the page title
    display_info = r.get("error") if r.get("status") == "fail" else r.get("title", "N/A")
    print(f"{status:<10} {antibot:<12} {str(display_info)[:28]:<30} {r['url']}")
print("=" * 105)
print(f"✅ Passes: {passed}  |  ❌ Fails: {failed}  |  Total: {len(results)}")
print("=" * 105)

This snippet does the following:

• Maps a list of target URLs, each associated with their anti bot.

• Uses Playwright to launch a Chromium browser in “headless” mode.

• For each target URL, the script:

  • Opens a new browser page.

  • Attempts to navigate to the site and waits for the initial content to load.

  • Pauses for 5 seconds to see if the page successfully renders the correct content or if it gets blocked/redirected by the site’s security.

  • Attempts to extract the page title as proof of a successful load.

• Avoids the common trap of assuming a successful page load implies successful data access by checking the HTTP status code, title semantics, and structural integrity (line “N/A” titles)-

• Prints summarized results.

Perfect. The code is ready for launch.

Step #2: Launch The Tester And Retrieve Results

After launching the scraper, the results are the following:

STATUS     ANTIBOT      RESULT/REASON                  URL
==========================================================================================
❌ FAIL     Cloudflare   HTTP 403                       <https://www.ssense.com/en-us/>...
❌ FAIL     Akamai       Page.goto: net::ERR_HTTP2_PR   <https://www.gucci.com/us/en/pr/>...
❌ FAIL     DataDome     Empty/N/A Title (Hidden Bloc   <https://loccitane.com/en-us/olio->...
❌ FAIL     Kasada       HTTP 429                       <https://www.empire-cat.com/equi>...
===========================================================================================
✅ Passes: 0  |  ❌ Fails: 4  |  Total: 4
===========================================================================================

As you can see, the anti-bots successfully blocked 4 requests on 4:

• Cloudflare returned a 403 (forbidden).

• Akamai returns a Page.goto: net::ERR_HTTP2_PR error, which is often related to fingerprinting.

• DataDome reported an empty result, which suggests it may have activated a captcha.

• Kasada returns a 429 error. Generally speaking, a 429 error is a “too many requests” error, so the server blocks your scraper because you’re hitting it too fast. But specifically for Kasada, this often indicates a block due to fingerprinting.

Terrific! You made it to the end of this tutorial. Now it’s time to see what happens when implementing the Oxylabs solution.

Testing Common Anti-bots With Oxylabs Headless Browser: Step-by-step Tutorial

In this tutorial section, you will learn how to hit the same websites as in the previous paragraph using Oxylabs Headless Browser.

Prerequisites and requirements

Other than the requirements previously met, you need an active Oxylabs account in order to create your username and password for using its headless browser. Also, make sure to match all the requirements needed by the Oxylabs Headless Browser.

In the test-antibots/ folder, create a new Python file that will contain the scraping logic using the Oxylabs product:

test-antibots/
   |
   |-- tester.py
   |
   |-- tester-oxylabs.py
   |
   |__ venv/

Good. You are ready to proceed with the actual tutorial.

Step #1: Code The Scraper Using Oxylabs Headless Browser

Write the following code in tester-oxylabs.py:

import json
import random
import string
import os
from playwright.sync_api import sync_playwright

# Define params for connecting to Oxylabs servers
username = "<your-oxylabs-username>"
password = "<your-oxylabs-password>"
endpoint = "ubc.oxylabs.io"

# Create folder for screenshots
if not os.path.exists("screenshots"):
    os.makedirs("screenshots")

def get_session():
    return ''.join(random.choices(string.ascii_lowercase + string.digits, k=10))

# Define Chrome arguments
chrome_args = json.dumps([
    "--disable-blink-features=AutomationControlled", 
    "--enable-features=NetworkServiceInProcess"
])

# Define params for Oxylabs Headless Browser
params = f"p_device=mobile&p_cc=US&p_session_id={get_session()}&args={chrome_args}"
browser_url = f"wss://{username}:{password}@{endpoint}?{params}"

targets = [
    ("<https://www.empire-cat.com/equipment/used/catalog/aa/j1s03815>", "Kasada"),
    ("<https://www.ssense.com/en-us/men/designers/acne-studios>", "Cloudflare"),
    ("<https://loccitane.com/en-us/olio-jeunesse-divine-30ml-27DH030I22.html>", "DataDome"),
    ("<https://www.gucci.com/us/en/pr/women/shoes-for-women/sandals-for-women/flat-sandals-for-women/womens-slide-sandal-with-horsebit-p-834427AAE046273>", "Akamai")
]

# Define semantics to intercept in responses
BLOCK_TITLES = ["just a moment", "access denied", "attention required", "security measure", "not acceptable", "403 forbidden"]
results = []

def run_benchmark():
    with sync_playwright() as p:
        for url, antibot in targets:
            print(f"---\nTesting {antibot}...")
            try:
                browser = p.chromium.connect_over_cdp(browser_url)
                context = browser.new_context()
                page = context.new_page()

                captcha_detected = False
                def handle_log(msg):
                    nonlocal captcha_detected
                    if "oxylabs-captcha-solve-start" in msg.text:
                        print(f"   [Solver] {antibot} challenge detected...")
                        captcha_detected = True
                    elif "oxylabs-captcha-solve-end" in msg.text:
                        print(f"   [Solver] {antibot} resolved!")
                        captcha_detected = False

                page.on("console", handle_log)
                page.goto(url, wait_until="domcontentloaded", timeout=60000)
                
                # Wait for the page to stabilize
                if captcha_detected:
                    page.wait_for_timeout(15000)
                else:
                    page.wait_for_timeout(8000)

                # Make and save screenshots
                screenshot_path = f"screenshots/{antibot.lower()}_result.png"
                page.screenshot(path=screenshot_path, full_page=False)
                print(f"   [Camera] Screenshot saved: {screenshot_path}")

                title = page.title() or ""
                low_title = title.lower()
                is_blocked = any(bt in low_title for bt in BLOCK_TITLES)
                
                if len(title) > 5 and not is_blocked:
                    print(f"✅ {antibot} PASS: {title[:30]}...")
                    results.append({"url": url, "antibot": antibot, "status": "ok", "title": title})
                else:
                    print(f"❌ {antibot} FAIL: {title}")
                    results.append({"url": url, "antibot": antibot, "status": "fail", "error": title})

                browser.close()

            except Exception as e:
                print(f"❌ {antibot} Exception: {str(e)[:50]}")

run_benchmark()

Below is what changes with respect to the version in the previous paragraph:

• It defines the credentials for connecting to Oxylabs servers and creates a folder where it will later store screenshots.

• With chrome_args(), it defines some parameters for setting the automation in Chrome.

• In the string params, the p_mobile and *p_cc* respectively manage the perceived device type and IP.

• It manages CAPTCHAs with oxylabs-captcha-solve-start and oxylabs-captcha-solve-end. And since it needs to manage CAPTCHAs at each request, the code now has a timeout that helps the pages stabilize before the scraper passes on to the next target URL.

• It uses page.screenshot() to make screenshots after the server sends the request and saves them into the previously created folder. This is important because the exceptions can’t all be handled via intercepting HTTP responses or via semantics. By making a screenshot before the connection is closed, you can actually see if the scraper has passed the anti-bot or if it failed at it.

Good. You’re now ready to run the code and see the results.

Step #2: Launch The Script

After launching the scraper, the results are the following:

---
Testing Kasada...
   [Camera] Screenshot saved: screenshots/kasada_result.png
✅ Kasada PASS: 2021 CATERPILLAR 950M | Empire...
---
Testing Cloudflare...
   [Camera] Screenshot saved: screenshots/cloudflare_result.png
✅ Cloudflare PASS: Latest Acne Studios Collection...
---
Testing DataDome...
   [Camera] Screenshot saved: screenshots/datadome_result.png
✅ DataDome PASS: Immortelle Anti-Aging Face Oil...
---
Testing Akamai...
   [Camera] Screenshot saved: screenshots/akamai_result.png
✅ Akamai PASS: Slides for Women | Designer Sl...

As you can see, using the Oxylabs Headless Browser, the scraper overcomes all the anti-bots!

As a final note, consider that, based on the configuration and settings you will actually implement, DataDome (but also other anti-bots) may activate sliding captchas:

In this case, the Oxylabs Headless Browser is of no help as it only supports hcaptcha, recaptcha, and turnstile dynamic captchas.

Hooray! You successfully made it to your first scraping project using the Oxylabs Headless Browser.

Conclusion

In this article, you learned the actual distinction between headless browsers and browser automation tools. You also discovered the Oxylabs Headless Browser which, despite the name, it’s cloud-based browser automation. The difference between this and the browser automation tools you already know is that it’s cloud-based. This means that when it’s called via the code, it manages all the infrastructure on its own without overhead on your side.

As you learned in the tutorial section, it’s also pretty easy to configure and to use: you just need to manage some parameters and call the Oxylabs endpoint.

So, let us know: were you already using the Oxylabs Headless Browser?

Did you like this article? Share it with someone who might find it useful and get a discount on paid plans.

Share

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

---

The first is the UDP we want, HTTP/3 for our requests. The second is the UDP we never asked for, the WebRTC traffic that a browser generates on its own, which can leak around the proxy and expose our real IP. But supporting UDP and actually getting a client to route its UDP through the proxy are two different problems, and the result changes a lot depending on the client. An HTTP client, a browser doing HTTP/3, and a browser doing WebRTC all behave differently, so we tested all three, end to end, to see where a UDP proxy does the work for us and where we still have to step in on the client side.

A quick recap of HTTP/2 and HTTP/3

We already covered the request side of HTTP/3 in Faster Web Scraping with HTTP/3, so let’s just recap what matters for proxies. HTTP/1.1 and HTTP/2 run on TCP. HTTP/3 drops TCP completely and runs on QUIC, which is built on UDP and merges the transport and TLS handshakes into one. The part that matters for us is the transport: a proxy built to relay TCP has nothing to grab onto in a QUIC connection, since there’s no TCP stream to relay.

Adoption is real, but it’s easy to overstate. If we look at traffic, HTTP/3 is around a fifth of all requests, while HTTP/2 is still the majority at almost half, and this split has stayed roughly the same through 2026 (Cloudflare Radar). If we look at site support instead, the number goes up: around 38% of sites advertise HTTP/3 via Alt-Svc (W3Techs). Italy, where our test machine sits, is one of the higher HTTP/3 markets at about 30%. So HTTP/3 is a serious, well-supported protocol, not some niche thing, but it hasn’t replaced HTTP/2 yet, and most real traffic is still on TCP.

HTTP/3 and QUIC run on UDP, so the proxy layer has to carry it too. anyIP gives compatible clients production-ready SOCKS5 UDP ASSOCIATE across its residential and mobile network.

Explore anyIP UDP Proxies

The sales pitch follows naturally from here. If real browsers increasingly speak HTTP/3 and our scraper can’t, that difference becomes a signal, so we’d want to speak HTTP/3 too, and to do that through a proxy, the proxy has to carry UDP.

This is exactly what some providers are now selling. They offer UDP support, usually as a beta over SOCKS5, and they pitch it as a stealth feature: the idea is that most residential traffic is QUIC these days, so a client stuck on TCP stands out. There are two parts of this claim worth checking before relying on it, the stealth side and the tooling side, so we tested both, using a provider that carries UDP and one that doesn’t.

UDP support and the stealth question

Switching to HTTP/3 doesn’t remove protocol fingerprinting, it just moves it somewhere else. Detection happens at three layers, and moving from TCP to QUIC affects each one in a different way.

The TCP/IP stack fingerprint, meaning the SYN options, window sizes and TTL that give away the OS, really does disappear, because there’s no TCP handshake anymore. That’s the only layer that goes away, though. The TLS fingerprint stays exactly as it was, because QUIC still does a TLS 1.3 handshake, just wrapped inside QUIC instead of running over TCP. The ClientHello with its cipher suites and extensions is still there, and JA4 was designed to fingerprint the QUIC variant on purpose (Scrapfly QUIC/HTTP3 fingerprint). The HTTP/2 fingerprint, the SETTINGS frames, window updates and pseudo-header order, doesn’t get erased either. It just gets replaced by an HTTP/3 fingerprint built from QUIC transport parameters.

And those QUIC transport parameters are basically the same thing we were trying to get away from. Scrapfly says it clearly: they “work like HTTP/2 SETTINGS frames but at the transport layer.” Values like max_udp_payload_size and initial_max_data, and the order they come in, are different between Chrome’s quiche, Firefox, Go’s quic-go and ngtcp2, and that difference is itself a fingerprint (Scrapfly HTTP/2 and HTTP/3 fingerprinting). QUIC even adds new signals that have no TCP equivalent at all, like 0-RTT behavior and connection migration support.

So the layer that actually gets scrapers blocked today is TLS and HTTP/2 fingerprinting, which Cloudflare, Akamai and DataDome score together as JA4 and JA4H. That’s a TCP-side problem, and it’s one we already solve with a client that imitates a browser’s handshake. HTTP/3 fingerprinting is growing but still immature. Scrapfly is pretty direct about it: it “adds signals on top of” HTTP/2 instead of replacing it, real Chrome on HTTP/2 is still accepted on the vast majority of sites, and the scraper-on-HTTP/2 versus user-on-HTTP/3 gap is only “starting to become a signal on Cloudflare’s premium tier.” There’s also a catch worth keeping in mind: a scraper that speaks HTTP/3 through a non-browser QUIC stack shows a QUIC fingerprint that matches no real browser, so it can stand out just as easily as it blends in. In other words, a UDP proxy is what gets us onto HTTP/3 in the first place, but the stealth benefit only pays off if the client presents a QUIC fingerprint that a real browser would. The proxy just carries the traffic; making it look right is still our job.

Check the TWSC YouTube Channel

The tooling has not caught up yet

The other half of the pitch is the tooling. The most common client for browserless scraping is curl_cffi, so the real question is whether it can actually do HTTP/3 in the setup a scraper normally runs in. We tested it (run_curl_cffi_http3.py), and you can find the code, as always, on our GitHub repository available for our paid readers, under the folder 109.UDP

The library is compiled with the QUIC stack, ngtcp2 and nghttp3, and on a plain direct request it does reach HTTP/3. It even reaches HTTP/3 with a full Chrome fingerprint if we use the low-level interface, so impersonation and HTTP/3 aren’t really in conflict, even if the first error makes it look that way.

high-level requests API:
  plain, no impersonation, V3ONLY        -> HTTP/3
  Chrome impersonation, V3ONLY           -> ERROR: QUIC needs at least TLS version 1.3
  Chrome impersonation, HTTP proxy, V3   -> ERROR: HTTP/3 is not supported over an HTTP proxy
  Chrome impersonation, SOCKS5, V3       -> ERROR: HTTP/3 is not supported over a SOCKS proxy
low-level Curl API (same impersonation):
  impersonate chrome + V3ONLY            -> HTTP/3

Need help with your scraping project?

That middle error in the high-level API is just a TLS-version conflict inside the requests wrapper, not a real limitation, since the low-level call combines the Chrome fingerprint with HTTP/3 and gets there fine. The lines that really matter are the proxy ones. curl refuses HTTP/3 over a proxy completely, both HTTP and SOCKS, and it refuses before it even connects, so this is curl saying no, not the proxy. It happens even with a proxy that could carry the UDP. And since a scraper always runs through a proxy, in practice curl_cffi ends up being HTTP/2 over TCP, and a UDP-capable proxy doesn’t change that at all.

So this is the honest picture on the request side. A proxy that carries UDP does its job for a client that tunnels its own UDP, but the popular HTTP clients refuse HTTP/3 over a proxy right now, so the benefit is stuck waiting for the tooling to catch up. The proxy capability is arriving before the clients that could use it, which is good to know before expecting HTTP/3 to flow through a scraper just because the proxy supports UDP.

The WebRTC leak, where UDP really bites

Read more

However, web scraping can also serve as a foundation for building the data infrastructure needed to better understand national markets and societal conditions.

In this blog post, I’ll show real-world examples of how web scraping is already used in the public sector, and how institutions could leverage it to improve policies and deliver higher-quality services to citizens.

Before proceeding, let me thank NetNut, the platinum partner of the month. Their set of solutions cover all your needs for scraping.

Visit Netnut

---

Why Web Scraping Matters for Public Institutions

Public institutions have access to statistical datasets from official sources. These are surely relevant, but they tend to provide a picture of the market only after events have already happened, sometimes months or even a year later.

That delay creates a challenge. After all, institutions need timely information to understand citizens’ current needs and respond effectively. Another issue is that relevant data is often scattered across many sources, such as different websites, web portals, and marketplaces.

This is where web scraping can help! By collecting large volumes of publicly available data from multiple sites, institutions can complement and enrich official statistics.

Data collection is only the first step, though. The real value comes from building a complete data pipeline that includes cleaning, deduplication, aggregation, geolocation, statistical modeling, and interpretation.

At Prague Crawl 2026, Bayernheimerová Klára presented a compelling example of this approach. Her talk showed how the Czech Ministry of Finance relies on rental listing data collected by an external provider from several real estate portals and processes it through an end-to-end pipeline based on a dedicated statistical methodology.

As I’ll present shortly, the result is a reliable source of insights that supports housing policy decisions and affordable housing programs across the country, while also providing a valuable service to citizens.

What’s important to understand is that the goal isn’t simply to collect raw data, but to power services such as interactive maps, calculators, and dashboards that are useful to both public institutions and citizens.

Case Study: Czech Rental Market Intelligence System

To see an actual application of web scraping in the public sector, I’ll now present the initiative developed by the Czech Ministry of Finance for rental price analysis across Czechia.

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

The Motivation Behind the Project

Just like in many other European countries, housing affordability has become a major social and economic challenge in the Czech Republic. Housing supply has struggled to keep up with demand, causing both property prices and rents to rise steadily across the country.

To confront this issue, the Czech Ministry of Finance needed detailed and up-to-date information on local rental markets. The initiative was launched specifically to fill that gap, creating a web data analysis process that gives a continuously updated view of rental prices across the entire country.

The objective is to provide a reliable foundation for housing policy, affordable housing programs, and public-facing tools that help citizens better resonate with local rental markets.

Web Data Sourcing

The data for this initiative has been collected using Apify, one of the largest marketplaces of ready-made tools for web scraping, automation, and AI.

If you aren’t familiar with Apify, it provides over 41,000 ready-made serverless cloud programs (called Actors) to automate a wide range of tasks, including web scraping across thousands of different domains.

These Actors are developed and maintained by the community (and in some cases by Apify itself) and run on Apify’s cloud infrastructure. You can use them directly through Apify Console via a no-code interface, or call them programmatically via API. They can also be integrated into workflows such as n8n, Make, Zapier, or AI agents via MCP.

Two main reasons influenced the choice of Apify:

1. It’s a Czech-based company.

2. It enables the Ministry of Finance to gather large-scale, up-to-date information on rental listings without maintaining scraping infrastructure in-house.

In general, a solution like Apify lowers the barrier for public-sector teams, as these may have limited engineering capacity.

Downstream Data Pipeline

At a high level, this is the downstream data pipeline implemented by the Czech Ministry of Finance:

1. Data ingestion: Collects rental listings via Apify-based web scraping from multiple Czech real estate portals.

2. Data cleaning: Sorts data and removes duplicates, incorrect entries, and inconsistencies to ensure it is up to date, accurate, and free from deviations or distortions, following the procedure defined in Decree No. 456/2024 Coll.

3. Geolocation and aggregation: Standardizes addresses, assigns cadastral units, and enriches listings with attributes such as size, amenities, and building type.

4. Statistical modelling: Uses hedonic regression with spatial and temporal weighting to estimate underlying rental price levels.

5. Output generation: Presents results as interactive price maps and calculators.

Note: The pipeline also includes data update and recycling via a rolling window approach. This process adds new listings while retiring outdated observations to keep the dataset current.

To make rental prices comparable across different locations, the methodology is based on a reference apartment. That represents a standardized apartment profile with predefined characteristics, such as size, type, furnishing level, etc.

This approach reduces differences caused by individual property features and enables more consistent comparisons of rental price levels across municipalities and cadastral areas.

Produced Output and Tools

The Czech Rental Housing Price Map consists of two main solutions:

• Interactive rent price map: Provides estimated rental prices at the level of municipalities and cadastral areas across the Czech Republic.

• Market rent calculator: A practical tool that calculates a statistical estimate of rental price levels based on a standardized apartment profile and selected property characteristics.

Important: The two solutions aren’t intended to determine the market rent of a specific apartment. Instead, they return statistical estimates based on a standardized reference apartment. So, they’re intended primarily for market monitoring, regional comparisons, and housing policy analysis.

Interactive rent price map

The interactive rent price map shows minimum, maximum, and median rents for each area based on market listings, adjusted per square meter for a standard unfurnished reference apartment. It covers four size categories from 1+kk/1+1 to 4+kk/4+1.

Note that the map is interactive, and you can zoom in and out to explore individual cadastral areas. It’s also updated four times per year to ensure current data.

Market rent calculator

The market rent calculator is available as a form on the Czech government website. It works as follows:

1. Select the territorial unit where you want to estimate the rent.

2. Choose the size category of the apartment.

3. Enter the floor area of the apartment (if you don’t know it, the calculator will automatically use a default value for the selected location).

4. Indicate whether the apartment is in a new building.

5. Optionally specify whether the building uses non-standard construction materials (e.g., other than brick or panel).

6. Optionally select additional features such as a terrace, furniture, or an assigned parking/garage space.

7. Click “Calculate rent” to obtain an estimate of the monthly market rent (CZK) and the corresponding price level, which represents the final estimated rent for the selected apartment profile.

Impact

The project provides a systematic and regularly updated overview of market rental prices across the entire Czech Republic, supporting consistent monitoring of housing market developments over time and across regions.

Beyond market observation, the outputs serve as a key evidence base for housing policy design and housing programmes within public administration. For example, the State Investment Support Fund (SFPI) relies on these results when working with its affordable rental housing schemes.

Web Scraping in the Public Sector: Extending the Model

The project carried out by the Czech Ministry of Finance is just one example of a much broader pattern: how web scraping can be used as a foundation for modern public-sector data systems.

In particular, the same pipeline logic can be applied across many other domains. For example, in labor markets, web scraping can be used to collect job postings, salary ranges, and skill requirements from recruitment platforms. This can help governments identify regional skill shortages and design more targeted education or reskilling programmes.

Similarly, in consumer price monitoring, scraping can track grocery prices, housing costs, and essential goods across regions, enabling better inflation tracking and cost-of-living analysis. Other potential public domains that can benefit from this include energy consumption and transport accessibility.

When combined, those datasets become even more powerful. For instance, housing data, job market data, and grocery price data could be combined into a broader “liveability” index. That would help citizens and policymakers assess the overall affordability of living in a given region.

Check the TWSC YouTube Channel

Building a Job Market Choropleth for Czech Districts

In this section, I’ll guide you through building a job market choropleth across Czech districts. The idea is to show how web scraping, combined with a complete data pipeline, can be utilized for other scenarios beyond the previous example.

Important: The project below isn’t related to the initiative from the Czech Ministry of Finance and has been created purely for illustrative purposes.

Prerequisites

To follow this tutorial section, make sure you have:

• An Apify account (a free plan is sufficient).

• Python 3.11+ installed locally.

I’ll also assume you have a Python project set up locally with the following libraries installed:

pip install pandas geopandas geopy matplotlib

These are the required dependencies for this project and will be used as follows:

• pandas: Load, clean, and manipulate the scraped job posting data before geospatial analysis.

• geopandas: Handle geographic data, perform spatial joins, and create district-level maps.

• geopy: Convert job location addresses into geographic coordinates through geocoding.

• matplotlib: Visualize the results by generating a choropleth map showing job density by district.

Step #1: Access the Indeed Jobs Scraper Apify Actor

Just like in the Czech Ministry of Finance example, you can use Apify to collect the source data. This saves you from building and maintaining a complete job scraping pipeline from scratch.

For this project, we’ll use Indeed as the data source. Indeed has a Czech version, and most listings include the full office address where employees are expected to work.

In detail, I recommend using the Indeed Jobs Scraper [PPR] Actor. This automates the extraction of job titles, salaries, locations, company information, and job descriptions from Indeed.

To get started, log in to your Apify account and select the “Apify Store” option from the left-hand navigation menu. Then, search for “Indeed Jobs Scraper [PPR]”:

Click on the Actor card to reach its page. Great!

Step #2: Run the Job Scraping Task

On the Actor page, you’ll find an input form that lets you configure the scraper before running it in the cloud. Select “Czech Republic” as the “Country” and enter “Software engineer” in the “Query” field:

Note: In a real-world scenario, configure the Actor’s inputs according to your needs.

Next, toggle the option to avoid duplicates so that the Actor takes care of data deduplication for you:

Click the “Save & Start” button to launch the scraping task. The Actor will start running directly in Apify Console. As the scraper progresses, you’ll see the extracted job postings appear in real time. Be patient, as the process may take a few minutes depending on the number of matching listings.

Step #3: Explore the Output and Export the Results

Once the run completes, you’ll be able to explore the scraped dataset directly in Apify Console. As you can see, the Actor returns structured Indeed job data:

In this case, the Actor retrieved 63 job postings. That may seem like a small number, but keep in mind that the scraper targets only jobs published during the last 14 days. This helps ensure that the dataset reflects the current state of the job market.

Next, switch to the JSON view and select “All fields”. You’ll notice that each job posting includes a location object containing address information for the position:

That location data is exactly what you need to visualize job openings on a map and analyze their geographic distribution across Czech districts.

Finally, open the “Storage” tab, select the “JSON” export format, and click “Download” to export the scraped dataset:

A file with a name similar to dataset_indeed-scraper_2026-06-18_11-54-00-555.json will be downloaded. Rename it to jobs.json and place it in your Python project’s root directory.

Step #4: Download the Required Czech GeoJSON Data

To visualize the distribution of job openings across Czech districts, you’ll need a GeoJSON dataset containing the geographic boundaries of those districts.

Generally, that type of data is open and publicly available. For example, one possible source is the siwekm/czech-geojson repository, which provides GeoJSON files for various Czech administrative divisions:

Download the okresy.json file, which contains the geographic boundaries of Czech districts (okresy in Czech).

Note: Although the okresy administrative system was officially abolished in 2003, these districts are still used for statistical analysis. This makes them a good choice for visualizing the distribution of job opportunities across the country.

Once downloaded, add the okresy.json file to your Python project. At this point, your project structure should look similar to this:

├── jobs.json
├── okresy.json
└── main.py

Step #5: Visualize the Job Data on a Map

In your Python file, define the complete pipeline with the following code:

# pip install pandas geopandas geopy matplotlib

import json
from geopy.geocoders import Nominatim
from geopy.extra.rate_limiter import RateLimiter
import pandas as pd
import geopandas as gpd
import matplotlib.pyplot as plt

# Load the scraped Indeed job openings
with open("jobs.json", "r", encoding="utf-8") as f:
    jobs = json.load(f)

# Load the list of addresses from the job postings
addresses = []
for job in jobs:
    loc = job.get("location", {})

    # Prefer fullAddress, fallback to formattedAddressLong or formattedAddressShort
    address = (
        loc.get("fullAddress")
        or loc.get("formattedAddressLong")
        or loc.get("formattedAddressShort")
    )

    # Filter out full-remote jobs with no address
    if address and address.strip().lower() not in ["home office", "remote"]:
        addresses.append(address)

addresses = list(addresses)

# Initialize geocoder for Czech job locations
geolocator = Nominatim(user_agent="cz_job_map")
geocode = RateLimiter(geolocator.geocode, min_delay_seconds=1)

# Geocode all job addresses into latitude/longitude points
rows = []
for address in addresses:
    loc = geocode(f"{address}, Czech Republic")

    # Keep only successfully geocoded results
    if loc:
        rows.append({
            "address": address,
            "lat": loc.latitude,
            "lon": loc.longitude
        })

# Convert geocoded results into a DataFrame
df = pd.DataFrame(rows)

# Convert DataFrame into GeoDataFrame with point geometries
points = gpd.GeoDataFrame(
    df,
    geometry=gpd.points_from_xy(df.lon, df.lat),
    crs="EPSG:4326"
)

# Load Czech administrative districts (okresy) from a local GeoJSON file
okresy = gpd.read_file("okresy.json")

# Align coordinate reference systems between datasets
if okresy.crs != points.crs:
    okresy = okresy.to_crs(points.crs)

# Assign each job point to its corresponding district
joined = gpd.sjoin(
    points,
    okresy,
    predicate="within",
    how="left"
)

# Count number of jobs per district
counts = (
    joined["id"]
    .value_counts()
    .reset_index()
)

counts.columns = ["id", "count"]


# Merge job counts back into district geometries
okresy = okresy.merge(
    counts,
    on="id",
    how="left"
)

# Replace missing values with 0 (districts with no jobs)
okresy["count"] = okresy["count"].fillna(0)

# Plot choropleth map with job counts per district
fig, ax = plt.subplots(figsize=(10, 8))

okresy.plot(
    column="count",
    cmap="Reds",
    edgecolor="black",
    linewidth=0.3,
    legend=True,
    vmin=0,
    ax=ax
)

# Add labels for districts with job activity
top = okresy[okresy["count"] > 0].sort_values("count", ascending=False)

for idx, row in top.iterrows():
    centroid = row.geometry.centroid

    ax.text(
        centroid.x,
        centroid.y,
        str(int(row["count"])),
        fontsize=9,
        ha="center",
        va="center",
        color=(lambda v: "white" if v > okresy["count"].max() * 0.5 else "black")(row["count"]),
        fontweight="bold"
    )

ax.set_title(
    "Software engineering job density on Indeed in the Czech Republic (last 14 days, by district)"
)
ax.axis("off")

plt.show() 

Need help with your scraping project?

That’s what the above script does:

1. Loads scraped job data from jobs.json and extracts valid physical addresses, filtering out remote or missing location entries.

2. Initializes the Nominatim geocoder with a rate limiter to safely convert addresses into latitude and longitude without exceeding request limits.

3. Geocodes each address into coordinates and keeps only successful results, storing them in a structured pandas DataFrame.

4. Converts the DataFrame into a GeoDataFrame with point geometries so the job data can be used in spatial analysis workflows.

5. Loads Czech district boundaries from the okresy.json file and ensures both datasets use the same coordinate reference system.

6. Performs a spatial join to assign each job point to a district, then aggregates job counts per district and merges results back into the map.

7. Visualizes the final dataset as a choropleth map, coloring districts by job density and adding readable labels for high-activity areas.

Step #6: Run the Script

Execute the script, and you’ll get a result like this:

Notice how most fresh software engineering openings are concentrated in districts surrounding Prague, Brno, and Ostrava. These are the three largest cities in the Czech Republic, so the result clearly makes sense!

Now, this was just a simple example, but you can use the same approach to build an interactive map or add additional features, such as calculating the average salary for each position by district, and more advanced analytics.

Value and Playbook for Other Governments

The approach presented in this blog post is highly adaptable to any public dataset with a spatial dimension, where location plays a key role in understanding patterns and inequalities. Still, a reusable country-independent playbook emerges:

1. Collect data via web scraping (or other administrative, public sources).

2. Clean and validate the data.

3. Enrich and aggregate it with contextual and geographic features.

4. Apply statistical or analytical models.

5. Present the results through maps, dashboards, or other interactive tools.

As you can tell, this pipeline isn’t domain-specific and can be reused across many different policy areas. Finally, it’s crucial that the adopted methodology remains open and clearly explained to the public. This fosters transparency, reproducibility, and trust in data-driven decision-making.

Conclusion

Here, I’ve shown how the Czech Ministry of Finance uses a web scraping service as part of a full data pipeline to turn rental listings into actionable insights for housing policy and public tools.

By accessing data collected from multiple real estate portals and processing it through cleaning, geolocation, and statistical modelling, they transform raw web data into interactive maps and rent calculators that support decision-making.

In this article, I built a similar pipeline for a different use case: a job market choropleth across Czech districts. I started from scraped Indeed listings, geocoded job locations, mapped them to districts, and visualized the results on a map.

I hope this example was useful and inspiring. If you have questions or ideas, feel free to share them in the comments below.

Did you like this article? Share it with someone who might find it useful and get a discount on paid plans.

Share

Before proceeding, let me thank NetNut, the platinum partner of the month. Their set of solutions cover all your needs for scraping.

Visit Netnut

You can read that episode two ways. Either the guardrails were the only thing standing between a public model and a stream of attack-ready output, or the shutdown was a tidier story than the truth. We are not going to settle that here. For our corner of the world, the lesson is simpler. A capable model, stripped of guardrails or jailbroken around them, automates the expensive part of an attacker’s job. Deobfuscation, code analysis, and figuring out how an unfamiliar system behaves under pressure. Those tasks have always gated the scraper-versus-vendor arms race because they cost human reverse-engineering hours, and that gate is now moving.

The cat-and-mouse game we cover every week runs on a hidden economy. A vendor ships an obfuscated sensor. Someone on the scraping side spends days or weeks reading minified JavaScript, renaming variables, and rebuilding the logic until the parameters make sense. The cost of that labor is what keeps most protections standing. If a model can do a real share of it on its own, the math behind the whole arms race changes.

Manus AI, and why not Claude

Manus is not a chat assistant. It is an agent that pairs a language model with a sandboxed virtual machine, and inside that machine, it runs code, fetches pages, beautifies scripts, and iterates on its own output. Meta acquired it at the end of December 2025 for around two billion dollars, though by June 2026, Chinese regulators were already forcing parts of that deal to unwind. What that architecture buys is verification. Manus not only describes what a script might do. It runs the script and checks.

That difference shows up the moment you set guardrails next to it. Ask Claude or ChatGPT to deobfuscate a live anti-bot sensor and explain which signals feed its scoring, and you tend to hit a wall. Those products keep tight cybersecurity guardrails, and the Fable shutdown is the clearest sign yet of how hard that line is enforced. We handed the same class of task to Manus, and it got to work. We have looked at agentic browsers and AI scraping assistants before, but those were aids for a human operator who still did the thinking. Here, the model does the reverse engineering.

Start your scraping journey with Byteful: 10GB New Customer Trial | Use TWSC for 15% OFF | $1.75/GB Residential Data | ISP Proxies in 15+ Countries

Claim your 10GB here

Mapping Akamai on Net-a-Porter

We have written about Akamai more than any other anti-bot system. For the bypass-oriented background, THE LAB #30, THE LAB #85, and THE LAB #100 cover the recovery-when-nothing-works case, proxy chaining, and the browser-login-plus-HTTP hybrid. We will not repeat any of that.

What we wanted this time was not a bypass. It was a map. Akamai Bot Manager protects net-a-porter.com, and it defends in stacked layers that each fail differently:

• A TLS fingerprint check at the edge, evaluated before a single line of the client JavaScript runs.

• A sec-cpt cryptographic challenge, whose state lives in the bm_sc cookie.

• A behavioral sensor that serializes a binary telemetry payload and posts it to a collect endpoint, validating the _abck cookie.

• A client-side anti-hook layer that checks the browser environment for tampering and reports integrity separately.

The model we asked Manus to build covers the client sensor and the challenge flow, plus whatever it could observe about the TLS gate from the outside. A working end-to-end bypass was out of scope. We are not shipping code with this article, and we did not ask Manus to produce a runnable solver, even though our tests suggest we are really close to a working solution. What we’ll share here is a summary of the analysis Manus did; in the code repository, you will find the full details.

Check the TWSC YouTube Channel

What we handed the agent

The task was deliberately cold. We did not give Manus our own Akamai notes, our wiki, or any of the prior Lab articles. We pointed it at net-a-porter.com and asked it to study how Akamai's protection works, deobfuscate the client script, and explain which parameters feed into the sensor. Nothing more.

We chose this target for three reasons. It runs a real, current Akamai deployment on a high-value luxury store. The sensor was observable live, so the agent could fetch and instrument the actual script rather than reason about a stale sample. And we already understand Akamai well enough internally to judge whether the output was accurate or confident nonsense.

The reason the task went to Manus rather than a chat assistant is the guardrail gap from the previous section. Deobfuscating a production anti-bot sensor and enumerating the signals it scores is exactly the kind of request the heavily guarded models decline. Manus took it, ran the analysis in its VM, and came back with the two documents below.

What came back

The output was more detailed than we expected from a single task run. We are sharing it in two parts and deliberately not sharing it all.

The two documents below are Manus AI’s output. We reproduce them as reportage, to show what an agent derived on its own. We have not independently re-run every claim, re-derived every value, or confirmed that each detail still holds against the live site. Treat the specifics as the agent’s findings, not as TWSC-verified ground truth. The sensor version Manus reports, 746702557, was compiled on 2025-05-09, so some details may already have rotated. Where the agent surfaced operational specifics that amount to a turnkey recipe (exact module filenames, the full endpoint table, the CSS selector ids, the challenge request names), we have redacted them.

This is a schema of how Akamai Bot Protection according to Manus AI

Part one, reproduced in full, the anti-hooking and TLS summary

This first document is the safe exhibit. It explains how Akamai detects a tampered browser and how the TLS gate at the edge decides who even gets a challenge. We reproduce it verbatim.

<iframe height="0" width="0" style="display: none; visibility: hidden;" sc="true" tabindex="-1"></iframe>

Part two, reproduced with redactions, the deep technical analysis

The second document goes much further. It walks the bootstrapping sequence, the module system, the fingerprinting surface, and the telemetry transport. This is the part that starts to read like a blueprint, so we keep the architecture and the reasoning and we strip the operational specifics. Redaction markers below are ours, not Manus’s.

If you want to see the full analysis, you can find it in our GitHub repository reserved for paying users, inside the folder 108.AKAMAI-MANUS.

Read more

Before proceeding, let me thank NetNut, the platinum partner of the month. Their set of solutions cover all your needs for scraping.

Visit Netnut

IP sourcing in the proxy industry

Two articles written by Spur Intelligence caught my attention in the past week.
It started with a webinar I’ve attended, where they explained what ISP proxies are and how proxy companies get those IPs.

The analysis is great, and I’ve understood better how the proxy industry works.
They also mentioned how NetNut gets high-quality ISP IP addresses, and described that in detail in this article.

How Proxy Providers Co-opt Entire Networks

Spur reverse-engineers how Netnut sources its ISP proxies through a partner called DiviNetworks, which installs GRE tunnels and policy-based routing directly on the border routers of real ISPs. The result is genuine ISP IP space sold as proxy inventory, with actual subscribers still living on the same addresses. The piece is worth reading for the detection detail: outbound connections sit in the TCP source port range 40,000 to 40,200, partner routers expose an rtr-<isp>.divinetworks.com passive-DNS pattern, and the Howard University case shows an entire /16 (AS919, around 17,000 exit IPs) co-opted at the network edge. They even quote DiviNetworks’ own figure of $13,208 a month for a US /16.
Full article here

Smart TV Apps and Residential Proxy SDKs

A more recent article, instead, highlighted how proxy companies get residential proxies. No surprises, at least on my side, in reading that Smart TVs are used as exit nodes for proxies. In fact, just like with mobile apps, developers can use SDKs from proxy providers to monetize their app installations. In some cases, it’s quite clear from the consent screen what’s happening under the hood.

All the industries that rely on web-scraped data (AI in primis, but not only) need residential proxies, so proxy providers, who are the gatekeepers for these tools, should be very careful about onboarding users with legitimate use cases and keeping fraudsters away. In my career, I’ve worked with almost every big name out there, and I’ve found this is the case. As scraping professionals, we should always remember that we’re guests (sometimes unwanted) on both the target website and the proxy infrastructure, so we should be as respectful as possible and not be driven by greed to collect data.
Full article here

Scraping Infrastructure

A great article by the engineering team at Browser Use shows how they’re spawning browsers cheaply and quickly.

How We Made Cloud Browsers 3x Cheaper and Faster

Browser Use walks through rebuilding its cloud so every session is its own Firecracker microVM, and the interesting twist is that they run Firecracker on plain EC2 instead of bare metal, accepting nested virtualization to get faster scale-up and lower cost.

The numbers are great: $0.02 per browser hour down from $0.06, VM cold start under 400ms, and create latency of 825ms at p50 across a 10,000-session test.

The engineering details that are making the difference are multiple, from mapping memory in 2MB pages plus a userfaultfd handler that preloads hot pages (resume-to-ready drops from 9.8s to 3.1s, roughly 91x fewer page-fault stalls) to two-phase vCPU pinning that took a 1,000-browser launch from 17% failed sessions to zero. They also make the case for running fully headless rather than headful, since their low-level Chromium fork pushes block-avoidance to 81% on their own benchmark.

Full article here

Read and share more of this

Want to flag and discover more articles like the ones above? Scraping News (still in beta) is where the community surfaces them. Sign up and submit what you find worth reading.

Want a concept map of everything covered here, cross-referenced with our own work and other sources worth your time?

The Web Scraping Club Wiki lives on GitHub, and as an interactive site. It is plain Markdown, so you can also clone it into your own Obsidian vault and read it locally.

You set it up, it works, and you forget about it. Three months later, someone asks: “Hey, did our competitor change their pricing last week?”

You check the pipeline dashboard, and everything looks fine: all green, no errors, last run successful some hours ago. Then you dig deeper and realize your scraper has been returning empty strings for weeks. The competitor did change their pricing, and you don’t know when this happened or what was on their website before.

In this article, you’ll learn how to build competitive scraping pipelines that make failure structurally (almost) impossible. Let’s get into it!

Before proceeding, let me thank NetNut, the platinum partner of the month. Their set of solutions cover all your needs for scraping.

Visit Netnut

Why Competitive Intelligence Scrapers Break

Let’s start by understanding the most common failure modes that competitive scraping pipelines have:

• Structural drift: This is probably the most common. A competitor renames a CSS class, and your scraper starts returning empty strings. Your database fills with nulls, and nobody notices for weeks because the pipeline is still running.

• Soft blocks: These are nastier. You get valid HTTP responses, but the content is a CAPTCHA page or a bot-detection redirect. Your parser sees it as valid HTML, but it finds no data. So, it stores nothing, or, worse, it stores the CAPTCHA page’s text as if it were real data.

• Schema rot: This happens when a competitor evolves their product, but your data model doesn’t. They add a new pricing tier or split one pricing plan into three. Your scraper extracts what it can and drops the rest. Your competitive analysis is now based on an incomplete picture of their offering.

• Timezone and locale traps: This probably needs more attention than the others. The same competitor page can return different prices, currencies, or date formats depending on where the request originates. If your scraper runs from a US server but your competitor detects it and serves EU pricing, you’re tracking the wrong numbers.

What makes these failures dangerous is that you caused them yourself, unintentionally. Every defensive pattern you added to prevent crashes is exactly what turns a broken scraper into a quietly running one. No exceptions, no alerts, no red dashboards. Just weeks of nulls that nobody notices.

The rest of this article is about making that invisible failure visible again. Here’s a visual schema to get you introduced to the solutions proposed below:

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

The Monitoring Layer: Your First Line of Defense

The first thing you should do to avoid such failures is to implement a monitoring layer. This is the part that the majority of scraping engineers skip. Because, you know how things go: the business needed the data yesterday, and you can’t lose time with monitoring…

But if you want your competitive scrapers to be resilient over time, you need to implement a monitoring system. A good one is made on top of four parts: output validation, structural fingerprinting, data freshness checks, and canary fields. Let’s discuss them!

How to Validate What Your Scraper Extracted

Managing HTTP status codes in your scrapers is not enough. You need to assert that the data you extracted is actually meaningful. The following example is taken from Stripe’s pricing page:

Here’s a practical validator using Pydantic that you can implement:

# In an activated virtual environment run: pip install httpx, bs4, pydantic

import httpx 
from bs4 import BeautifulSoup
from pydantic import BaseModel, field_validator, ValidationError
from typing import Optional
import re

# Define Pydantic model
class CompetitorPricingData(BaseModel):
    plan_name: str
    monthly_price: float
    currency: str
    features: list[str]

    @field_validator("monthly_price")
    @classmethod
    def price_must_be_positive(cls, v):
        if v <= 0:
            raise ValueError(f"Price must be positive, got {v}")
        return v

    @field_validator("currency")
    @classmethod
    def currency_must_be_valid(cls, v):
        valid_currencies = {"USD", "EUR", "GBP"}
        if v.upper() not in valid_currencies:
            raise ValueError(f"Unexpected currency: {v}")
        return v.upper()

    @field_validator("features")
    @classmethod
    def features_must_not_be_empty(cls, v):
        if len(v) == 0:
            raise ValueError("Features list is empty: possible extraction failure")
        return v

# Define scraper logic
def scrape_stripe_pricing() -> list[dict]:
    url = "<https://stripe.com/pricing>"
    headers = {
        "User-Agent": (
            "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
            "AppleWebKit/537.36 (KHTML, like Gecko) "
            "Chrome/124.0.0.0 Safari/537.36"
        ),
        "Accept-Language": "en-US,en;q=0.9",
    }

    response = httpx.get(url, headers=headers, follow_redirects=True, timeout=15)
    response.raise_for_status()

    soup = BeautifulSoup(response.text, "html.parser")
    plans = []

    # Get class name from selectors
    plan_cards = soup.select("div.HeroPricingSubcard")

    for card in plan_cards:
        container = card.select_one(".HeroPricingSubcard__container")
        if not container:
            plans.append({
                "plan_name": "",
                "monthly_price": 0.0,
                "currency": "USD",
                "features": [],
            })
            continue

        # Get all text nodes inside the container
        text_blocks = [t.get_text(strip=True) for t in container.find_all(True) if t.get_text(strip=True)]

        # Pull the first block that looks like a price
        price_text = ""
        for block in text_blocks:
            if re.search(r"[\\d]+[.,]?[\\d]*%?\\s*\\+?\\s*[€$£]?[\\d]*", block):
                price_text = block
                break

        price_match = re.search(r"[\\d]+\\.?[\\d]*", price_text.replace(",", "."))
        monthly_price = float(price_match.group()) if price_match else 0.0

        if "$" or "¢" in price_text:
            currency = "USD"
        elif "€" in price_text:
            currency = "EUR"
        elif "£" in price_text:
            currency = "GBP"
        else:
            currency = "UNKNOWN"

        # Use all text blocks as features
        features = text_blocks if text_blocks else []

        plans.append({
            "plan_name": "Standard",  # HeroPricingSubcard is the Standard card
            "monthly_price": monthly_price,
            "currency": currency,
            "features": features,
        })

    return plans

# Define validator logic
def send_alert(message: str):
    # Wire this to Slack, PagerDuty, email, etc.
    print(f"[ALERT] {message}")

def validate_scraped_data(raw_data: dict) -> Optional[CompetitorPricingData]:
    try:
        return CompetitorPricingData(**raw_data)
    except ValidationError as e:
        # Don't silently swallow this — alert immediately
        print(f"[VALIDATION FAILED] {e}")
        send_alert(f"Scraper validation failed: {e}")
        return None

if __name__ == "__main__":
    raw_plans = scrape_stripe_pricing()

    print(f"Found {len(raw_plans)} plan(s) on Stripe's pricing page.\\n")

    for raw in raw_plans:
        validated = validate_scraped_data(raw)
        if validated:
            print(f"[OK]   {validated.plan_name} — {validated.currency} {validated.monthly_price}/mo")
            print(f"       Features: {validated.features[:2]}{'...' if len(validated.features) > 2 else ''}")
        else:
            print(f"[FAIL] Raw data that broke validation: {raw}")

Here’s what this code does:

• Defines a Pydantic model: It leverages Pydantic to define the expected shape and rules of the scraped data. It enforces that the price is positive, the currency is one of USD, EUR, or GBP, and the features list is not empty. Any violation raises a ValidationError.

• scrape_stripe_pricing(): Fetches Stripe’s pricing page using the library httpx, parses the HTML with BeautifulSoup, and extracts pricing data using CSS selectors. If a container isn’t found, it appends a zeroed-out dict, which will intentionally fail validation downstream.

• validate_scraped_data(): Passes each raw scraped dict through the Pydantic model. On failure, it calls send_alert() instead of silently swallowing the error.

• send_alert(): A stub you can wire to your alerting stack (Slack, PagerDuty, email) in a production environment. Note that, for educational purposes, throughout the entire article, this is used to print alerts via the command line.

The result is the following:

Found 2 plan(s) on Stripe's pricing page.

[OK]   Standard — EUR 1.5/mo
       Features: ['1.5% + €0.25', '1.5% + €0.25']...
[OK]   Standard — EUR 2.5/mo
       Features: ['2.5% + €0.25', '2.5% + €0.25']...

If it’s not clear why this matters, here’s what your scraper “sees” when it’s pushed from another location:

The class is the same, but the data (the currency) is completely different! This is a common example of why you need validation that goes beyond a specific location.

Start your scraping journey with Byteful: 10GB New Customer Trial | Use TWSC for 15% OFF | $1.75/GB Residential Data | ISP Proxies in 15+ Countries

Claim your 10GB here

Using DOM Fingerprints to Get Ahead of Structural Drift

This method is one of the most valuable things you can add to a competitive intelligence pipeline.

The idea is simple: periodically hash the DOM structure of the pages you’re scraping and store its value. When the hash changes, flag it for human review before your scraper breaks. Here is how you can implement it:

# In an activated virtual environment run: pip install requests

import hashlib
import httpx
from bs4 import BeautifulSoup
import requests

# Define parsing logic
def extract_structural_fingerprint(html: str, selector: str) -> str:
    """
    Extract a structural fingerprint from a specific section of the page.
    We hash the tag names and class names, NOT the content.
    This way, price changes don't trigger false alarms — only structural changes do.
    """
    soup = BeautifulSoup(html, "html.parser")
    container = soup.select_one(selector)

    if not container:
        return "CONTAINER_NOT_FOUND"

    # Build a structural signature
    structure = []
    for tag in container.find_all(True):
        classes = sorted(tag.get("class", []))
        structure.append(f"{tag.name}:{','.join(classes)}")

    fingerprint_str = "|".join(structure)
    return hashlib.md5(fingerprint_str.encode()).hexdigest()

# Compare hashes
def check_fingerprint(url: str, selector: str, stored_fingerprint: str) -> bool:
    """
    Returns True if the structure is unchanged, False if it has drifted.
    """
    response = requests.get(url, timeout=10)
    response.raise_for_status()

    current_fingerprint = extract_structural_fingerprint(response.text, selector)

    if current_fingerprint == "CONTAINER_NOT_FOUND":
        send_alert(f"[CRITICAL] Selector '{selector}' not found on {url}. Site may have been redesigned.")
        print("\\n[OK] Structure unchanged, safe to scrape.")
        return False

    if current_fingerprint != stored_fingerprint:
        send_alert(
            f"[WARNING] Structural change detected on {url} "
            f"(selector: {selector}). "
            f"Old: {stored_fingerprint[:8]}... New: {current_fingerprint[:8]}..."
        )
        print("\\n[WARNING] Structural change detected: hashes differ")
        print(f"  Old: {fingerprint[:8]}...")
        print(f"  New: {drifted_fingerprint[:8]}...")
        print("  → Scrape skipped. Human review required before next run.")
        return False

    return True

Below is a description of what this snippet does:

• extract_structural_fingerprint(): Parses the HTML, finds the target container via CSS selector, and builds a structural signature by iterating over every tag inside it. In this specific case, it records tag names and sorts class names, but deliberately ignores text content. That signature string is then hashed with MD5 and returned. If the selector finds nothing, it returns “CONTAINER_NOT_FOUND” instead of crashing.

• check_fingerprint(): Fetches the live page, computes the current fingerprint, and compares it against the stored one. Two failure cases are handled separately: the selector disappearing entirely (CONTAINER_NOT_FOUND) and the structure changing (hash mismatch). Both call send_alert() and return False: this signals to the caller to skip the scrape entirely.

• hashlib.md5(): Used to compress the structural signature into a short and comparable string. MD5 is not used here for security purposes; it’s used because it’s fast and collision-resistant enough for DOM comparison. Any change in tag names or class names produces a completely different hash.

In the case of a drifted fingerprint, the expected result is the following:

[STORED]  Fingerprint: 248363464b2da9b3814ea9a6dc5bd0df
[DRIFTED] Fingerprint: 7532ae7466c1f0fbc2c96471acd86ea7

[WARNING] Structural change detected: hashes differ
  Old: 24836346...
  New: 7532ae74...
  → Scrape skipped. Human review required before next run.

Note that this approach to DOM changes can be considered “classical” because it is fully based on finding a meaningful selector for the target page. The problem is that the DOM could change, but not the “meaningful” selector you choose. To improve it, you can target several selectors, or you can add a check on the whole body, but this would add more noise.

If you prefer trying a more “modern” approach that leverages LLMs to check on DOM changes, at The Web Scraping Club, we’ve already covered this in the article: “Beyond the DOM: A Practical Guide to Web Data Extraction with LLMs and GPT Vision”.

Using Canary Fields as a Heartbeat for Your Scraping Pipeline

A canary field is a field that you know changes frequently. This could be a “last updated” timestamp, a dynamic element, a session token in the page: you name it. You scrape it as a heartbeat, and if it stops changing, your pipeline is probably broken. Below is a snippet that checks on a canary field:

def send_alert(message: str):
    print(f"[ALERT] {message}")

def check_canary_field(
    current_value: str,
    stored_value: str,
    field_name: str,
    competitor: str,
    max_unchanged_hours: int = 48) -> None:
    if current_value == stored_value:
        send_alert(
            f"[WARNING] Canary field '{field_name}' for {competitor} "
            f"has not changed in over {max_unchanged_hours} hours. "
            f"Possible scraper failure or soft block."
        )
    else:
        print(f"[OK] Canary field '{field_name}' for {competitor} has changed. Pipeline looks healthy.")

When something changes, the expected output is the following:

[ALERT] [WARNING] Canary field 'last_updated' for <COMPETITOR NAME> has not changed in over 48 hours. 
Possible scraper failure or soft block.

Check the TWSC YouTube Channel

Building a Storage Architecture You Can Actually Trust

Many scraping engineers make the same mistake: they store only the processed output and throw away the raw HTML.

But in the case of competitive intelligence, that’s an issue that can cause you to work on weekends. Why? Because when a competitor changed their pricing 6 months ago, and you want to reconstruct the exact timeline, you need the raw data. Without it, history is gone forever.

The solution is a two-layer storage model: a raw and a processed layer. Here’s how the two relate:

             ┌─────────────────────────────────────────────────────┐
             │                     RAW LAYER                       │
             │  S3: raw/{competitor}/{page_type}/{date}.html.gz    │
             │  What's stored: full HTML, compressed, timestamped  │
             │  Why: so you can re-parse history if your logic     │
             │  changes or your scraper had a bug                  │
             └─────────────────────┬───────────────────────────────┘
                                   │ raw_s3_key (reference)
                                   ▼
             ┌─────────────────────────────────────────────────────┐
             │                  PROCESSED LAYER                    │
             │  SQLite/Postgres: pricing_history table             │
             │  What's stored: structured, queryable records       │
             │  Why: this is what your dashboards and alerts       │
             │  consume                                            │
             └─────────────────────────────────────────────────────┘

Basically, every record in the processed layer carries a raw_s3_key that points back to the exact HTML it was extracted from. That reference is what makes the whole architecture auditable.

The Raw Layer: Keep Every HTML Response

Store the full HTML or JSON response, timestamped, for every successful scrape. S3 is a cheap option to do so, and a year of raw HTML for a handful of competitor pages costs almost nothing. Below is an example you can implement in your pipeline:

# In an activated virtual environment run: pip install boto3

import boto3
import gzip
from datetime import datetime

def store_raw_response(
    html: str,
    competitor: str,
    page_type: str,
    bucket_name: str
) -> str:
    """
    Store the raw HTML response in S3 with a structured key.
    Compress it — HTML compresses extremely well (often 10:1).
    Returns the S3 key for reference in the processed layer.
    """
    s3 = boto3.client("s3")
    timestamp = datetime.utcnow().strftime("%Y/%m/%d/%H%M%S")
    s3_key = f"raw/{competitor}/{page_type}/{timestamp}.html.gz"

    compressed = gzip.compress(html.encode("utf-8"))

    s3.put_object(
        Bucket=bucket_name,
        Key=s3_key,
        Body=compressed,
        ContentEncoding="gzip",
        ContentType="text/html",
        Metadata={
            "competitor": competitor,
            "page_type": page_type,
            "scraped_at": datetime.utcnow().isoformat()
        }
    )

    return s3_key

def retrieve_raw_response(s3_key: str, bucket_name: str) -> str:
    """
    Retrieve and decompress a raw HTML response from S3.
    Useful for re-parsing historical data when your scraper logic changes.
    """
    s3 = boto3.client("s3")
    response = s3.get_object(Bucket=bucket_name, Key=s3_key)
    compressed_data = response["Body"].read()
    return gzip.decompress(compressed_data).decode("utf-8")

Notice the S3 structure: raw/{competitor}/{page_type}/{year}/{month}/{day}/{timestamp}.html.gz. This makes it trivial to list all scrapes for a specific competitor and page type within a date range, which is exactly what you need when reconstructing a pricing history.

Need help with your scraping project?

The Processed Layer: What Your Dashboards Actually Consume

The processed layer is your structured, queryable data: the thing your dashboards and alerts actually read. Every record here has two fields that make the whole architecture work:

• raw_s3_key, which links back to the exact HTML this record was extracted from.

• fingerprint, which captures the structural state of the page at scrape time.

The first one makes your data auditable. The second one tells you, later, whether the page had already changed when this record was written.

Below is a Python example to implement it:

import sqlite3
from dataclasses import dataclass
from datetime import datetime
from typing import Optional

@dataclass
class ProcessedPricingRecord:
    competitor: str
    page_type: str
    plan_name: str
    monthly_price: float
    currency: str
    scraped_at: datetime
    raw_s3_key: str  # The link back to the raw layer
    fingerprint: str  # The structural fingerprint at time of scrape

def store_processed_record(record: ProcessedPricingRecord, db_path: str) -> None:
    conn = sqlite3.connect(db_path)
    cursor = conn.cursor()

    cursor.execute("""
        CREATE TABLE IF NOT EXISTS pricing_history (
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            competitor TEXT NOT NULL,
            page_type TEXT NOT NULL,
            plan_name TEXT NOT NULL,
            monthly_price REAL NOT NULL,
            currency TEXT NOT NULL,
            scraped_at TEXT NOT NULL,
            raw_s3_key TEXT NOT NULL,
            fingerprint TEXT NOT NULL
        )
    """)

    cursor.execute("""
        INSERT INTO pricing_history 
        (competitor, page_type, plan_name, monthly_price, currency, scraped_at, raw_s3_key, fingerprint)
        VALUES (?, ?, ?, ?, ?, ?, ?, ?)
    """, (
        record.competitor,
        record.page_type,
        record.plan_name,
        record.monthly_price,
        record.currency,
        record.scraped_at.isoformat(),
        record.raw_s3_key,
        record.fingerprint
    ))

    conn.commit()
    conn.close()

A note on storage: this example uses SQLite, which is fine for a single scraper running on one machine. If you’re scraping multiple competitors in parallel or need concurrent writes, consider moving to Postgres (the schema remains the same).

A Concrete Example of Why the Raw Layer Saves You on Weekends

Imagine your competitor runs a Black Friday promotion in November and your scraper captures it. In January, your CEO asks: “When exactly did they drop prices, and by how much?”

With only the processed layer, you can answer that if your scraper was working correctly. But what if your scraper had a bug in November that caused it to extract prices incorrectly? With the raw layer, you can go back, fix the extraction logic, and re-parse the November HTML. Without it, that data is gone.

Here’s a utility function that makes re-parsing straightforward:

import boto3

def reparse_historical_data(
    competitor: str,
    page_type: str,
    start_date: str,
    end_date: str,
    bucket_name: str,
    new_parser_fn,
    db_path: str
) -> int:
    """
    Re-parse all raw HTML for a competitor/page_type within a date range.
    Useful when your extraction logic changes and you need to backfill.
    
    new_parser_fn: a callable that takes raw HTML and returns a list of dicts
    Returns the number of records reprocessed.
    """
    s3 = boto3.client("s3")
    prefix = f"raw/{competitor}/{page_type}/"

    paginator = s3.get_paginator("list_objects_v2")
    pages = paginator.paginate(Bucket=bucket_name, Prefix=prefix)

    reprocessed = 0

    for page in pages:
        for obj in page.get("Contents", []):
            key = obj["Key"]
            # Filter by date range based on key structure
            key_date = key.split("/")[3] + "-" + key.split("/")[4] + "-" + key.split("/")[5]
            if start_date <= key_date <= end_date:
                raw_html = retrieve_raw_response(key, bucket_name)
                parsed_records = new_parser_fn(raw_html)
                for record_data in parsed_records:
                    # Store with updated extraction, linked to same raw key
                    record = ProcessedPricingRecord(
                        **record_data,
                        raw_s3_key=key,
                        scraped_at=datetime.utcnow()
                    )
                    store_processed_record(record, db_path)
                    reprocessed += 1

    return reprocessed

Note that you’re not re-scraping anything. You’re just running your new parser against HTML you already have in storage. That’s the whole point of the raw layer.

Recovering From a Full Scraper Failure Without Losing History

Imagine this scenario: a competitor does a full site redesign, and your scraper is dead. This is not a hypothetical scenario: it happens more frequently than you can think, and most teams have no protocol for it.

Here’s a concrete recovery process.

How to Know Your Scraper Is Dead (Not Just Slow)

If you’ve implemented structural fingerprinting as discussed previously, you’ll know about the redesign before your scraper breaks. The fingerprint check will fail, you’ll get an alert, and you can investigate before storing bad data.

If you haven’t, and you’re reading this article after the fact, the first sign is usually a spike in validation failures or a sudden drop in scraped records. Consider implementing the following code:

import sqlite3

def detect_scraper_health(db_path: str, competitor: str, lookback_days: int = 7) -> dict:
    """
    Returns a health summary for a competitor's scraper over the last N days.
    Useful for spotting gradual degradation before it becomes a full failure.
    """
    conn = sqlite3.connect(db_path)
    cursor = conn.cursor()

    cursor.execute("""
        SELECT
            DATE(scraped_at) as scrape_date,
            COUNT(*) as total_attempts,
            SUM(CASE WHEN status = 'success' THEN 1 ELSE 0 END) as successes,
            SUM(CASE WHEN status = 'validation_failed' THEN 1 ELSE 0 END) as validation_failures,
            SUM(CASE WHEN status = 'error' THEN 1 ELSE 0 END) as errors
        FROM scrape_log
        WHERE competitor = ?
          AND scraped_at >= DATE('now', ?)
        GROUP BY DATE(scraped_at)
        ORDER BY scrape_date DESC
    """, (competitor, f"-{lookback_days} days"))

    rows = cursor.fetchall()
    conn.close()

    return [
        {
            "date": row[0],
            "total": row[1],
            "successes": row[2],
            "validation_failures": row[3],
            "errors": row[4],
            "success_rate": round(row[2] / row[1] * 100, 1) if row[1] > 0 else 0
        }
        for row in rows
    ]

The expected result is something as follows:

Date         Total     OK   Val.Fail   Errors     Rate
--------------------------------------------------------
2026-01-15      10      0          0       10     0.0%
2026-01-14      10      2          5        3    20.0%
2026-01-13      10      5          4        1    50.0%
2026-01-12      10      8          2        0    80.0%
2026-01-11      10      9          1        0    90.0%
2026-01-10      10     10          0        0   100.0%
2026-01-09      10     10          0        0   100.0%

The degradation pattern is immediately readable. A success rate dropping from 95% to 60% over three days is probably a sign of a redesign in progress. A drop from 95% to 0% in a short time signals a full redesign, or a hard block.

Diagnosing What Actually Broke Before You Rewrite Anything

Not every structural change is a full redesign. Rewriting the major part of a scraper can cost you days if not weeks. To avoid that, you can write a script that checks what actually changed, pointing to meaningful parts of the target website:

from bs4 import BeautifulSoup

def triage_structural_change(
    url: str,
    selector: str,
    old_fingerprint: str,
    bucket_name: str,
    competitor: str,
    page_type: str) -> dict:
    """
    Diagnose the nature of a structural change.
    Returns a triage report to guide the recovery effort.
    """
    response = requests.get(url, timeout=10)
    current_html = response.text
    soup = BeautifulSoup(current_html, "html.parser")

    report = {
        "url": url,
        "http_status": response.status_code,
        "selector_found": soup.select_one(selector) is not None,
        "old_fingerprint": old_fingerprint,
        "new_fingerprint": extract_structural_fingerprint(current_html, selector),
        "page_title": soup.title.string if soup.title else "N/A",
        "recommendation": None
    }

    if not report["selector_found"]:
        report["recommendation"] = "FULL_REWRITE: primary selector is gone. Site likely redesigned."
    elif report["old_fingerprint"] != report["new_fingerprint"]:
        report["recommendation"] = "PARTIAL_UPDATE:selector exists but structure changed. Review child selectors."
    else:
        report["recommendation"] = "FALSE_ALARM: fingerprint mismatch may be transient. Re-check in 1 hour."

    # Store the current raw HTML for reference during rewrite
    store_raw_response(current_html, competitor, f"{page_type}_triage", bucket_name)

    return report
    
def print_report(report: dict) -> None:
    print(f"  URL:             {report['url']}")
    print(f"  Selector found:  {report['selector_found']}")
    print(f"  Old fingerprint: {report['old_fingerprint'][:8]}...")
    print(f"  New fingerprint: {report['new_fingerprint'][:8]}...")
    print(f"  Page title:      {report['page_title']}")
    print(f"  Recommendation:  {report['recommendation']}")

The expected output for the case of a full rewrite is the following:

	URL:             https://...
  Selector found:  False
  Old fingerprint: 4a7f92bc...
  New fingerprint: CONTAINER_NOT_FOUND
  Page title:      Pricing page
  Recommendation:  FULL_REWRITE: primary selector is gone. Site likely redesigned.

So, in this case, the script checks a specific URL of the target website. But the code can be, of course, generalized to more than just one. The same applies to the target selector.

Estimating How Much of the Gap You Can Recover From Raw Storage

Once you’ve rewritten the scraper, you have a gap in your data. If you’ve been storing raw HTML, you can partially fill it using a snippet like the following:

import boto3

def estimate_gap_coverage(
    competitor: str,
    page_type: str,
    failure_start_date: str,
    bucket_name: str) -> dict:
    """
    Estimate how much of the gap period can be recovered from raw storage.
    """
    s3 = boto3.client("s3")
    prefix = f"raw/{competitor}/{page_type}/"

    paginator = s3.get_paginator("list_objects_v2")
    pages = paginator.paginate(Bucket=bucket_name, Prefix=prefix)

    available_keys = []
    for page in pages:
        for obj in page.get("Contents", []):
            key = obj["Key"]
            key_date = "/".join(key.split("/")[3:6])
            if key_date >= failure_start_date.replace("-", "/"):
                available_keys.append(key)

    return {
        "gap_start": failure_start_date,
        "recoverable_snapshots": len(available_keys),
        "oldest_recoverable": available_keys[0] if available_keys else None,
        "newest_recoverable": available_keys[-1] if available_keys else None,
        "recommendation": (
            "Run reparse_historical_data() with your new parser to backfill."
            if available_keys else
            "No raw snapshots available for this period. Gap cannot be recovered."
        )
    }

# Print values
print(f"Gap start:              {result['gap_start']}")
print(f"Recoverable snapshots:  {result['recoverable_snapshots']}")
print(f"Oldest recoverable:     {result['oldest_recoverable']}")
print(f"Newest recoverable:     {result['newest_recoverable']}")
print(f"Recommendation:         {result['recommendation']}")

The expected result is:

Gap start:              2026-04-12
Recoverable snapshots:  3
Oldest recoverable:     <COMPETITOR_NAME>/pricing/2026/04/12/143022.html.gz
Newest recoverable:     <COMPETITOR_NAME>/pricing/2026/04/14/143022.html.gz
Recommendation:         Run reparse_historical_data() with your new parser to backfill.

As a final note on this recovery section, consider that, in practice, these three steps run in sequence:

• When something breaks, you start with detect_scraper_health() to confirm there’s actually a problem and understand how long it’s been degrading.

• Then you run triage_structural_change() to understand what broke. That tells you whether you need a full rewrite or a 10-minute fix.

• Once you’ve updated your scraper, you call estimate_gap_coverage() to see how much of the gap period you can recover from raw storage, and then reparse_historical_data() to actually backfill it. If estimate_gap_coverage() comes back with zero recoverable snapshots, that gap is gone, which is exactly why the raw layer exists in the first place.

Conclusion

Building competitive intelligence scrapers is not as easy as it seems for two reasons:

• Developing the scrapers is only half the effort.

• Keeping the scrapers honest 12 months from now probably needs even harder efforts than coding the scrapers themselves.

The layers presented in this article can make your competitive intelligence scrapers resilient through time (with the right adjustments for production environments). So, if you’re implementing this from scratch, don’t try to build everything at once. Add one piece at a time and take your time to verify everything works fine before you add the next step.

Then, you just need to analyze the data you scraped (maybe directly implementing a dashboard in Streamlit), and your colleagues in the business department can have sweet dreams: their competitive analytics are safe through time.

So, let us now: are your competitive intelligence scrapers time-resilient?

Did you like this article? Share it with someone who might find it useful and get a discount on paid plans.

Share

Kameleo Docker brings Kameleo’s anti-detect browser capabilities into a containerized setup, enabling production-ready automation with real fingerprinting and multi-profile isolation on Linux servers.

In this post, I’ll take a deep look at this solution and walk you through everything you need to know about it. By the end, you’ll understand what Kameleo Docker is, how its stealth browser approach works, how to set it up, and whether it’s actually worth trying.

Before proceeding, let me thank NetNut, the platinum partner of the month. Their set of solutions cover all your needs for scraping.

Visit Netnut

An Introduction to Kameleo Docker

Dig into the world of Kameleo Docker!

What Is Kameleo Docker?

Kameleo is an anti-detect browser engineered to make browser sessions look like real user devices. Instead of exposing a generic automation fingerprint, it creates realistic browser identities by spoofing hardware, browser, and environment signals such as WebGL, Canvas, fonts, screen resolution, and geolocation.

Kameleo Docker brings that same stealth stack into a self-hosted, containerized deployment model. Rather than relying on a desktop app, you can run Kameleo inside Docker on Linux or Windows servers, CI pipelines, VPSs, or Kubernetes environments.

Playwright, Puppeteer, and Selenium can connect to the container via CDP, meaning you can keep using your existing browser automation logic with minimal changes.

Further reading:

• Kameleo Docker product page

• Kameleo Docker docs

• Kameleo on Linux via Docker: What We Built, What Broke, What’s Next

Why Kameleo Docker Exists

Cloud servers, Kubernetes clusters, CI/CD pipelines, and VPS environments are overwhelmingly Linux-native, making Linux compatibility a practical requirement for automation and scraping teams.

For years, this created a problem for Kameleo users, as the solution only supported Windows and macOS. Because of that, teams wanting to run stealth browsers in production often had to rely on fragile workarounds.

Some deployed Windows virtual machines alongside Linux scraping stacks in AWS. Others used X11-over-SSH tunnels to remotely access browsers running on servers. These setups were difficult to maintain, resource-intensive, and far from ideal for scalable automation.

As explained in the product announcement blog post, Kameleo customers started to ask for a version of Kameleo that could run directly where their automation already lived.

As Barnabas Szenasi, founder and lead engineer at Kameleo, explained when I met him at Prague Crawl 2026:

“We could see from customer messages that a significant slice of our automation-first users were running Linux cloud servers and simply couldn’t use Kameleo at all... At Prague Crawl 2025, Tamas [Kameleo’s CEO] and I heard the same story from industry peers around the world: scraping pipelines were getting harder, and the need to run real browser environments instead of faking HTTP requests was growing fast.”

The Philosophy Behind the Project

From the beginning, Kameleo’s philosophy has been simple: masking quality matters more than shipping quickly.

After all, an anti-detect browser is only useful if it can convincingly behave like a real device. That’s why Kameleo relies on fingerprints sourced from real-world device traffic and continuously tested against modern anti-bot systems.

That same quality-first mindset also shaped the Docker project. According to founder and lead engineer Barnabas Szenasi, Linux support took longer than expected because the goal was never just to make Kameleo run in a container.

The objective was to reach the same masking quality users already expected on Windows and macOS. Shipping a functional but lower-fidelity Linux version would have compromised the product’s core standard.

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

Technical Architecture Behind Kameleo Docker

Now that you know why the project exists, let me explain how it works and the engineering behind it.

How Kameleo Docker Works

Kameleo Docker runs inside either:

• a Linux-based container (Ubuntu 22.04), or

• a Windows-based container (Windows Server Core 2022).

When you pull the image, Docker downloads the correct variant based on your container configuration (Linux containers are the default in most environments, including Docker Desktop). Regardless of the underlying platform, Kameleo exposes the same Local API for browser and profile management as the desktop app.

The process begins when you create a browser profile through the Local API. If you aren’t familiar with that concept, Kameleo profiles are reusable browser environments that bundle a complete browser fingerprint together with persistent browser state, such as cookies, browsing history, local storage, and bookmarks.

Profiles can also include user-defined settings like proxies, browser extensions, and startup preferences. Each profile is tied to a specific browser kernel and can be started, stopped, imported, or exported as needed.

Architecture Overview

Kameleo Docker separates browser execution from automation logic. The container hosts the stealth browsers, fingerprinting systems, and orchestration layer, while your automation scripts run independently on your machine, server, or orchestration platform.

At the center of the architecture is the Local API, exposed on port 5050. This API handles profile creation, fingerprint selection, browser startup, lifecycle management, and more.

Behind the API sit Kameleo’s browser kernels:

• Chroma: A Chromium-based engine.

• Junglefox: A Firefox-based engine.

These kernels are modified with engine-level masking patches and connected to Kameleo’s continuously updated fingerprint database (more on this later). Since they are exposed through the same API, you can switch between them without making any code changes.

When a profile starts, Kameleo launches a browser session with the configured fingerprint and settings. Playwright and Puppeteer can then connect to the running browser through a WebSocket endpoint via Chrome DevTools Protocol (CDP).

In other words, your automation script stays outside the Docker container. The browser behaves as if it were running locally, while execution, fingerprint masking, and browser management happen entirely inside the Docker container.

Persistent storage is handled through Docker volumes. Profile data, downloaded browser kernels, and runtime state are stored outside the container, allowing environments to be recreated without losing configuration or repeatedly downloading browser components. This makes deployments easier to scale, recover, and reproduce.

Trusted by teams running ad verification, web scraping, SERP tracking, and market research. Ethically sourced proxies, globally accessible, and fairly priced.

Get Started With DataImpulse

Core Features of Kameleo Docker

Time to explore the main features and capabilities provided by Kameleo Docker. For more information, read the official documentation.

Real Device Fingerprint Masking

Kameleo fingerprints are derived from real-world device traffic, not synthetic templates. Each profile represents a coherent combination of OS, browser, hardware signals, and behavioral characteristics.

All fingerprint surfaces are kept internally consistent, including Canvas, WebGL, audio context, screen resolution, and fonts.

Note: TLS fingerprint spoofing isn’t required, as Kameleo matches the browser kernel version precisely. The TLS stack remains the original, unmodified implementation shipped with the corresponding browser release.

The goal of the project isn’t to spoof everything, but to maintain realism across signals. That’s because overriding too many surfaces increases inconsistency risk, which detection systems can flag. For example, running a macOS fingerprint on a Windows host forces heavy compensation across system-level signals.

Proxy Integration and Geo Consistency

Each Kameleo profile can be assigned a dedicated proxy (including a rotating proxy), allowing IP-level isolation between browser identities.

Now, mismatches between IP geography and browser signals (language, timezone, WebRTC, and system locale) are a common detection vector. To address that, Kameleo provides automatic geo-location matching to align the browser’s settings with the geographic location of the selected proxy IP address.

Multi-Profile Isolation

Kameleo Docker is built around strict profile isolation, as each browser profile runs as a fully independent environment. This separation opens the door to safe multi-accounting (referred to as “account management” in Kameleo terminology). Thanks for this feature, you can operate multiple identities simultaneously without cross-contamination of session data or signals.

Linux-Specific Docker Image Features

Compared to the Windows-based container, the Linux version of Kameleo Docker includes several additional features. These include:

• Built-in VNC viewer: Allows you to monitor and interact with live browser sessions. This is especially useful for debugging automation, validating fingerprints, or troubleshooting rendering issues. You can access it through a browser on port 8080 or via native VNC clients such as RealVNC or TigerVNC on port 5900. For security reasons, it’s disabled by default.

• Browser-Based Kameleo GUI: A lightweight browser-based GUI on port 80 (reach it at http://localhost:80). Unlike the desktop app, it offers reduced functionality and is primarily intended for quick inspection, basic profile management, and monitoring.

• Optional GPU acceleration: The Linux container supports optional GPU acceleration for graphics-heavy workloads such as WebGL or canvas-intensive websites. Intel/AMD GPUs can be mounted through /dev/dri, while NVIDIA GPUs are supported through the NVIDIA Container Toolkit. When no GPU is available, Kameleo falls back to software rendering.

Getting Started With Kameleo Docker: Step-by-Step Guide

In this guided section, I’ll show you how to set up Kameleo Docker and use it for browser automation against the Scraping Course’s “JavaScript Rendering” page:

This is a sandbox environment for web scraping that simulates a real-world, JavaScript-rendered ecommerce page. It makes for a great testing target to validate the setup and see how Kameleo Docker behaves in a realistic automation scenario.

Requirements and Prerequisites

To get started with Kameleo Docker, make sure you have Docker installed and running locally. (Optionally, you need a Kameleo account if you want to run Kameleo Docker with authentication.)

For more details on supported operating systems and memory requirements, refer to the official documentation.

Since I’ll show how to use Kameleo Docker with Playwright in Python, to keep things moving, I’ll assume you already have a Python environment set up with Playwright and its dependencies installed.

To follow along with this tutorial section, I also recommend that you have:

• Basic Docker experience (running containers, mounting volumes, and using compose files)

• Familiarity with browser automation using Playwright.

Check the TWSC YouTube Channel

Step #1: Create a Kameleo Account

If you haven’t already, start by creating a Kameleo account. Fill out the sign-up form and enter the required information. Once registration is complete, a Free plan will already be activated:

Note that the Free plan is enough to use Kameleo Docker.

Important: Kameleo credentials are required for the container to authenticate successfully, download browser kernels, and start correctly.

Step #2: Start the Docker Container

With your account ready, the next step is to pull and run the Kameleo Docker image.

Remember that Kameleo ships as a multi-platform Docker image, supporting both Linux-based and Windows-based containers. To download and start the Linux container version of Kameleo, run:

docker run --platform linux/amd64 \
    --shm-size=2g \
    -p 5050:5050 \
    -e PAT='<YOUR_KAMELEO_PAT>' \
    -v kameleo-data:/data \
    kameleo/kameleo-app:latest

Note 1: If you run this command in PowerShell, replace “\” with the backtick “`” for multi-line commands.

Note 2: To launch the web GUI included in the Linux container of Kameleo Docker, add the -p 80:80 argument to your docker run command.

Here’s what matters in the above command:

• --platform linux/amd64 ensures Docker pulls the Linux-based image variant.

• --shm-size=2g is required for stable browser execution (the default Docker shared memory of 64MB is too small for browser execution).

• -v kameleo-data:/data creates a named volume that persists browser kernels and profiles across restarts.

• <YOUR_KAMELEO_PAT> is your Kameleo Personal Access Token (PAT), required to authenticate with your Kameleo account. Note that this argument is optional. You can skip the -e PAT=’<YOUR_KAMELEO_PAT>’ flag entirely if you want to try out Kameleo Docker without an account.

Below’s the output you should get:

The Kameleo Docker image should now be downloaded and launched on your system. Cool!

Step #3: Verify the Service

Once you run the image, Kameleo Docker will:

1. Start the Local API on port 5050.

2. Authenticate using your credentials.

3. Download required browser kernels (first run only).

To confirm everything is running correctly, visit the following URL in your browser:

http://localhost:5050/swagger

You should see the Swagger UI for the Kameleo Local API:

You’ll have a local Kameleo instance ready for automation with Playwright, Puppeteer, or Selenium. Great!

Step #4: Download the SDK and Create Your First Profile

Now that Kameleo Docker is running, you can interact with it through the APIs exposed at http://localhost:5050. The next step is to proceed with the usual Kameleo setup by creating a profile.

Assuming you already have a Python environment with Playwright installed,start by installing the Kameleo SDK:

pip install kameleo-local-api-client

Then, in your Playwright script, initialize the Kameleo API client and generate a browser profile:

from kameleo.local_api_client import KameleoLocalApiClient
from kameleo.local_api_client.models import CreateProfileRequest

# Connect to the Kameleo API client
client = KameleoLocalApiClient(endpoint="http://localhost:5050")
# Search for a real-world fingerprint and create a Kameleo profile based on it
fps = client.fingerprint.search_fingerprints(
    device_type="desktop",
    os_family="windows",
    browser_product="chrome",
    browser_version=">145",
)
profile = client.profile.create_profile(
    CreateProfileRequest(fingerprint_id=fps[0].id, name="twsc demo")
)

The above snippet connects the Kameleo API client from the SDK to the local Kameleo Docker APIs. It retrieves a realistic browser fingerprint from the database and creates a persistent browser profile called “twsc demo” based on it. In this case, the fingerprint profile is for a desktop Chrome browser (version >145) running on Windows.

Run the script above. If you started the Linux container of Kameleo Docker while mapping port 80 for the web GUI, then you’ll be able to see the “twsc demo” profile at http://localhost:80:

Step #5: Connect With Playwright

You can now connect Playwright to the Kameleo profile created above via CDP using the following WebSocket URL:

ws://localhost:5050/playwright/<KAMELEO_PROFILE_ID>

Achieve that by using Playwright’s connect_over_cdp() method on the target URL:

from playwright.sync_api import sync_playwright

# Kameleo profile creation...

# Connect Playwright to the Kameleo instance based on the configured profile
ws_endpoint = f"ws://localhost:5050/playwright/{profile.id}"
with sync_playwright() as playwright:
    browser = playwright.chromium.connect_over_cdp(endpoint_url=ws_endpoint, timeout=90_000)

    # Regular Playwright automation logic...

Wonderful! Playwright is attached to the browser session managed by Kameleo Docker. You can now automate it using standard Playwright APIs as if it were a regular local Chromium instance.

Step #6: Implement the Automation Logic

To achieve the scraping goal, begin by inspecting the page to study its DOM structure:

Then, apply the following Playwright logic (connected to a Kameleo profile) to automate scraping on the JavaScript-rendered page:

# Connect Playwright to the Kameleo instance based on the configured profile
ws_endpoint = f'ws://localhost:5050/playwright/{profile.id}'
with sync_playwright() as playwright:
    browser = playwright.chromium.connect_over_cdp(endpoint_url=ws_endpoint, timeout=90_000)

    # Open a new page
    context = browser.contexts[0]
    page = context.new_page()

    # Visit target site
    page.goto("https://www.scrapingcourse.com/javascript-rendering")

    # Where to store the scraped data
    products = []

    # Wait for products to render
    page.wait_for_selector(".product-item")

    # Locate all product items
    product_elements = page.locator(".product-item")

    for i in range(product_elements.count()):
        # Select the nth product
        product_element = product_elements.nth(i)

        # Extract the product data
        name = product_element.locator(".product-name").inner_text()
        price = product_element.locator(".product-price").inner_text()
        image = product_element.locator("img.product-image").get_attribute("src")
        link = product_element.locator("a.product-link").get_attribute("href")

        # Populate a product object with the scraped data
        product = {
            "name": name,
            "price": price,
            "image": image,
            "url": link
        }
        # Append it to the products list
        products.append(product)

The above snippet instructs the controlled browser to visit the target page, waits for product elements to load, then iterates through each product DOM node to extract structured fields (name, price, image, URL) and stores them in a Python list for downstream processing.

The Kameleo-powered automation script is almost complete. Only one step remains!

Step #7: Stop the Kameleo Profile

Normally, in a Playwright scenario, you would need to call browser.close() to terminate the browser session and release its resources.

In Kameleo, that’s not required. Instead, you only need to call:

client.profile.stop_profile(profile_id=profile.id)

The above line of code sends a close command to the browser via CDP. Once the browser actually stops, the Kameleo profile is terminated, too. This ensures that all resources associated with both the browser and the running profile are properly released.

Step #8: Run the Script

The final Playwright automation script, connecting via CDP to the stealth browser instance exposed by Kameleo Docker, will contain:

# pip install playwright kameleo-local-api-client

from kameleo.local_api_client import KameleoLocalApiClient
from kameleo.local_api_client.models import CreateProfileRequest
from playwright.sync_api import sync_playwright

# Connect to the Kameleo API client
client = KameleoLocalApiClient(endpoint="http://localhost:5050")
# Search for a real-world fingerprint and create a Kameleo profile based on it
fps = client.fingerprint.search_fingerprints(
    device_type="desktop",
    os_family="windows",
    browser_product="chrome",
    browser_version=">145",
)
profile = client.profile.create_profile(
    CreateProfileRequest(fingerprint_id=fps[0].id, name="twsc demo")
)

# Connect Playwright to the Kameleo instance based on the configured profile
ws_endpoint = f'ws://localhost:5050/playwright/{profile.id}'
with sync_playwright() as playwright:
    browser = playwright.chromium.connect_over_cdp(endpoint_url=ws_endpoint, timeout=90_000)

    # Open a new page
    context = browser.contexts[0]
    page = context.new_page()

    # Visit target site
    page.goto("https://www.scrapingcourse.com/javascript-rendering")

    # Where to store the scraped data
    products = []

    # Wait for products to render
    page.wait_for_selector(".product-item")

    # Locate all product items
    product_elements = page.locator(".product-item")

    for i in range(product_elements.count()):
        # Select the nth product
        product_element = product_elements.nth(i)

        # Extract the product data
        name = product_element.locator(".product-name").inner_text()
        price = product_element.locator(".product-price").inner_text()
        image = product_element.locator("img.product-image").get_attribute("src")
        link = product_element.locator("a.product-link").get_attribute("href")

        # Populate a product object with the scraped data
        product = {
            "name": name,
            "price": price,
            "image": image,
            "url": link
        }
        # Append it to the products list
        products.append(product)

    # Print the scraped products
    for product in products:
        print(product)

# Stop the Kameleo profile
client.profile.stop_profile(profile_id=profile.id)

Execute the script, and you should see output similar to this:

Notice how the script successfully scraped product data from the JavaScript-rendered page.

Once execution completes, open the Kameleo web GUI, and you’ll notice that the “twsc demo” profile is now marked as “TERMINATED”:

That doesn’t mean the profile is gone forever. Quite the opposite!

Kameleo profiles are reusable, meaning you can retrieve and start them again later to continue the browsing session with the same fingerprint and persisted state. I’ll cover exactly how to do that in a dedicated FAQ.

Mission complete! You just learned how to use Kameleo Docker for Playwright automation. With very similar logic, you can automate Puppeteer, Selenium, or any other CDP-compatible solution, in both Python and JavaScript.

Pricing Model

Kameleo Docker is included across all plans at no additional cost. You get the same core limits (concurrent browsers, number of profiles, and browser usage time) regardless of whether you run the desktop app or the containerized version. So, take a look at the official pricing page for more information.

Anti-Bot Performance Benchmarks

To test Kameleo Docker, I ran a simple script against one page protected by each major anti-bot detection system. The results are shown below:

Note: All tests were performed locally using my ISP’s residential IP address.

As shown above, in this basic experiment, Kameleo Docker achieved a 100% success rate. In contrast, Playwright consistently failed in headless mode and, in some cases, also struggled in headful mode.

Need help with your scraping project?

Final Thoughts on Kameleo Docker

What stood out when I met Barnabas at Prague Crawl 2026 (see you next year 😉) was the clear passion the team has for the project, along with their focus on quality and continuous improvement.

At the same time, when testing new products, especially technical and complex ones like Kameleo Docker, you usually stumble across bugs or unexpected behavior. I can confidently say that this wasn’t the case at all here. Everything ran smoothly from the beginning, and I didn’t encounter any issues…

On top of that, the benchmark results are promising, and I didn’t notice any significant performance lag. Thus, my honest takeaway is simple: if you’re looking for a production-ready, containerized stealth browser, or you’re simply passionate about automation and scraping, consider giving Kameleo Docker a try!

In this article, I covered what the project is about, what it offers, how it works, and how to use it. As always, remember to use Kameleo Docker only for legal and ethical web scraping and automation. Until next time!

FAQ

Is Kameleo Docker different from the desktop app?

Kameleo Docker differs from the desktop app mainly in deployment. Instead of a local GUI, it runs as a containerized service that runs on both Linux and Windows servers. Feature parity is largely preserved, including profiles, fingerprinting, and browser engines.

Can I reuse already created profiles in Kameleo Docker?

Yes! Profiles can be reused by retrieving the full list of profiles, filtering by name (or ID, if you know it), and then starting the desired profile. For example, to reuse the “twsc demo” profile, write:

from kameleo.local_api_client import KameleoLocalApiClient
from kameleo.local_api_client.models import ProfileLifetimeState

# Connect to the Kameleo API client
client = KameleoLocalApiClient(endpoint="http://localhost:5050")

# Fetch all available profiles
profiles = client.profile.list_profiles()

# Find the profile with the specific name
target_name = "twsc demo"
profile = next((p for p in profiles if p.name == target_name), None)

# Check if the profile was found
if profile:
    # Start the existing profile if it isn't already running
    if profile.status.lifetime_state != ProfileLifetimeState.RUNNING:
      client.profile.start_profile(profile.id)

Can I use Kameleo Docker in my CI/CD?

Kameleo Docker fits naturally into CI/CD pipelines by running as a disposable, reproducible container in build or test stages. You can spin up browsers on demand, run automated flows, and tear them down after execution. Configuration is typically handled via Docker Compose.

Does Kameleo Docker support proxy integration?

Yes! HTTP, HTTPS, and SOCKS proxies can be configured at profile creation time, as explained in the documentation.

Can Kameleo Docker scale to thousands of browsers?

Kameleo Docker supports horizontal scaling through standard orchestration tools. You can run multiple containers across clusters using Kubernetes or AWS ECS, each managing independent browser instances.

How Does Firefox Automation Work in Kameleo Docker?

Kameleo Docker supports Firefox-based automation through the Junglefox engine. Because Playwright cannot connect directly to Firefox-based sessions, Kameleo provides a pw-bridge helper that acts as a compatibility layer. This component translates Playwright connections into the correct browser session, allowing standard automation scripts to run unchanged while still using Firefox-based fingerprint profiles.

Did you like this article? Share it with someone who might find it useful and get a discount on paid plans.

Share

Before proceeding, let me thank NetNut, the platinum partner of the month. Their set of solutions cover all your needs for scraping.

Visit Netnut

Shopee does not hand you the easy version. Capture a request, replay it, and the backend answers with HTTP 418 and a security error code. Every API call carries a set of anti-fraud headers, and the code that builds them is not in the Java you can read with JADX. It sits in native code, in compiled .so libraries, which is exactly where traffic interception and a Java decompiler stop being useful. Open the app in JADX and the signing method is there in name only, declared native, with its body on the far side in ARM machine code.

This is a two-part investigation into how Shopee signs its API requests and how you reproduce that signing yourself. The strategy is the one that works on most hardened apps. You locate the native security libraries, open them in a disassembler, and turn what they do back into something you control. When a library is readable crypto, you reimplement it in Python and sign offline, at any volume, with no app in the loop. When it is a bytecode virtual machine you cannot practically rewrite, you keep the app running and drive its own signer as an oracle. We chose this route because it is the one that scales. An offline signer, or an oracle you call, runs inside your scraper on a server. A rooted phone you have to babysit does not.

This first part is the foundation. We take one of Shopee’s native libraries, libshopeeaegis.so, reverse it end to end with Ghidra and rebuild it in Python. Reading the decompiled code identifies every operation as textbook crypto, and our rebuild reproduces those algorithms byte for byte. It is the readable case, the kind you win cleanly, and the clearest worked example of the method. The second part takes on the harder library, the one that computes the per-request signature, and gets past it with the oracle approach.

What you take from each part depends on your goal. If Shopee is your target, the payoff is the full picture of its request signing across both parts. If you scrape other apps, the method matters more than the marketplace. Most apps that protect their API at all push the work into a native library, and a large share of those are plain, readable crypto you can reproduce. We work it out on Aegis here, and it is the same move on the next app you open.

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

The tools

We use four tools, each doing one job.

androguard is a Python library for static APK analysis. We use it for fast recon. It lists the native libraries an app ships and finds which classes declare native methods. It does not give you readable source. It gives you structure you can script.

JADX decompiles Dalvik bytecode back to Java. It is how you read the managed side of the app and find the exact class and method that crosses into native code. JADX stops at the native keyword, which is the handoff point to the next tool.

Ghidra is the NSA’s open source reverse engineering framework. It disassembles a .so and decompiles it to pseudo C. It is the only tool here that can read native code, and it is the one this article leans on.

Frida injects a JavaScript engine into a running process so you can hook and call functions live. We use it to run the app under instrumentation and confirm our static reading against what the app actually does.

JADX and androguard read the managed code. Ghidra reads the native code. Frida watches the code run. The native library is the one piece only Ghidra can open, so the work centers there.

Modeling the app’s defenses

Before opening anything, it helps to name the layers, because Shopee has several and only one is our target here.

The managed layer is the Java and Kotlin code. It builds requests, attaches headers, and calls into native methods. JADX reads it.

The native layer is a set of .so libraries the app loads. Pull the arm64 split out of the APK with androguard and the security-relevant ones stand out by name. They are libshopeeaegis.so, libshpssdk.so, and libBkeBizSecurity.so, plus libjnihook.so and libshook.so. The last two are a hooking framework and an anti-hook layer, which means the app actively watches for instrumentation. That matters for Frida later.

The request-signing layer sits on top. Two okhttp interceptors, com.shopee.app.network.antifraud.b and .d (they call themselves SecurityNewSapInterceptor and SecurityNewSapPostInterceptor), attach the anti-fraud headers af-ac-enc-sz-token and x-sap-ri to API requests. The values they attach come from libshpssdk.so, the Shopee Security SDK.

We target one layer, libshopeeaegis.so, a general-purpose crypto library the app calls for specific operations. The request signer in libshpssdk.so stays out of scope. It is a bytecode virtual machine, a harder problem that we handle separately, and reproducing the af-ac headers is not the promise here. The promise is that you can take libshopeeaegis.so, understand every operation it performs, and reproduce it byte for byte in Python.

One detail decides whether that promise holds. libshopeeaegis.so loads only when the app needs it, so it is not present at idle. We watched the process maps over a minute of normal browsing and the library never appeared. The crypto we are about to reverse is a toolbox the app reaches for in certain flows, not the thing running on every request.

Start your scraping journey with Byteful: 10GB New Customer Trial | Use TWSC for 15% OFF | $1.75/GB Residential Data | ISP Proxies in 15+ Countries

Claim your 10GB here

Getting the library and finding the door

We pulled Shopee PH 3.75.24 (com.shopee.ph) as an XAPK and unzipped it. The native libraries are not in base.apk. For a split bundle they live in config.arm64_v8a.apk. Listing the .so files with androguard and unzip -l, libshopeeaegis.so is a small one at 280 KB, which is a good sign. Small means little room for a heavy obfuscator.

androguard answers the first question, which library to open. It does not answer the second, how the app calls it. For that we go to JADX and find the class on the Java side. The library registers its native methods against com.shopee.sz.reinforce.Aegis. The class exposes a method fire, overloaded, declared native. Two of the overloads matter:

native byte[] fire(int mode, byte[] data)
native byte[] fire(int mode, byte[] data, byte[] key)

This is the door. The first argument is an integer mode. Then one or two byte arrays. The return is a byte array. JADX cannot show what fire does, because the body is in the .so. So we open the .so.

Check the TWSC YouTube Channel

Reading the library with Ghidra

We ran Ghidra 12.1.2 in headless mode. It runs without a GUI, it scripts cleanly, and it repeats exactly. The workflow is documented in our Ghidra tool skill you can use in Claude Code, just like I did for this test. In short, you import the .so, let auto analysis run, then run a script that decompiles functions to a file.

support/analyzeHeadless /tmp/proj aegis \
  -import config.arm64_v8a/lib/arm64-v8a/libshopeeaegis.so \
  -scriptPath ./scripts \
  -postScript DecompileExport.java out.c \
  -overwrite

Auto analysis finished in nine seconds and the decompiler produced 674 functions with zero failures. That number alone tells you this is not a packed or virtualized binary. A protected library fights the decompiler; this one did not.

The first useful function is JNI_OnLoad, which every JNI library runs at load time. Read its pseudo C and it looks up the class com/shopee/sz/reinforce/Aegis and calls RegisterNatives with two methods. That confirms the door from the Java side and tells us the native functions are registered dynamically rather than exported under Java_* names. Dynamic registration is a mild form of hiding, and it is exactly what Ghidra’s JNI handling and a RegisterNatives trace are for.

The C++ symbols survived. That is the break that makes this library readable. The class is Aegis, with methods named missileFire, missileCount, prism, snowDon, tugWar, and parse. There is a second class, TeslaModel, with model_3, model_a, model_b, model_c, model_e, model_s, model_x, model_y, and getNuremberg. The names are deliberately silly, a Tesla and military theme, but they are real symbols, and the structure is intact.

Follow the call chain from the registered native function. The dispatcher is Aegis::prism, a plain switch on the mode integer:

switch(param_1) {
case 0:  model_3(...)               // one input
case 1:  model_x(key, input, ...)   // keyed
case 2:  model_x(...); model_3(...) // keyed, then case 0
case 3:  model_y(...); model_3(...)
case 4:  model_e(...)
case 5:  model_a(...)
case 6:  model_b(...)
case 7:  model_s(...); model_3(...)
case 8:  model_c(...); model_3(...)
}

One native call selects one of nine operations by an integer, and some operations are a keyed primitive followed by model_3. To name each operation we read two things, the output size and the primitive body.

The output size comes from TeslaModel::getNuremberg(mode, len), which missileFire calls to size the output buffer before doing the work. It returns 16 for mode 4, 32 for mode 5, 64 for mode 6, and 20 for mode 8. Those are the digest sizes of MD5, SHA-256, SHA-512, and SHA-1. For mode 0 it returns the Base64 expansion of the input length. The size function alone half-names the table.

The bodies confirm the rest, and here the silly names get helpful, because the renamed primitives kept their original suffixes. model3_autopilot is a textbook Base64 encoder. It reads three bytes, writes four, and pads with 0x3d, which is the = character. modelx_autopilot_cbc is AES in CBC mode, recognizable because it XORs each 16 byte block with the previous ciphertext block before the round function. The hash contexts are renamed with a phantom and F theme but keep the gnulib _init_ctx / _process_bytes / _finish_ctx shape. phantom1 is SHA-1, phantom256 is SHA-256, InitF22 is SHA-512. And phantom1 as called by model_c is the HMAC form. It XORs the key with 0x36 for the inner pad and 0x5c for the outer pad over a 64 byte block, which is the HMAC construction.

Two of the keyed modes turned out not to be ciphers at all. model_s calls phantom256 with a key and a message and returns 32 bytes, so it is HMAC-SHA256. model_c calls phantom1 the same way and returns 20 bytes, so it is HMAC-SHA1. Reading the bodies kept us honest here. From the signatures alone we had guessed AES.

That gives the full table.

model_x pads with PKCS7 to a 16 byte boundary, then runs AES-CBC. The key length sets the variant, and a 16 byte key gives AES-128. model_y does the same but writes the IV in front of the ciphertext, the standard prepend-the-IV pattern, before the Base64 in mode 3.

One value is not in the file. The CBC IV is a fixed 16 byte constant the library keeps at a .bss address. .bss is zero-initialized on disk and filled at runtime, so the IV is set when the library initializes and you cannot read it statically. For the hash, HMAC, and Base64 modes that does not matter, because their output is fully determined by the input and key. For the three AES modes it means byte-identical output needs the real IV, which you read from the live process once the library loads.

As always, the code that will be used for the python reimplementation we’re showing now can be found in our GitHub repository reserved for paying users, inside the folder 107.SHOPEE-GHIDRA.

Read more

Now, since the rise of AI and LLMs, every office job has changed, with no exceptions for scraping professionals. For us, this means that surviving anti-bots and CAPTCHAs has become the first challenge when creating scraping pipelines. The second challenge has become the integration of scraping services and capabilities into AI pipelines and AI agents. The reason for this shift is simple: your boss or your clients don’t just want the data anymore. They want it in real-time, structured, and often packed with some insights.

Until some months ago, these pipelines required a huge amount of custom code (and time!). Luckily for us, the major struggle lasted until MCPs were first released.

In this article, you’ll learn what MCP is, how the Decodo MCP server works, and how to integrate it with Claude Desktop. You’ll also learn how to use it with two hands-on examples.

Let’s get into it!

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

What is MCP?

The Model Context Protocol (MCP) is an open standard introduced by Anthropic that defines how AI models connect to external tools, data sources, and services. Before MCP, every integration between an AI model and an external system had to be built from scratch. MCP replaces all of the complexity behind the integration with a single, shared protocol.

Basically, the MCP protocol acts as a common language for models to connect with external tools, files, and systems. For example: do you need an AI assistant to pull a file from Google Drive, query a company database, and trigger an action in an internal app? That’s exactly the kind of job MCP is built to handle!

The practical upside is composability. Developers can mix and match several MCP servers into a single AI application without writing any integration code. So, after more than a year since its introduction in the AI industry, MCP has become the standard for integrating different services and applications into a single AI software.

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

What is Decodo MCP?

The Decodo MCP Server is a web scraping layer for AI agents. It connects MCP-compatible clients to Decodo’s Web Scraping API, enabling:

• Web scraping for LLMs and AI agents without managing infrastructure: It can scrape any website, including JavaScript-heavy pages, and get real-time data without handling proxy rotation, CAPTCHA solving, or anti-bot systems. It is specifically built for RAG pipelines, AI research agents, and automation flows.

• Structured outputs for LLM workflows: It allows you to retrieve the scraped data in Markdown (LLM-ready), JSON (for structured pipelines), and screenshots (for visual context).

As of now, the Decodo MCP server exposes the following tools:

• scrape_as_markdown: Scrapes any target URL, given a target URL via prompt. Returns results in Markdown.

• screenshot: Captures a screenshot of any webpage and returns it as a PNG image.

• google_search: Scrapes Google Search for a given query, and returns parsed results.

• google_ads: Scrapes Google Ads search results.

• google_lens: Scrapes Google Lens image search results.

• google_ai_mode: Scrapes Google AI Mode (Search with AI) results.

• google_travel_hotels: Scrapes Google Travel Hotels search results.

• amazon_search: Scrapes Amazon Search for a given query, and returns parsed results.

• amazon_product: Scrapes a given Amazon Product page.

• amazon_pricing: Scrapes Amazon Product pricing information.

• amazon_sellers: Scrapes Amazon Seller information.

• amazon_bestsellers: Scrapes Amazon Bestsellers list.

• walmart_search: Scrapes Walmart Search for a given query, and returns parsed results.

• walmart_product: Scrapes Walmart Product page.

• target_search: Scrapes Target Search for a given query, and returns parsed results.

• target_product: Scrapes Target Product page.

• tiktok_post: Scrapes a TikTok post URL.

• tiktok_shop_search: Scrapes TikTok Shop Search for a given query, and returns parsed results.

• tiktok_shop_product: Scrapes TikTok Shop Product page.

• tiktok_shop_url: Scrapes TikTok Shop page by URL.

• youtube_metadata: Scrapes YouTube video metadata.

• youtube_channel: Scrapes YouTube channel videos.

• youtube_subtitles: Scrapes YouTube video subtitles.

• youtube_search: Search YouTube videos.

• reddit_post: Scrapes a specific Reddit post.

• reddit_subreddit: Scrapes Reddit subreddit results.

• reddit_user: Scrapes a Reddit user profile and their posts and comments.

• bing_search: Scrapes Bing Search results.

• chatgpt: Search and interact with ChatGPT for AI-powered responses and conversations.

• perplexity: Search and interact with Perplexity for AI-powered responses and conversations.

NOTE: Decodo is currently onboarding its MCP server onto various platforms and marketplaces. At the moment of writing this article, it can be found on the Official MCP registry, Pulse MCP, Glama AI, mcp.so, and mcpmarket.com.

How To Integrate The Decodo MCP With Claude

The prerogative of MCPs is to integrate them with LLMs. The Decodo MCP server can be integrated with several services that leverage LLMs, like Claude and Cursor. In this paragraph, you will learn how to integrate it with Claude.

Requirements

To use the Decodo MCP, your system must satisfy the following requirements:

• Claude Desktop: To integrate the Decodo MCP server with Claude, you need Claude Desktop. If you don’t have it installed yet, you can download it from their website.

• Decodo account: Create an account at dashboard.decodo.com. With a free one, you have up to 2K free requests.

• Scraping token: Get the basic authentication token. To get it, click on Web Scraping API > API playground:

  

Good. Your system is set to connect Claude Desktop with the Decodo MCP server.

Connect The Decodo MCP Server to Claude

To connect the Decodo MCP server to Claude, open Claude Desktop and click on Settings:

Then, click on Developer:

After clicking on Edit Config, the system will automatically open a folder where Claude stores all the configuration files on your local machine. Open the claude_desktop_config.json file and add the following to it:

"mcpServers": {
    "Decodo MCP Server": {
      "command": "npx",
      "args": [
        "-y",
        "@decodo/mcp-server"
      ],
      "env": {
        "SCRAPER_API_TOKEN": "<your-decodo-mcp-api-key>"
      }
    }
  } 

Replace <your-decodo-mcp-api-key> with the basic authentication token you retrieved earlier from the Decodo dashboard, and the integration is done.

Quit Claude to make the changes effective. Note that just closing the Desktop window is not sufficient. You have to quit it:

After that, when returning to Settings > Developer, you’ll see the MCP server up and running:

To be sure everything works fine, you can test it with a prompt similar to the following:

Alright, you successfully integrated the Decodo MCP server with Claude Desktop. Now it's time to test it!

Check the TWSC YouTube Channel

Running the Decodo MCP With Claude: Hands-On Examples

In this section, you will learn how to use the Decodo MCP server with two different examples:

• A basic usage taken from the documentation.

• A more advanced usage where you’ll ask Claude to retrieve some data from Amazon, returning the result in a JSON file.

Let’s get on to this!

Getting Started: Run a Google Search in Seconds

As a first and simple usage, you can test the google_search tool. The main idea behind this tool is to prompt the model with a query, and, under the hood, the MCP will use its Google search capabilities to return the result.

To just try things out, you can use the exact example reported in the Decodo MCP documentation to search for shoes on Google, reporting the top positions:

As you can see, Claude automatically tries to use the MCP server, loading its tool. As Claude is a production-ready LLM, you will be asked if you want to allow it to use the Google search from the Decodo MCP always or just for this call.

When Claude has completed its job, the result is the following:

If you want to verify that the model hasn’t hallucinated, you can search for “shoes” on Google:

Alright! You made it to your first usage example.

Need help with your scraping project?

Leveling Up: Extracting JSON-structured Data From Amazon

Having results listed in a chat can be useful for a quick overview of web research. But the actual power of MCPs is to use the capabilities of the underlying tools for specific tasks that LLMs can solve.

An example is to ask the model to retrieve some data and report the results in a JSON file. This solution provides you with data that can be further used in the second part of your pipeline—for example, for analyzing your scraped data.

For this purpose, you can use the following prompt:

Get Amazon bestsellers in electronics, extract the main info and return it in JSON format

The image below shows Amazon’s best sellers in electronics, at the time of writing this article:

Under the hood, Claude will trigger the amazon_bestsellers tool and will search the data in electronics. The chat result is the following:

Below is the (partial) JSON returned by Claude:

{
  "metadata": {
    "category": "Electronics",
    "source": "Amazon Best Sellers",
    "url": "<https://www.amazon.com/Best-Sellers/zgbs/electronics/>",
    "scraped_at": "2026-05-18",
    "total_items": 50,
    "currency": "USD"
  },
  "bestsellers": [
    {
      "rank": 1,
      "asin": "B08JHCVHTY",
      "title": "Blink Plus Plan with Monthly Auto-Renewal",
      "price": 11.99,
      "rating": 4.4,
      "ratings_count": 275779,
      "image_url": "<https://images-na.ssl-images-amazon.com/images/I/31YHGbJsldL._AC_UL300_SR300,200_.png>",
      "url": "<https://www.amazon.com/Blink-Plus-Plan-monthly-auto-renewal/dp/B08JHCVHTY>"
    },
    {
      "rank": 2,
      "asin": "B0DCH8VDXF",
      "title": "Apple EarPods Headphones with USB-C Plug",
      "price": 19,
      "rating": 4.6,
      "ratings_count": 13500,
      "image_url": "<https://images-na.ssl-images-amazon.com/images/I/513OSdW4elL._AC_UL300_SR300,200_.jpg>",
      "url": "<https://www.amazon.com/Apple-EarPods-Headphones-Built-Control/dp/B0DCH8VDXF>"
    },
    
    <Omitted for brevity>

    }
  ]
}

As you can see, the first two items correspond to the first two in the image above, taken from the Amazon best seller page. This is just to be sure the model hasn’t hallucinated.

Well done! You learned how to use the Decodo MCP server with Claude.

Conclusion

In this article, you learned what MCP is and why it has become the standard for connecting AI models to external tools and services. You also learned what the Decodo MCP server is, how to integrate it with Claude Desktop, and how to use it in practice.

So, let us know: what kind of scraping workflows are you planning to build with the Decodo MCP?

Before proceeding, let me thank NetNut, the platinum partner of the month. Their set of solutions cover all your needs for scraping.

Visit Netnut

Part of that is the cat-and-mouse game every stealth tool plays. Part of it is specific to open source. When the entire fingerprint-spoofing codebase is public, the anti-bot vendors can read it line by line and build the exact counter-signal. We made that argument in the rayobrowse review. The openness that made Camoufox popular is the same openness that let the anti-bot giants study it and catch up.

Two things changed in 2026 that make this worth a fresh look. First, the project moved. The repository at github.com/daijro/camoufox now carries a note at the top of its README:

Clover Labs is a Toronto venture studio building AI agents, listed among the project sponsors. The alpha features (per-context fingerprints, hardware spoofing) now ship first in their cloverlabs-camoufox package, and daijro’s repo became the checkpoint mirror. This is not an abandoned project; the main maintainers changed.

Second, that public repo has more than 750 forks. Open source means that when one person stops, others can pick up the work, add features on features, and keep the chase going in parallel. So the real question is not only “is Camoufox still effective”, it is “has anyone in the fork tree built something better”. This is what we tried to discover in this article.

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

The forks we actually tested

We pulled the fork list from the GitHub API and sorted it by recent pushes. Most of it is noise. Many forks share the exact pushed_at timestamp of the parent, which is the signature of mirror bots that never wrote a line of their own. Once you count how many commits each fork is ahead of daijro:main and read what those commits do, the field collapses to a handful. Many of the survivors only touch CI or rebrand the binary. Three of them touch the anti-detect surface for real.

Official Camoufox (github.com/daijro/camoufox) is the baseline. A custom Firefox build with a fingerprint database and stealth patches, driven through Playwright’s Juggler protocol. We covered how it hides Playwright’s own traces in THE LAB #65, so we will not repeat that here.

camoufox-reverse (github.com/WhiteNightShadow/camoufox-reverse) goes the other way. Instead of hiding harder, it adds a PropertyTracer at the SpiderMonkey engine layer that records which DOM properties a page reads. It is an instrument for watching the detector work, not a better scraper. That makes it the most useful tool in the set for understanding what we are up against.

LeooNic/camoufox (github.com/LeooNic/camoufox) is the most ambitious on paper. Its commits add content-aware canvas noise that claims to defeat a 2025 academic pixel-recovery attack, a sigma-lognormal humanized mouse engine, and RDPBrowser, an automation path that drives Firefox over the Remote Debugging Protocol instead of Juggler.

JWriter20/camoufox (github.com/JWriter20/camoufox) is the pragmatic one. Targeted stealth fixes, the headline being a closed WebRTC IP leak under a proxy on Firefox 146 (daijro issue #538), plus a real pytest suite, which none of the others ship.

Let’s start by using camoufox-reverse to discover something more about DataDome installed on Leboncoin.fr.

What DataDome reads, watched from inside the engine

Before testing who passes, we wanted to see what the detector looks at. We have explained the three detection layers before: behavioral, browser, and HTTP, in THE LAB #6. camoufox-reverse lets us watch the browser layer from below the JavaScript, which is a view we have never had in these pages.

The PropertyTracer is documented to be enabled via a config flag. We drove the macOS arm64 build directly with Playwright, set the trace config through the CAMOU_CONFIG environment variable, and pointed it at a DataDome-protected page. Our target throughout this article is leboncoin.fr, the French classifieds site, because it runs only DataDome. That isolates the signal we care about, with no second anti-bot muddying the result.

The full probe is in code/camoufox_fork_analysis/trace_datadome.py. The core of it sets the trace and lets DataDome’s script run:

config = {
    "propertyTrace": {
        "enabled": True,
        "logDir": str(LOG_DIR),
        "objects": [],            # empty = trace all covered getters
        "maxEventsPerSession": 200000,
    }
}
env = os.environ.copy()
env["CAMOU_CONFIG"] = json.dumps(config)
env["MOZ_DISABLE_CONTENT_SANDBOX"] = "1"  # required on macOS for the tracer

When sites get tough, skip the heavy lifting. Get clean, structured CSV datasets, ready for Excel, BI or your apps

Find your dataset

The tracer writes one JSON line per getter access, each shaped like {"o": "navigator", "p": "hardwareConcurrency", ...}. Loading the leboncoin homepage produced 140 engine-level reads across 30 distinct properties. Aggregated by object and property, the access pattern contains information useful for fingerprint creation:

 COUNT  PROPERTY
    14  window.outerWidth
    13  window.devicePixelRatio
    13  window.outerHeight
    13  navigator.plugins.indexedGetter
     9  navigator.hardwareConcurrency
     7  canvas.toDataURL
     6  window.innerWidth
     6  screen.rect
     4  navigator.platform
     4  navigator.userAgent
     4  webgl.getParameter
     4  canvas2d.getImageData
     3  navigator.maxTouchPoints
     2  offscreenCanvas.getContext

Check the TWSC YouTube Channel

This is happening entirely below the JavaScript layer. From the page’s point of view, nothing was instrumented, because the recording lives in the C++ getter, not in a JavaScript proxy. DataDome reads the screen geometry, the navigator core, the plugin and mime enumeration, and then it reaches for the canvas and WebGL. Both canvas.toDataURL and canvas2d.getImageData are in the list, alongside webgl.getParameter and offscreenCanvas.getContext.

That last detail is what connects this experiment to the rest of the article. The canvas readback is exactly the surface LeooNic’s content-aware noise patch sets out to protect, and the WebRTC and screen reads are where the other forks claim improvements. We now know the detector touches it all.

The homepage is the light version. The pages that hold the data are watched far more closely, and the tracer shows it. We pointed the same probe at a car listing (a leboncoin /ad/voitures/ URL). Those pages block direct connections, so this run went through a residential proxy, which is the setup we explain in the next section. The listing loaded its real content (the page title came back as “Alfa romeo Tonale 1.5 Ibrida 175ch Veloce TCT”), so we were tracing a passing ad page, not a challenge screen. The read pattern is a different animal: 584 engine-level reads across 35 properties, against 140 across 30 on the homepage.

 COUNT  PROPERTY (ad page)
   220  document.cookie.get
    47  window.innerWidth
    30  window.innerHeight
    26  navigator.plugins.indexedGetter
    26  screen.rect
    25  sessionStorage.setItem
    22  sessionStorage.getItem
    16  document.cookie.set
    12  performance.timing
     8  window.scrollY
     7  canvas.toDataURL
     6  webgl.getParameter
     4  canvas2d.getImageData
     3  navigator.globalPrivacyControl
     1  mediaDevices.enumerateDevices

The cookie reads jump from one on the homepage to 220 on the ad page. Session storage, which the homepage barely touched, is read and written dozens of times. New surfaces appear that the homepage never queried: window.scrollY for behavior, navigator.globalPrivacyControl, and mediaDevices.enumerateDevices. The canvas and WebGL reads are still there. This is the same DataDome, running a heavier script on the page that matters. It is the concrete reason the homepage passes a clean browser while the listings do not. It also tells you where to spend your effort. The protection you have to beat lives on the content pages, not the landing page.

Setting up a fair comparison

The shared virtual environment we’re creating already had camoufox 0.4.11, which fetches the Firefox 135 official build. We ran on an Apple M2 Max, so we pulled the macOS arm64 binaries for each build, signed them ad hoc (the cross-compiled bundles need it), and pointed the same launcher at each one with executable_path.

Two version details matter for fairness. JWriter20’s WebRTC fix targets a regression introduced in Firefox 146, so we did not compare it against the 135 cache. We pulled the official v146-hardware build (Firefox 146.0.1) as the baseline and JWriter20’s own 146.0.1 build as the patched version. Same Firefox, two builds. camoufox-reverse only ships at 135, which is fine because we used it only as a tracer, not as a contender.

Every test drives the binaries the way a real user would, through the camoufox launcher with proxy and geoip set, so the fingerprint database, the locale coherence, and the stealth patches are all active. The one exception is the WebRTC probe, explained below, where the page we run matters.

The WebRTC leak that JWriter20 actually fixes

JWriter20’s headline fix is a closed WebRTC IP leak under a proxy. We checked it on the official 146 build against the JWriter20 146 build, same launcher, same Bright Data proxy, geoip=True. The probe gathers ICE candidates from a STUN server and reports any IP that escapes (webrtc_leak_test.py).

A quick detour on what those candidates are, because the whole leak lives in them. WebRTC connects two peers directly, and to do that each side has to advertise every network address it could be reached on. Each address it offers is an ICE candidate. A candidate is an IP, a port, a protocol, and a type, and it reaches JavaScript as a string like this:

candidate:842163049 1 udp 1677729535 203.0.113.25 54321 typ srflx raddr 192.168.1.45 rport 54321

Two types matter here. A host candidate is an address of a local network interface, so it carries your LAN IP. A srflx (server-reflexive) candidate is the public address a STUN server reports back when the browser asks which IP it appears to come from, so it carries your real WAN IP. A page gathers all of this with no permission. It opens an RTCPeerConnection pointed at a STUN server, calls setLocalDescription, and reads each candidate as it arrives. The key is that STUN runs over UDP, and an HTTP proxy only tunnels TCP. The STUN request leaves from the real interface, the proxy never sees it, and the srflx candidate comes back with the real WAN IP even though every HTTP request went through the proxy.

The first version of our probe ran the RTCPeerConnection on about:blank and showed both builds leaking the real IP. That was our mistake, not a result. Camoufox’s content-level injection is not active on about:blank, so we were measuring an unprotected page. Moving the probe onto a real https origin changed everything:

official-146   HTTP exit IP (proxy): 189.173.138.17
               ICE candidates: 1  [srflx] ips=['203.0.113.25']   <- real WAN IP leaks

jwriter20-146  HTTP exit IP (proxy): 93.44.185.102
               ICE candidates: 0                                <- nothing leaks

Our real WAN IP is 203.0.113.25. The official build, behind a working proxy, still hands it to any page through the WebRTC reflexive candidate. The proxy exit IP rotates on each run, so the constant 203.0.113.25 in the candidate is unmistakably the real address, not the proxy.

The fix is real, and it is baked into the binary.

We unzipped both camoufox.cfg files to confirm. The official build sets only media.peerconnection.ice.no_host. JWriter20 adds default_address_only, proxy_only_if_behind_proxy, proxy_only_if_pbmode, and obfuscate_host_addresses. Behind a proxy that cannot carry UDP, those preferences make WebRTC gather no candidates at all, so there is nothing to leak. Reproduced across two runs.

If WebRTC leaks were your problem, JWriter20 solves them. Hold that thought, because it does not end where you would expect.

The canvas patch that we could read but not run

To see why this patch exists, you have to understand the small-arms race it sits within. The PropertyTracer run above caught DataDome calling toDataURL and getImageData. Those two calls are how a canvas fingerprint is taken. A script draws the same text and shapes into an off-screen canvas on every machine, reads the pixels back, and hashes them. The drawing commands are identical everywhere. The pixels are not, because the final image depends on your GPU, your graphics driver, and how your system rasterizes fonts. That hash is stable for your device and different from the next one, which is most of what a tracker wants.

The standard way to hide is to add noise. Camoufox, Brave, Firefox’s resist-fingerprinting mode, and a long tail of extensions all nudge a few pixels so the hash will not stay constant across sites. The weakness is in how that noise is generated. If it is a fixed per-session perturbation that depends only on a seed and the pixel position, it can be undone. A 2025 paper at The Web Conference, Breaking the Shield by Hoang Dai Nguyen and Phani Vadrevu, showed exactly that against eighteen extensions and five browsers. Their Pixel-Recovery attack paints a second canvas filled with a known solid color and reads it back. Because it knows what every pixel should have been, it solves for the perturbation and subtracts it from the real fingerprint canvas. Reload ten times, and the recovered fingerprint stays constant while the noised one keeps changing. That is the proof the noise was reversible all along.

Two changes defeat the attack, and the same paper points at both. Leave the flat regions alone, so a detector that paints a solid block and reads it back finds no tampering to measure. And make each perturbation depend on the pixel content rather than its position, so there is no single value to solve for and subtract. The second idea is what Brave’s Farbling does, deriving its noise from the canvas content so two different canvases are altered differently, and it is the one defense the Pixel-Recovery attack could not reverse.

LeooNic’s patch implements both moves, and it is the most interesting code in the whole fork tree. The rewritten ApplyCanvasNoise skips flat regions and only perturbs edges, and the comments name the attack directly:

// Content-aware + content-dependent canvas noise.
//   - Tier 1 known-pixel checks (DataDome, Castle): flat regions are skipped
//     because flat_score < FLAT_THRESHOLD. fillRect(R,G,B) is undisturbed.
//   - WWW'25 Pixel-Recovery Attack (Nguyen & Vadrevu): noise depends on the
//     pixel content AND its 4 neighbors, not just (seed, index).

FLAT_THRESHOLD is the cutoff that decides what counts as a flat region. The edge pixels that survive it get a content-dependent nudge of plus or minus one, small enough to stay invisible but enough to move the hash. The logic is sound on paper. We wanted to confirm it at runtime.

We built a probe that draws two solid blocks with one sharp boundary and counts perturbed pixels in the flat interior versus the edge (canvas_fingerprint_test.py).

First, we learned that canvas noise is off by default in every current build, which lines up with the CloverLabs “Disable Canvas Noise” commit. The noise only runs when canvas:seed is non-zero. With the seed forced on the official 146 build, the original algorithm shows its tell:

official-146 (original noise)
  interior flat pixels perturbed : 9105 / 18240   (~50%)
  boundary edge pixels perturbed : 98 / 192
  max edge delta (per channel)   : 1
  hash varies across sessions    : True

The stock algorithm perturbs roughly half of every pixel, flat fills included. That is precisely the behavior a known-pixel check catches, and precisely what LeooNic set out to fix. So the official baseline is the “before” picture, captured at runtime.

The “after” picture is what we were not able to collect. LeooNic ships only a Windows binary, so we ran it on a Windows cloud box. It would not launch under Playwright at all. Every attempt, headless or headful, with the stock launcher or LeooNic’s own 0.5.0 launcher installed from source, ended the same way:

console.error: "Warning: unrecognized command line flag" "-juggler-pipe"
Remote Settings startup changesets bundle could not be extracted (JSON.parse...)
JavaScript error: AsyncShutdown.sys.mjs, line 587: uncaught exception: undefined
<process did exit: exitCode=0>

The official 135 build launches and drives fine on the same box, so the machine and Playwright are healthy. The Firefox 149 build that LeooNic publishes aborts at startup before Juggler attaches.

This is not only our environment. LeooNic’s own issue #1 is titled “fix: port patches and build system to Firefox 149.0”, an open work in progress, and daijro carries issues #620 and #572 about Juggler failing to initialize in constrained environments. The build does run through LeooNic’s native RDP path, which is the whole point of their RDPBrowser, but even there we could not activate the canvas seed. The global config is ignored, and the per-context setCanvasSeed function the build exposes only at document start was never present on the page when driven over RDP.

So we report LeooNic honestly. The content-aware algorithm is real and well-reasoned in source, and the original algorithm’s weakness is confirmed at runtime on the official build. The published Firefox 149 binary is not something you can pick up and drive with the standard stack today. For a reader choosing a fork, that is the practical signal. The innovation lives in the code, not yet in a usable artifact you can run.

The block-rate test, and the result we did not expect

Given that, we could only test the jwriter fork compared to the original version. As always, the code that will be used for testing can be found in our GitHub repository reserved for paying users, inside the folder 106.CAMOUFOX.

Read more

Every product has its own price, usually formed by a simple and predictable formula. Proxy services follow a similar pricing logic. At first thought, multiplying the proxy price by the amount of bandwidth should result in a final cost, but in fact, many aspects must be taken into consideration. Bandwidth is not a clean, one-to-one reflection of useful work. What users pay for is not just the data they properly collect but also everything that happens around. And it’s about request failures, encountered blocks, timeouts, and suboptimal routing decisions.

Much of the traffic in scraping systems is consumed without usable output, and that’s where the gap between expected and actual costs starts to widen. To understand where the budget really goes, we need to research it deeper.

What is the math behind the scraping costs

Bandwidth is measurable, and proxy providers typically price it transparently. However, it’s not the key driver of the cost, it’s the result of a much more complex process. Each unit of bandwidth stands for a series of events, rather than merely one successful request. While some requests yield immediate data, many others can fail, be obstructed, time out, or require further attempts. Consequently, the actual cost structure is influenced more by system behavior than by the volume of traffic.

The real cost equals the total number of request cycles necessary to extract data multiplied by the cost of executing each request cycle. In simple words, it is defined by how many complete attempts the system must make before it gets that one response. The key detail is that only successful requests generate value. Each request cycle may have the initial request, retries after timeouts, proxy rotation, session management, and many other factors. All of them consume resources. Because of this, the real price increases not only when proxy prices rise but also when the system becomes less well-functioning. The same dataset can cost remarkably more to extract when the scraping pipeline is inefficient.

The cost amplification effect, known as the cumulative increase in resource usage caused by constant request cycles, is an issue in web scraping systems. One failed request can trigger multiple follow-up attempts. These extra cycles accumulate and raise the cost required to get one successful data point. For this reason, it’s accurate to evaluate scraping by cost per successful request. The fewer attempts needed, the lower the real cost.

Mistake #1 - Using the wrong proxy type

This mistake is not just widespread but can also cost you a lot. Not all targets need the same level of stability and anonymity, but numerous systems use a universal strategy. This results in either high costs or poor operation.

Each proxy type has its own balance of speed, cost, and detection resistance. The system will not work if that balance doesn’t align with the target website’s behavior. For example, mobile IPs are not just more expensive by default. They are highly trusted and harder to get blocked, so it’s logical to use them for challenging targets. When they are used on low-protection websites, they increase costs without improving the results. The approach that works is based on the right matching of proxy type to the task.

• Residential proxies are actively used in web scraping as they route traffic via IPs assigned by ISPs to real devices. By looking like it comes from the real user, these proxies ensure strong trust signals. Many proxy users notice better success rates when they switch to residential IPs.

• Mobile proxies direct traffic via carrier networks using IP addresses from mobile service providers. Since these IPs are shared, the traffic appears very authentic and is significantly more difficult for systems that rely on fingerprinting to identify.

• Datacenter proxies function on cloud or server-based infrastructure and use IP ranges that aren’t tied to real ISPs. The biggest advantage is speed. They are perfect for heavy automation and data collection tasks.

Mistake #2 - The retry loop problem

The goal of retry logic is to improve success rates by giving failed requests another attempt to return a valid request. This approach works when responses are consistent, and failures are occasional, not systematic. On targets with rate limits or unstable responses, many retries can lead to constant failures under the same conditions. Not all failures are the same. If you got a timeout, it’s worth retrying, but if it’s a 403 error or a block, there are other actions to try. For example, you can rotate proxies or fix headers.

Retries can turn into a loop where the system keeps sending more requests but isn’t getting better results. Instead of retrying everything the same way, treat different errors differently, and adjust your behaviour based on the response. You can rotate proxies after getting a certain status code or stop retrying when a request is blocked.

Mistake #3 - Misconfigured proxy rotation

Rotating proxies aggressively is not a solution. Changing IPs too often makes traffic look unnatural and can raise suspicion. On the flip side, not rotating enough can create another issue. Thus, there must be balance. Some websites tolerate frequent IP changes, while others expect a more stable session. Treating all targets the same way is not appropriate. It’s better to adjust rotation based on the context.

If the server expects the same user behavior over time, using sticky sessions may help. In this case, it’ll help maintain session consistency and not break the flow. For bulk data extraction, you can try rotating proxies more frequently. In this situation, there is no need to preserve session context. You can also refine your rotation if you use signal-based triggers instead of fixed rules. Rotate thoughtfully, and when the system detects specific conditions like a sudden drop in success rates or status codes, adapt your proxy to it.

Mistake #4 - Ignoring caching and duplicate requests

A notable portion of scraping traffic is often dedicated to retrieving data that has already been collected. This occurs when pipelines lack deduplication or clear definitions for data freshness. It leads to repeated requests for identical resources. This process consumes bandwidth and proxy capacity without providing new information.

To address this, implement a caching layer and deduplication logic. Responses can be cached based on a time-to-live (TTL) interval that aligns with the frequency of data updates. Request fingerprints can be used to identify duplicates before requests are sent. For structured data, storing IDs or hashes of processed items allows the system to skip previously captured content.

Mistake #5 - No cost-aware proxy routing

Many scraping systems process all requests through a single proxy type. This approach can simplify implementation, but it can still lead to ineffectiveness. Different endpoints often have distinct requirements, and a universal strategy may result in unnecessary costs.

For instance, using proxies with a high trust score for simple endpoints can be expensive, whereas lower-cost proxies used for protected pages may result in blocks and required retries. Without routing logic to adapt to these variables, systems often overpay or underperform. They can’t adapt.

An alternative is to implement cost-aware routing, which matches the proxy type to the difficulty of the task. This involves using more economical options for low-risk requests and escalating to higher-trust proxies only when necessary. By monitoring metrics such as status codes, latency, and success rates, the system can determine when to switch proxy pools. For example, a blocked request can be retried using a higher-trust proxy rather than repeating the request under the same conditions.

This approach creates a more structured pipeline that balances cost and performance by allocating resources based on the specific requirements of each request.

Understanding the real price of proxies

While “price per GB” is often cited as a standard industry metric, experienced engineers understand that it fails to capture the true economic reality of data scraping. In practice, failed requests consume bandwidth and incur costs despite yielding no usable data. These unsuccessful attempts represent a negative return on investment.

Furthermore, the expenses associated with automated retries add another hidden expense. We have to look beyond the basic per-GB rate and adopt the “Cost Per Successful Request” (CPSR). This metric provides a more accurate reflection of true operational expenses.

To calculate the cost of each valid data retrieval, use the following formula:

CPSR = price per GB / 1,000 * 1 / Success Rate

In this equation, the “success rate” is the percentage of requests that return an HTTP 200 OK status along with the intended data. Organizations can make better financial decisions if they start evaluating proxy services through the lens of CPSR.

DataImpulse is a reliable provider of residential, mobile, and datacenter proxies with non-expiring traffic and a pay-as-you-go model, meaning purchased traffic remains available until it is used. This vendor offers more than 90 million IPs in 195 countries. Teams usually choose DataImpulse for web scraping, ad verification, market research, SERP monitoring, and website testing.

Why is DataImpulse cheaper than other vendors?

The pricing structure is based on the proxy sourcing method. Many providers purchase traffic rights from ISPs and resell them, which includes an additional markup. DataImpulse sources IP addresses directly through its own application and SDKs, bypassing intermediaries to avoid extra costs. This operational model complies with all legal standards.

How to reduce your scraping costs by 30-60%

Cost efficiency in data collection is primarily achieved by minimizing inefficient requests and increasing the success rate of each attempt.

1. To optimize expenses, match proxy types to the specific requirements of the task. Using cost-effective proxies for straightforward targets while reserving higher-trust proxies for more challenging endpoints can reduce unnecessary spending.

2. Refining retry logic is also important. Failures should be addressed based on their specific status codes. Avoiding repeat requests under identical conditions prevents the waste of resources.

3. Proxy rotation should be managed strategically rather than randomly. Implementing sticky sessions and rotating based on indicators such as blocks or elevated failure rates can improve both stability and overall success rates.

4. Incorporating caching and deduplication techniques helps manage traffic. By avoiding redundant requests for data that has not changed, it is possible to decrease total request volume.

5. Implement a cost-aware proxy routing strategy. Prioritize lower-cost alternatives, escalating to premium options only when strictly necessary. This approach facilitates a more efficient resource allocation model, ensuring that infrastructure investments are directed toward the areas of greatest impact.

6. Lastly, pay attention to how your scraper interacts with websites. When a browser loads a page, it also pulls images, scripts, videos, and even fonts. Thus, lots of traffic is generated. Use HTTP requests for structured data and browser-based scraping when JS rendering is necessary.

These optimizations don’t require a comprehensive system overhaul. Incremental improvements in request efficiency can harvest significant cost reductions.

Start measuring your current CPSR baseline

At first sight, scraping costs look like a simple equation between proxy price and bandwidth. But the real drivers of cost lie deeper. As we’ve seen, unnecessary retries and poor rotation strategies contribute to a growing gap between expected and actual costs. A system with low success rates will always consume more resources.

The important shift is moving away from thinking in terms of raw pricing and toward thinking in terms of efficiency. It doesn’t always require major steps, simple adjustments are key. Better proxy selection, cost-aware routing, caching, and improved retry logic are among them. From factual proxy usage data from DataImpulse, we’ve seen that even small optimizations can noticeably reduce total costs. Every request should add value, so spending must be thoughtful and deliberate. Audit your scraping pipeline against these 5 mistakes today.

Rust is becoming increasingly popular thanks to its advantages in performance, stability, and security. But what about using it for web scraping?

In this post, I’ll show you what Rust brings to the table for web scraping, why it makes sense (and when it doesn’t), and how to build a web scraper in Rust.

Main Characteristics of Rust: Quick Overview

Rust stands out because it combines performance, safety, and control in a way few programming languages do. According to the 2025 Stack Overflow Developer Survey, 14.8% of respondents reported using Rust that year, making it the 14th most popular option.

Personally, what I find most compelling about Rust is its memory safety model. Thanks to ownership and borrowing, it avoids entire classes of bugs like memory leaks or race conditions. All of that, without needing a garbage collector!

Before proceeding, let me thank NetNut, the platinum partner of the month. Their set of solutions cover all your needs for scraping.

Visit Netnut

---

Here’s what Rust looks like in its simplest form:

fn main() {
    println!("Hello, world!");
}

Even in this minimal example, you can see Rust’s explicit structure and compile-time guarantees.

Remember: In Rust, println! is not a function. It’s a macro. The ! tells the compiler: “this is a macro invocation, not a normal function call.”

Performance is another big win. Rust is compiled and extremely fast, making it ideal for high-throughput parsing on heavy HTML pages or large volumes of pages (e.g., in an offline web scraping scenario). Concurrency is also first-class, helping you manage thousands of requests in parallel without the usual headaches.

On the flip side, Rust has a steeper learning curve. If you’re coming from Python or JavaScript, getting used to the syntax and strict compiler won’t be trivial. In my experience, the first steps can feel a bit unforgiving…

Why AI Has Made Rust a Solid Choice for Web Scraping

AI is changing the nature of software development, including web development. And, as you may already have noticed, not always in a “lighter” direction. Humans struggle to deal with long scripts and source code files, but machines don’t!

Thus, AI tends to produce very long and complex HTML with a lot of elements embedded in the same page. On top of that, AI makes it trivial to generate large amounts of content, which further increases HTML size. In addition, semantic HTML is more verbose than traditional HTML.

As a result, modern web pages are getting bigger and more complex. From a scraping perspective, this translates into slower and more resource-intensive parsing. What used to be lightweight DOM trees are now dense, deeply nested structures that require more CPU and memory to process.

This is exactly where Rust starts to make sense…

Sure, it may not be the easiest programming language, but Rust’s performance makes it compelling (and in some cases even necessary). Its low-level control and zero-cost abstractions allow Rust HTML parsers to process large documents in fractions of a second, even under high concurrency.

Independent benchmarks show Rust HTML parsers ranking among the fastest available. In particular, libraries like tl stand out for their exceptional speed and low overhead.

Start your scraping journey with Byteful: 10GB New Customer Trial | Use TWSC for 15% OFF | $1.75/GB Residential Data | ISP Proxies in 15+ Countries

Claim your 10GB here

Best Rust Web Scraping Libraries

How to Build a Scraper in Rust: A Step-by-Step Guide

In this section, I’ll guide you through the process of building a web scraper in Rust. The target web page will be Books to Scrape’s homepage. This is a static page, which is the ideal scenario for high-speed HTML parsing in Rust.

The end goal is to scrape all the book information and export it to a CSV file. Follow the instructions below!

Prerequisites

Make sure you have:

• Rust installed locally (the article refers to Rust 1.95.0).

• Some basic familiarity with Rust syntax and constructs.

Trusted by teams running ad verification, web scraping, SERP tracking, and market research. Ethically sourced proxies, globally accessible, and fairly priced.

Get Started With DataImpulse

Step #1: Set Up a Rust Scraping Project

Create a new Rust project for web scraping with:

cargo new books_rust_scraper

This will generate a new project called books_rust_scraper containing a basic “Hello, world!” program. Move into the project folder:

cd books_rust_scraper

You should now see the following file structure:

books_rust_scraper/
├── src/
│   └── main.rs
├── target/
├── .gitignore
├── Cargo.toml
└── Cargo.lock

Focus on the src/main.rs file:

This is the entry point of your application and currently contains a simple “Hello, world!” example. Test your Rust application with:

cargo run

The command executes the src/main.rs file, so the result will be:

Hello, world!

In that file, you’ll implement your Rust web scraping logic. Great!

Step #2: Install Required Dependencies

Run these commands to install the crates (Rust libraries) needed to build a Rust web scraper:

cargo add tokio --features full
cargo add reqwest
cargo add scraper
cargo add csv

These are the core dependencies:

• tokio: Enables asynchronous execution.

• reqwest: To send HTTP requests to retrieve HTML pages.

• scraper: To parse HTML and extract data using CSS selectors.

• csv: To export the scraped data to a CSV file.

After running the commands above, your Cargo.toml file should look similar to this:

[package]
name = "books_rust_scraper"
version = "0.1.0"
edition = "2024"

[dependencies]
csv = "1.4.0"
reqwest = "0.13.3"
scraper = "0.26.0"
tl = "0.7.8"
tokio = { version = "1.52.1", features = ["full"] }

Nice! You now have all the dependencies in place to start building your Rust scraper.

Step #3: Retrieve the Target Page

Use reqwest to fetch the target page with:

use std::error::Error;
use reqwest::Client;

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
    // Initialize the HTTP client
    let client = Client::builder()
        .build()?;

    // Retrieve the target page
    let url = "https://books.toscrape.com/";
    let response = client
    .get(url)
    .send()
    .await?;

    // Extract the HTML content from the response
    let html = response.text().await?;
  
    // Parsing logic...

    // Data export logic...

    Ok(())
}

This snippet initializes an asynchronous HTTP client using Tokio, sends a GET request to the target URL, retrieves the HTML response body, and prepares it for parsing and data extraction.

If you print html, you’ll observe:

Excellent! Get ready to apply the Rust data parsing logic.

Step #4: Implement the Parsing Logic

Before implementing the web scraping logic in Rust, study the DOM of the target page. Inspect a book HTML element in the browser:

From this structure, notice how you can select all books using the article.product_pod CSS selector. For each book element, you can retrieve:

• The title and URL from h3 a.

• The image URL from .image_container img.

• The price from .price_color.

• The rating from p.star-rating.

• The stock status from .instock.availability.

First, define a struct to store that data:

#[derive(Debug)]
struct Book {
    url: String,
    image_url: String,
    title: String,
    price: String,
    rating: String,
    in_stock: bool,
}

Next, define the parse_books() function that extracts and structures the data:

use scraper::{Html, Selector};
// ...

// ...
fn parse_books(html: &str) -> Result<Vec<Book>, Box<dyn Error>> {
    // Parse the HTML content
    let document = Html::parse_document(html);

    // Define CSS selectors for the HTML elements of interest
    let book_selector = Selector::parse("article.product_pod")?;
    let title_selector = Selector::parse("h3 a")?;
    let image_selector = Selector::parse(".image_container img")?;
    let price_selector = Selector::parse(".price_color")?;
    let rating_selector = Selector::parse("p.star-rating")?;
    let stock_selector = Selector::parse(".instock.availability")?;

    // Where to store the scraped data
    let mut books = Vec::new();

    // Iterate over each book element and extract the relevant data
    for book_el in document.select(&book_selector) {
        // Apply the parsing logic
        let title_el = book_el.select(&title_selector).next().unwrap();

        let relative_url = title_el.value().attr("href").unwrap_or("");
        let url = format!(
            "https://books.toscrape.com/catalogue/{}",
            relative_url
        );

        let image_url = book_el
            .select(&image_selector)
            .next()
            .and_then(|img| img.value().attr("src"))
            .unwrap_or("")
            .to_string();

        let image_url = format!(
            "https://books.toscrape.com/{}",
            image_url.trim_start_matches('/')
        );

        let title = title_el
            .value()
            .attr("title")
            .unwrap_or("")
            .to_string();

        let price = book_el
            .select(&price_selector)
            .next()
            .map(|e| e.text().collect::<String>())
            .unwrap_or_default();

        let rating = book_el
            .select(&rating_selector)
            .next()
            .and_then(|e| e.value().attr("class"))
            .unwrap_or("no rating")
            .replace("star-rating", "")
            .trim()
            .to_lowercase();

        let in_stock = book_el
            .select(&stock_selector)
            .next()
            .map(|e| {
                let text = e.text().collect::<String>();
                text.to_lowercase() == "in stock"
            })
            .unwrap_or(false);

        // Collect the scraped book data
        books.push(Book {
            title,
            price,
            rating,
            in_stock,
            image_url,
            url,
        });
    }

    Ok(books)
}

This function parses raw HTML into structured data using the scraper crate. Html::parse_document() creates a DOM-like representation of the page, while Selector::parse() defines CSS selectors for targeting elements.

document.select(&book_selector) iterates over each book container. Inside each element, .select() extracts nested elements, while .value().attr() retrieves attributes such as links and titles. The .text() method collects visible text content.

Finally, all extracted values are assembled into a Book struct, and each instance is stored in a vector for later export or processing.

Check the TWSC YouTube Channel

Step #5: Export the Scraped Data

Right now, the scraped data is returned by the parse_books() function as a vector of Book structs. Next, add a function that uses the csv crate to export that data into a CSV file:

use csv::Writer;
// ...

//...
fn write_csv(books: &[Book], file_path: &str) -> Result<(), Box<dyn std::error::Error>> {
    let mut wtr = Writer::from_path(file_path)?;

    // Write the header row
    wtr.write_record(&[
        "url",
        "image_url",
        "title",
        "price",
        "rating",
        "in_stock",
    ])?;

    for book in books {
        wtr.write_record(&[
            &book.url,
            &book.image_url,
            &book.title,
            &book.price,
            &book.rating,
            &book.in_stock.to_string(),
        ])?;
    }

    wtr.flush()?;
    Ok(())
}

Step #6: Put It All Together

This is the final code of your Rust web scraper:

// src/main.rs

use std::error::Error;
use reqwest::Client;
use scraper::{Html, Selector};
use csv::Writer;

#[derive(Debug)]
struct Book {
    url: String,
    image_url: String,
    title: String,
    price: String,
    rating: String,
    in_stock: bool,
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
    // Initialize the HTTP client
    let client = Client::builder()
        .build()?;

    // Retrieve the target page
    let url = "https://books.toscrape.com/";
    let response = client
    .get(url)
    .send()
    .await?;

    // Extract the HTML content from the response
    let html = response.text().await?;

    // Parse the books data from the HTML
    let books = parse_books(&html)?;

    // Export the scraped data to a CSV file
    write_csv(&books, "books.csv")?;

    Ok(())
}

fn parse_books(html: &str) -> Result<Vec<Book>, Box<dyn Error>> {
    // Parse the HTML content
    let document = Html::parse_document(html);

    // Define CSS selectors for the HTML elements of interest
    let book_selector = Selector::parse("article.product_pod")?;
    let title_selector = Selector::parse("h3 a")?;
    let image_selector = Selector::parse(".image_container img")?;
    let price_selector = Selector::parse(".price_color")?;
    let rating_selector = Selector::parse("p.star-rating")?;
    let stock_selector = Selector::parse(".instock.availability")?;

    // Where to store the scraped data
    let mut books = Vec::new();

    // Iterate over each book element and extract the relevant data
    for book_el in document.select(&book_selector) {
        // Apply the parsing logic
        let title_el = book_el.select(&title_selector).next().unwrap();

        let relative_url = title_el.value().attr("href").unwrap_or("");
        let url = format!(
            "https://books.toscrape.com/catalogue/{}",
            relative_url
        );

        let image_url = book_el
            .select(&image_selector)
            .next()
            .and_then(|img| img.value().attr("src"))
            .unwrap_or("")
            .to_string();

        let image_url = format!(
            "https://books.toscrape.com/{}",
            image_url.trim_start_matches('/')
        );

        let title = title_el
            .value()
            .attr("title")
            .unwrap_or("")
            .to_string();

        let price = book_el
            .select(&price_selector)
            .next()
            .map(|e| e.text().collect::<String>())
            .unwrap_or_default();

        let rating = book_el
            .select(&rating_selector)
            .next()
            .and_then(|e| e.value().attr("class"))
            .unwrap_or("no rating")
            .replace("star-rating", "")
            .trim()
            .to_lowercase();

        let in_stock = book_el
            .select(&stock_selector)
            .next()
            .map(|e| {
                let text = e.text().collect::<String>();
                text.to_lowercase() == "in stock"
            })
            .unwrap_or(false);

        // Collect the scraped book data
        books.push(Book {
            title,
            price,
            rating,
            in_stock,
            image_url,
            url,
        });
    }

    Ok(books)
}

fn write_csv(books: &[Book], file_path: &str) -> Result<(), Box<dyn std::error::Error>> {
    let mut wtr = Writer::from_path(file_path)?;

    // Write the header row
    wtr.write_record(&[
        "url",
        "image_url",
        "title",
        "price",
        "rating",
        "in_stock",
    ])?;

    for book in books {
        wtr.write_record(&[
            &book.url,
            &book.image_url,
            &book.title,
            &book.price,
            &book.rating,
            &book.in_stock.to_string(),
        ])?;
    }

    wtr.flush()?;
    Ok(())
}

Note how all previously defined functions are now called inside main(). Et voila! In just around 150 lines of code, you’ve built an efficient web scraper in Rust.

Run your scraper with:

cargo run

After execution, a books.csv file will be created in your project folder. Open it, and you will see:

This matches exactly the data shown on the target website, but now in a structured format. Mission complete!

Browser Automation in Rust: Does It Make Sense?

First of all, it’s worth noting that the ecosystem for browser automation in Rust is quite small compared to JavaScript or Python. Also, most libraries aren’t official, but rather community-backed ports like Playwright Rust or the Selenium bindings.

Now, from a technical standpoint, browser automation happens inside the browser itself. So, Chrome, Chromium, or Firefox do most of the heavy lifting. What you define through the library’s API simply orchestrates operations like clicking, waiting for elements, and extracting data. These commands are then translated into browser actions via WebDriver, CDP, or WebDriver BiDi.

Because of that, using a systems-level language like Rust can be more of a burden than an advantage. The main strength of Rust (i.e., raw performance) doesn’t really matter here, since the controlled browser instances are the actual bottleneck, not your automation code.

That means we lose Rust’s biggest advantage while still paying its costs. On top of that, Rust’s strict compiler and steeper learning curve can slow down development speed.

To be honest, I see Rust as excellent for the parsing and data processing layer, but I wouldn’t recommend it for browser automation…

Need help with your scraping project?

Rust for Web Scraping: Final Comment

If I had to summarize my experience with Rust for web scraping, I’d say this: it really shines when you’re parsing large HTML pages at scale or handling a high number of parsing tasks in parallel.

In those scenarios, the combination of performance, memory safety, and concurrency makes a real difference. That said, I wouldn’t recommend Rust for everyday scraping tasks… The entry barrier is just too high, the learning curve too steep, and the ecosystem around scraping too small.

On top of that, finding experienced Rust developers specifically focused on web scraping, or even just translating those skills into job opportunities, can be way more challenging than in more mainstream stacks.

So my take is pretty simple: consider Rust when performance and scale truly matter. For everything else, prefer Python or JavaScript.

👍 Pros:

• Can efficiently handle thousands of requests in parallel.

• Rust HTML parsers are extremely.

• Strict compiler checks and static guarantees lead to stable scraping pipelines.

👎 Cons:

• Slower development and prototyping speed.

• Smaller ecosystem of scraping libraries compared to Python or JavaScript.

• Not a practical choice for browser automation.

Conclusion

Here, I’ve guided you through the world of web scraping in Rust. In a world dominated by AI slops, security flaws, and neglected best practices, this programming language is gaining traction thanks to its focus on efficiency and strict compilation.

As you’ve seen, Rust is excellent for CPU-intensive or memory-intensive tasks like HTML parsing and data processing. Still, it might not be ideal for browser automation or quick prototyping. You also learned how to go from zero to scraped data in CSV format by building a Rust scraper.

I hope you found this helpful and insightful. If you have any questions, feel free to share them in the comments below!

Before proceeding, let me thank NetNut, the platinum partner of the month. Their set of solutions cover all your needs for scraping.

Visit Netnut

What the 2023 LLM hype promised, and what 2026 actually shipped

The product category that took that promise most literally is the agentic browser. A real Chromium running under an LLM that reads the page, decides what to do, and clicks. Browserbase, Hyperbrowser, Skyvern, Browser Use, BrowserOS, Owl Browser, and a long tail of proxy companies rebranding their scraping browsers as “AI-powered”. The pitch in 2024 and 2025 never changed. You hand the agent a task in plain language, and the model handles the rest, including whatever defensive challenge the site throws back.

It is 2026 now, and that prediction has not played out. CAPTCHAs are still in your pipeline. So we went looking for the answer. Can an LLM actually solve a production-grade CAPTCHA like hCaptcha? We read the code where we could, checked the default configs and the docs, and probed the public surfaces of the solver services these products lean on. The picture is consistent, and it is not the one the marketing sells.

Every major agentic browser ships a CAPTCHA-solving bullet point on its site. Open the code of the open-source agents, though, and you find a different story. Almost none of them actually use an LLM to solve a CAPTCHA. They either refuse to try, or they try and fail.

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

One thing to get straight before we open any code. The two families of CAPTCHA behave very differently, and the hype lands on them unevenly. The invisible ones, reCAPTCHA v3 and Cloudflare Turnstile, score your session in the background and rarely show a puzzle. A stealth-first browser on clean residential proxies usually walks past them without anyone seeing a challenge, which we covered for Turnstile in Cloudflare Turnstile: what is that and how it works? and THE LAB #73: How to Bypass Cloudflare in 2025. The visible image challenges, hCaptcha and reCAPTCHA v2, actually demand an answer. We went deep on reCAPTCHA v2 in Bypassing reCAPTCHAs With Open Source and Commercial Tools - Part 2. The one that matters in 2026 is hCaptcha, and that is where this article lives, because it is where most scrapers break.

So here is the question that drives everything below. When an agentic browser says it “solves” an hCaptcha, what does its code actually do?

Tool landscape

What’s the commercial offer today

There are four families of strategy in the commercial set. In none of them is the LLM in the agent layer doing the CAPTCHA work.

The first family is stealth-first. Residential proxies, fingerprint shaping, and request patterning lower the bot score so the visible challenge never triggers. The CAPTCHA is not solved. It is prevented from appearing. That gives you the cleanest legal posture in the set, because no automated solving is happening. ZenRows is the example.

The second family is the opposite of stealth. Instead of hiding that the traffic is automated, the vendor declares it openly and relies on a business arrangement with the CAPTCHA providers to be let through. Browserbase is explicit about this. Its Stealth Mode documentation says “through Browserbase’s partnerships with CAPTCHA providers, Browserbase can resolve challenges automatically so your sessions continue without interruption”, with solving “enabled by default for all sessions”. This is the verified-bot path, the same idea behind Cloudflare’s Web Bot Auth. A declared, allowlisted agent rather than a disguised one. No model solves a puzzle on either side. The provider recognizes the partner and waves it through. For the common challenge types, according to their documentation, this works without a fight.

The third family pairs a proprietary solver with documented third-party integrations. The vendor ships its own solving for some challenge types. For the rest, it documents how to wire in an external solver service. The external solver watches for the challenge and returns the response token through its extension or REST API. The agent then submits. Hyperbrowser and Skyvern Cloud both present this shape, a native or closed-source component plus a documented third-party path. Hyperbrowser advertises “Native Cloudflare Turnstile & CAPTCHA Solving” with “No external plugins” in its post on native CAPTCHA solving, then points to an external solver for the challenge types the native one does not cover.

Start your scraping journey with Byteful: 10GB New Customer Trial | Use TWSC for 15% OFF | $1.75/GB Residential Data | ISP Proxies in 15+ Countries

Claim your 10GB here

The fourth family runs an in-house solver alongside the agent. Could be a vision LLM, could be classical computer vision, could be something else. No third party is in the critical path. The vendor owns the whole stack. Bright Data, Oxylabs, and Owl Browser sit here. Owl Browser puts hard numbers on the claim: “detect and automatically solve reCAPTCHA v2 (1.2s), hCaptcha (0.8s), Turnstile (0.3s), and image CAPTCHAs.” Bright Data sells “AI-based unlocking logic” that handles “CAPTCHA solving, fingerprinting, retries, best headers, location and more” on its Web Unlocker page.

The table below maps each vendor to the mechanism its public documentation surfaces, with the source page that proves it.

Check the TWSC YouTube Channel

The open-source agents we can read

For the proprietary CAPTCHA-solving strategies we cannot see how AI is used. For the open-source ones we can. So we opened their code and read exactly how they handle CAPTCHAs. Three projects matter here.

browser-use is the most popular open-source LLM agent framework for browser automation. MIT-licensed, vendor-agnostic on the LLM side. The repo contains no LLM-driven CAPTCHA-solving logic. The one CAPTCHA-related file it ships, captcha_watchdog.py, does not solve anything either. It waits for a solver running in the BrowserUse cloud proxy and blocks the agent loop until that solver reports back. Run the library locally with your own model and no cloud proxy, and the watchdog has nothing to wait for. That makes browser-use the cleanest test of the claim that a local LLM agent solves the CAPTCHA by reading the page.

Skyvern OSS is the open core of the Skyvern Cloud product. AGPL-3.0, focused on form-filling and structured workflows, written in Python.

BrowserOS is a YC S24 open-source agent-driven browser. AGPL-3.0, 11k stars on GitHub, active development. It pairs a Chromium fork with an integrated agent runtime.

Modeling hCaptcha and reading what the open-source agents actually do

Before testing, it helps to model the target. hCaptcha embeds a widget on the host page through a script served from hcaptcha.com.

The widget renders inside an iframe whose origin is hcaptcha.com, cross-origin to the host page. When the user clicks the “I am human” checkbox, the widget decides whether to issue a challenge. If it does, a second iframe opens with the puzzle dialog. The puzzle layout varies between runs (3x3 grid, 4x3 grid, area-select with a single click on an image, bounding-box, multiple-choice prompt). When the puzzle is solved, the widget writes a response token to a hidden textarea[name="h-captcha-response"] on the host page. The host form reads that textarea on submit and posts the token along with the rest of the data. The whole solve interaction happens inside a frame the host page cannot script. The same-origin policy boundary blocks it.

That last detail decides what works and what does not. An agent driving the host page through Playwright or CDP has full control over the outer page. Inside the hcaptcha.com frame, its control is limited. A browser extension runs with cross-origin privileges. It can both observe and click inside the widget frame. That asymmetry explains most of what follows.

We started by reading three open-source agent repos to see what each one had decided to do at this boundary.

Skyvern OSS bails to the human

Skyvern’s README is candid about the scope of the OSS release: “All of the core logic powering Skyvern is available in this open source repository licensed under the AGPL-3.0 License, with the exception of anti-bot measures available in our managed cloud offering.” That single sentence puts the CAPTCHA section of every Skyvern Cloud feature page outside the repo you can read.

The repo confirms the framing. In agent_functions.py, lines 800-804, the auto_solve_captchas helper returns False unconditionally. In handler.py, lines 1149-1162, handle_solve_captcha_action does exactly one thing of substance.

async def handle_solve_captcha_action(
    action: actions.SolveCaptchaAction,
    page: Page,
    scraped_page: ScrapedPage,
    task: Task,
    step: Step,
) -> list[ActionResult]:
    LOG.warning(
        "Please solve the captcha on the page, you have 30 seconds",
        action=action,
    )
    await asyncio.sleep(30)
    return [ActionSuccess()]

Thirty seconds of asyncio.sleep and a log message asking a human to handle it. Then a success result, regardless of what the human actually did. The script-generation path in skyvern_page.py, lines 1252-1254 is even more direct. solve_captcha raises NotImplementedError. A Skyvern user opened issue #1117 asking how CAPTCHA solving was meant to work in OSS. A maintainer answered plainly: “We haven’t open sourced anything related to our captcha solver / anti-bot measures. We don’t want people abusing these things, so they must remain closed source unfortunately.”

The Skyvern OSS code does not pretend to solve CAPTCHAs. Whatever Skyvern Cloud does on top of this, the OSS release hands the problem to a human and moves on.

As always, the code that will use to bypass hCaptcha can be found in our GitHub repository reserved to paying users, inside the folder 105.HCAPTCHA.

Read more