Kameleo Docker brings Kameleo’s anti-detect browser capabilities into a containerized setup, enabling production-ready automation with real fingerprinting and multi-profile isolation on Linux servers.

In this post, I’ll take a deep look at this solution and walk you through everything you need to know about it. By the end, you’ll understand what Kameleo Docker is, how its stealth browser approach works, how to set it up, and whether it’s actually worth trying.

Before proceeding, let me thank NetNut, the platinum partner of the month. Their set of solutions cover all your needs for scraping.

Visit Netnut

An Introduction to Kameleo Docker

Dig into the world of Kameleo Docker!

What Is Kameleo Docker?

Kameleo is an anti-detect browser engineered to make browser sessions look like real user devices. Instead of exposing a generic automation fingerprint, it creates realistic browser identities by spoofing hardware, browser, and environment signals such as WebGL, Canvas, fonts, screen resolution, and geolocation.

Kameleo Docker brings that same stealth stack into a self-hosted, containerized deployment model. Rather than relying on a desktop app, you can run Kameleo inside Docker on Linux or Windows servers, CI pipelines, VPSs, or Kubernetes environments.

Playwright, Puppeteer, and Selenium can connect to the container via CDP, meaning you can keep using your existing browser automation logic with minimal changes.

Further reading:

• Kameleo Docker product page

• Kameleo Docker docs

• Kameleo on Linux via Docker: What We Built, What Broke, What’s Next

Why Kameleo Docker Exists

Cloud servers, Kubernetes clusters, CI/CD pipelines, and VPS environments are overwhelmingly Linux-native, making Linux compatibility a practical requirement for automation and scraping teams.

For years, this created a problem for Kameleo users, as the solution only supported Windows and macOS. Because of that, teams wanting to run stealth browsers in production often had to rely on fragile workarounds.

Some deployed Windows virtual machines alongside Linux scraping stacks in AWS. Others used X11-over-SSH tunnels to remotely access browsers running on servers. These setups were difficult to maintain, resource-intensive, and far from ideal for scalable automation.

As explained in the product announcement blog post, Kameleo customers started to ask for a version of Kameleo that could run directly where their automation already lived.

As Barnabas Szenasi, founder and lead engineer at Kameleo, explained when I met him at Prague Crawl 2026:

“We could see from customer messages that a significant slice of our automation-first users were running Linux cloud servers and simply couldn’t use Kameleo at all... At Prague Crawl 2025, Tamas [Kameleo’s CEO] and I heard the same story from industry peers around the world: scraping pipelines were getting harder, and the need to run real browser environments instead of faking HTTP requests was growing fast.”

The Philosophy Behind the Project

From the beginning, Kameleo’s philosophy has been simple: masking quality matters more than shipping quickly.

After all, an anti-detect browser is only useful if it can convincingly behave like a real device. That’s why Kameleo relies on fingerprints sourced from real-world device traffic and continuously tested against modern anti-bot systems.

That same quality-first mindset also shaped the Docker project. According to founder and lead engineer Barnabas Szenasi, Linux support took longer than expected because the goal was never just to make Kameleo run in a container.

The objective was to reach the same masking quality users already expected on Windows and macOS. Shipping a functional but lower-fidelity Linux version would have compromised the product’s core standard.

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

Technical Architecture Behind Kameleo Docker

Now that you know why the project exists, let me explain how it works and the engineering behind it.

How Kameleo Docker Works

Kameleo Docker runs inside either:

• a Linux-based container (Ubuntu 22.04), or

• a Windows-based container (Windows Server Core 2022).

When you pull the image, Docker downloads the correct variant based on your container configuration (Linux containers are the default in most environments, including Docker Desktop). Regardless of the underlying platform, Kameleo exposes the same Local API for browser and profile management as the desktop app.

The process begins when you create a browser profile through the Local API. If you aren’t familiar with that concept, Kameleo profiles are reusable browser environments that bundle a complete browser fingerprint together with persistent browser state, such as cookies, browsing history, local storage, and bookmarks.

Profiles can also include user-defined settings like proxies, browser extensions, and startup preferences. Each profile is tied to a specific browser kernel and can be started, stopped, imported, or exported as needed.

Architecture Overview

Kameleo Docker separates browser execution from automation logic. The container hosts the stealth browsers, fingerprinting systems, and orchestration layer, while your automation scripts run independently on your machine, server, or orchestration platform.

At the center of the architecture is the Local API, exposed on port 5050. This API handles profile creation, fingerprint selection, browser startup, lifecycle management, and more.

Behind the API sit Kameleo’s browser kernels:

• Chroma: A Chromium-based engine.

• Junglefox: A Firefox-based engine.

These kernels are modified with engine-level masking patches and connected to Kameleo’s continuously updated fingerprint database (more on this later). Since they are exposed through the same API, you can switch between them without making any code changes.

When a profile starts, Kameleo launches a browser session with the configured fingerprint and settings. Playwright and Puppeteer can then connect to the running browser through a WebSocket endpoint via Chrome DevTools Protocol (CDP).

In other words, your automation script stays outside the Docker container. The browser behaves as if it were running locally, while execution, fingerprint masking, and browser management happen entirely inside the Docker container.

Persistent storage is handled through Docker volumes. Profile data, downloaded browser kernels, and runtime state are stored outside the container, allowing environments to be recreated without losing configuration or repeatedly downloading browser components. This makes deployments easier to scale, recover, and reproduce.

Trusted by teams running ad verification, web scraping, SERP tracking, and market research. Ethically sourced proxies, globally accessible, and fairly priced.

Get Started With DataImpulse

Core Features of Kameleo Docker

Time to explore the main features and capabilities provided by Kameleo Docker. For more information, read the official documentation.

Real Device Fingerprint Masking

Kameleo fingerprints are derived from real-world device traffic, not synthetic templates. Each profile represents a coherent combination of OS, browser, hardware signals, and behavioral characteristics.

All fingerprint surfaces are kept internally consistent, including Canvas, WebGL, audio context, screen resolution, and fonts.

Note: TLS fingerprint spoofing isn’t required, as Kameleo matches the browser kernel version precisely. The TLS stack remains the original, unmodified implementation shipped with the corresponding browser release.

The goal of the project isn’t to spoof everything, but to maintain realism across signals. That’s because overriding too many surfaces increases inconsistency risk, which detection systems can flag. For example, running a macOS fingerprint on a Windows host forces heavy compensation across system-level signals.

Proxy Integration and Geo Consistency

Each Kameleo profile can be assigned a dedicated proxy (including a rotating proxy), allowing IP-level isolation between browser identities.

Now, mismatches between IP geography and browser signals (language, timezone, WebRTC, and system locale) are a common detection vector. To address that, Kameleo provides automatic geo-location matching to align the browser’s settings with the geographic location of the selected proxy IP address.

Multi-Profile Isolation

Kameleo Docker is built around strict profile isolation, as each browser profile runs as a fully independent environment. This separation opens the door to safe multi-accounting (referred to as “account management” in Kameleo terminology). Thanks for this feature, you can operate multiple identities simultaneously without cross-contamination of session data or signals.

Linux-Specific Docker Image Features

Compared to the Windows-based container, the Linux version of Kameleo Docker includes several additional features. These include:

• Built-in VNC viewer: Allows you to monitor and interact with live browser sessions. This is especially useful for debugging automation, validating fingerprints, or troubleshooting rendering issues. You can access it through a browser on port 8080 or via native VNC clients such as RealVNC or TigerVNC on port 5900. For security reasons, it’s disabled by default.

• Browser-Based Kameleo GUI: A lightweight browser-based GUI on port 80 (reach it at http://localhost:80). Unlike the desktop app, it offers reduced functionality and is primarily intended for quick inspection, basic profile management, and monitoring.

• Optional GPU acceleration: The Linux container supports optional GPU acceleration for graphics-heavy workloads such as WebGL or canvas-intensive websites. Intel/AMD GPUs can be mounted through /dev/dri, while NVIDIA GPUs are supported through the NVIDIA Container Toolkit. When no GPU is available, Kameleo falls back to software rendering.

Getting Started With Kameleo Docker: Step-by-Step Guide

In this guided section, I’ll show you how to set up Kameleo Docker and use it for browser automation against the Scraping Course’s “JavaScript Rendering” page:

This is a sandbox environment for web scraping that simulates a real-world, JavaScript-rendered ecommerce page. It makes for a great testing target to validate the setup and see how Kameleo Docker behaves in a realistic automation scenario.

Requirements and Prerequisites

To get started with Kameleo Docker, make sure you have:

• Docker installed and running locally.

• A Kameleo account with valid credentials.

For more details on supported operating systems and memory requirements, refer to the official documentation.

Since I’ll show how to use Kameleo Docker with Playwright in Python, to keep things moving, I’ll assume you already have a Python environment set up with Playwright and its dependencies installed.

To follow along with this tutorial section, I also recommend that you have:

• Basic Docker experience (running containers, mounting volumes, and using compose files)

• Familiarity with browser automation using Playwright.

Check the TWSC YouTube Channel

Step #1: Create a Kameleo Account

If you haven’t already, start by creating a Kameleo account. Fill out the sign-up form and enter the required information. Once registration is complete, a Free plan will already be activated:

Note that the Free plan is enough to use Kameleo Docker.

Important: Kameleo credentials are required for the container to authenticate successfully, download browser kernels, and start correctly.

Step #2: Start the Docker Container

With your account ready, the next step is to pull and run the Kameleo Docker image.

Remember that Kameleo ships as a multi-platform Docker image, supporting both Linux-based and Windows-based containers. To download and start the Linux container version of Kameleo, run:

docker run --platform linux/amd64 \
    --shm-size=2g \
    -p 5050:5050 \
    -e EMAIL='<YOUR_KAMELEO_EMAIL>' \
    -e PASSWORD='<YOUR_KAMELEO_PASSWORD>' \
    -v kameleo-data:/data \
    kameleo/kameleo-app:latest

Note 1: If you run this command in PowerShell, replace “\” with the backtick “`” for multi-line commands.

Note 2: To launch the web GUI included in the Linux container of Kameleo Docker, add the -p 80:80 argument to your docker run command.

Here’s what matters in the above command:

• --platform linux/amd64 ensures Docker pulls the Linux-based image variant.

• --shm-size=2g is required for stable browser execution (the default Docker shared memory of 64MB is too small for browser execution).

• -v kameleo-data:/data creates a named volume that persists browser kernels and profiles across restarts.

• EMAIL and PASSWORD authenticate your Kameleo account and enable kernel downloads.

Below’s the output you should get:

The Kameleo Docker image should now be downloaded and launched on your system. Cool!

Step #3: Verify the Service

Once you run the image, Kameleo Docker will:

1. Start the Local API on port 5050.

2. Authenticate using your credentials.

3. Download required browser kernels (first run only).

To confirm everything is running correctly, visit the following URL in your browser:

http://localhost:5050/swagger

You should see the Swagger UI for the Kameleo Local API:

You’ll have a local Kameleo instance ready for automation with Playwright, Puppeteer, or Selenium. Great!

Step #4: Download the SDK and Create Your First Profile

Now that Kameleo Docker is running, you can interact with it through the APIs exposed at http://localhost:5050. The next step is to proceed with the usual Kameleo setup by creating a profile.

Assuming you already have a Python environment with Playwright installed,start by installing the Kameleo SDK:

pip install kameleo-local-api-client

Then, in your Playwright script, initialize the Kameleo API client and generate a browser profile:

from kameleo.local_api_client import KameleoLocalApiClient
from kameleo.local_api_client.models import CreateProfileRequest

# Connect to the Kameleo API client
client = KameleoLocalApiClient(endpoint="http://localhost:5050")
# Search for a real-world fingerprint and create a Kameleo profile based on it
fps = client.fingerprint.search_fingerprints(
    device_type="desktop",
    os_family="windows",
    browser_product="chrome",
    browser_version=">145",
)
profile = client.profile.create_profile(
    CreateProfileRequest(fingerprint_id=fps[0].id, name="twsc demo")
)

The above snippet connects the Kameleo API client from the SDK to the local Kameleo Docker APIs. It retrieves a realistic browser fingerprint from the database and creates a persistent browser profile called “twsc demo” based on it. In this case, the fingerprint profile is for a desktop Chrome browser (version >145) running on Windows.

Run the script above. If you started the Linux container of Kameleo Docker while mapping port 80 for the web GUI, then you’ll be able to see the “twsc demo” profile at http://localhost:80:

Step #5: Connect With Playwright

You can now connect Playwright to the Kameleo profile created above via CDP using the following WebSocket URL:

ws://localhost:5050/playwright/<KAMELEO_PROFILE_ID>

Achieve that by using Playwright’s connect_over_cdp() method on the target URL:

from playwright.sync_api import sync_playwright

# Kameleo profile creation...

# Connect Playwright to the Kameleo instance based on the configured profile
ws_endpoint = f"ws://localhost:5050/playwright/{profile.id}"
with sync_playwright() as playwright:
    browser = playwright.chromium.connect_over_cdp(endpoint_url=ws_endpoint, timeout=90_000)

    # Regular Playwright automation logic...

Wonderful! Playwright is attached to the browser session managed by Kameleo Docker. You can now automate it using standard Playwright APIs as if it were a regular local Chromium instance.

Step #6: Implement the Automation Logic

To achieve the scraping goal, begin by inspecting the page to study its DOM structure:

Then, apply the following Playwright logic (connected to a Kameleo profile) to automate scraping on the JavaScript-rendered page:

# Connect Playwright to the Kameleo instance based on the configured profile
ws_endpoint = f'ws://localhost:5050/playwright/{profile.id}'
with sync_playwright() as playwright:
    browser = playwright.chromium.connect_over_cdp(endpoint_url=ws_endpoint, timeout=90_000)

    # Open a new page
    context = browser.contexts[0]
    page = context.new_page()

    # Visit target site
    page.goto("https://www.scrapingcourse.com/javascript-rendering")

    # Where to store the scraped data
    products = []

    # Wait for products to render
    page.wait_for_selector(".product-item")

    # Locate all product items
    product_elements = page.locator(".product-item")

    for i in range(product_elements.count()):
        # Select the nth product
        product_element = product_elements.nth(i)

        # Extract the product data
        name = product_element.locator(".product-name").inner_text()
        price = product_element.locator(".product-price").inner_text()
        image = product_element.locator("img.product-image").get_attribute("src")
        link = product_element.locator("a.product-link").get_attribute("href")

        # Populate a product object with the scraped data
        product = {
            "name": name,
            "price": price,
            "image": image,
            "url": link
        }
        # Append it to the products list
        products.append(product)

The above snippet instructs the controlled browser to visit the target page, waits for product elements to load, then iterates through each product DOM node to extract structured fields (name, price, image, URL) and stores them in a Python list for downstream processing.

The Kameleo-powered automation script is almost complete. Only one step remains!

Step #7: Stop the Kameleo Profile

Normally, in a Playwright scenario, you would need to call browser.close() to terminate the browser session and release its resources.

In Kameleo, that’s not required. Instead, you only need to call:

client.profile.stop_profile(profile_id=profile.id)

The above line of code sends a close command to the browser via CDP. Once the browser actually stops, the Kameleo profile is terminated, too. This ensures that all resources associated with both the browser and the running profile are properly released.

Step #8: Run the Script

The final Playwright automation script, connecting via CDP to the stealth browser instance exposed by Kameleo Docker, will contain:

# pip install playwright kameleo-local-api-client

from kameleo.local_api_client import KameleoLocalApiClient
from kameleo.local_api_client.models import CreateProfileRequest
from playwright.sync_api import sync_playwright

# Connect to the Kameleo API client
client = KameleoLocalApiClient(endpoint="http://localhost:5050")
# Search for a real-world fingerprint and create a Kameleo profile based on it
fps = client.fingerprint.search_fingerprints(
    device_type="desktop",
    os_family="windows",
    browser_product="chrome",
    browser_version=">145",
)
profile = client.profile.create_profile(
    CreateProfileRequest(fingerprint_id=fps[0].id, name="twsc demo")
)

# Connect Playwright to the Kameleo instance based on the configured profile
ws_endpoint = f'ws://localhost:5050/playwright/{profile.id}'
with sync_playwright() as playwright:
    browser = playwright.chromium.connect_over_cdp(endpoint_url=ws_endpoint, timeout=90_000)

    # Open a new page
    context = browser.contexts[0]
    page = context.new_page()

    # Visit target site
    page.goto("https://www.scrapingcourse.com/javascript-rendering")

    # Where to store the scraped data
    products = []

    # Wait for products to render
    page.wait_for_selector(".product-item")

    # Locate all product items
    product_elements = page.locator(".product-item")

    for i in range(product_elements.count()):
        # Select the nth product
        product_element = product_elements.nth(i)

        # Extract the product data
        name = product_element.locator(".product-name").inner_text()
        price = product_element.locator(".product-price").inner_text()
        image = product_element.locator("img.product-image").get_attribute("src")
        link = product_element.locator("a.product-link").get_attribute("href")

        # Populate a product object with the scraped data
        product = {
            "name": name,
            "price": price,
            "image": image,
            "url": link
        }
        # Append it to the products list
        products.append(product)

    # Print the scraped products
    for product in products:
        print(product)

# Stop the Kameleo profile
client.profile.stop_profile(profile_id=profile.id)

Execute the script, and you should see output similar to this:

Notice how the script successfully scraped product data from the JavaScript-rendered page.

Once execution completes, open the Kameleo web GUI, and you’ll notice that the “twsc demo” profile is now marked as “TERMINATED”:

That doesn’t mean the profile is gone forever. Quite the opposite!

Kameleo profiles are reusable, meaning you can retrieve and start them again later to continue the browsing session with the same fingerprint and persisted state. I’ll cover exactly how to do that in a dedicated FAQ.

Mission complete! You just learned how to use Kameleo Docker for Playwright automation. With very similar logic, you can automate Puppeteer, Selenium, or any other CDP-compatible solution, in both Python and JavaScript.

Pricing Model

Kameleo Docker is included across all plans at no additional cost. You get the same core limits (concurrent browsers, number of profiles, and browser usage time) regardless of whether you run the desktop app or the containerized version. So, take a look at the official pricing page for more information.

Anti-Bot Performance Benchmarks

To test Kameleo Docker, I ran a simple script against one page protected by each major anti-bot detection system. The results are shown below:

Note: All tests were performed locally using my ISP’s residential IP address.

As shown above, in this basic experiment, Kameleo Docker achieved a 100% success rate. In contrast, Playwright consistently failed in headless mode and, in some cases, also struggled in headful mode.

Need help with your scraping project?

Final Thoughts on Kameleo Docker

What stood out when I met Barnabas at Prague Crawl 2026 (see you next year 😉) was the clear passion the team has for the project, along with their focus on quality and continuous improvement.

At the same time, when testing new products, especially technical and complex ones like Kameleo Docker, you usually stumble across bugs or unexpected behavior. I can confidently say that this wasn’t the case at all here. Everything ran smoothly from the beginning, and I didn’t encounter any issues…

On top of that, the benchmark results are promising, and I didn’t notice any significant performance lag. Thus, my honest takeaway is simple: if you’re looking for a production-ready, containerized stealth browser, or you’re simply passionate about automation and scraping, consider giving Kameleo Docker a try!

In this article, I covered what the project is about, what it offers, how it works, and how to use it. As always, remember to use Kameleo Docker only for legal and ethical web scraping and automation. Until next time!

FAQ

Is Kameleo Docker different from the desktop app?

Kameleo Docker differs from the desktop app mainly in deployment. Instead of a local GUI, it runs as a containerized service that runs on both Linux and Windows servers. Feature parity is largely preserved, including profiles, fingerprinting, and browser engines.

Can I reuse already created profiles in Kameleo Docker?

Yes! Profiles can be reused by retrieving the full list of profiles, filtering by name (or ID, if you know it), and then starting the desired profile. For example, to reuse the “twsc demo” profile, write:

from kameleo.local_api_client import KameleoLocalApiClient
from kameleo.local_api_client.models import ProfileLifetimeState

# Connect to the Kameleo API client
client = KameleoLocalApiClient(endpoint="http://localhost:5050")

# Fetch all available profiles
profiles = client.profile.list_profiles()

# Find the profile with the specific name
target_name = "twsc demo"
profile = next((p for p in profiles if p.name == target_name), None)

# Check if the profile was found
if profile:
    # Start the existing profile if it isn't already running
    if profile.status.lifetime_state != ProfileLifetimeState.RUNNING:
      client.profile.start_profile(profile.id)

Can I use Kameleo Docker in my CI/CD?

Kameleo Docker fits naturally into CI/CD pipelines by running as a disposable, reproducible container in build or test stages. You can spin up browsers on demand, run automated flows, and tear them down after execution. Configuration is typically handled via Docker Compose.

Does Kameleo Docker support proxy integration?

Yes! HTTP, HTTPS, and SOCKS proxies can be configured at profile creation time, as explained in the documentation.

Can Kameleo Docker scale to thousands of browsers?

Kameleo Docker supports horizontal scaling through standard orchestration tools. You can run multiple containers across clusters using Kubernetes or AWS ECS, each managing independent browser instances.

How Does Firefox Automation Work in Kameleo Docker?

Kameleo Docker supports Firefox-based automation through the Junglefox engine. Because Playwright cannot connect directly to Firefox-based sessions, Kameleo provides a pw-bridge helper that acts as a compatibility layer. This component translates Playwright connections into the correct browser session, allowing standard automation scripts to run unchanged while still using Firefox-based fingerprint profiles.

Did you like this article? Share it with someone who might find it useful and get a discount on paid plans.

Share

Before proceeding, let me thank NetNut, the platinum partner of the month. Their set of solutions cover all your needs for scraping.

Visit Netnut

Shopee does not hand you the easy version. Capture a request, replay it, and the backend answers with HTTP 418 and a security error code. Every API call carries a set of anti-fraud headers, and the code that builds them is not in the Java you can read with JADX. It sits in native code, in compiled .so libraries, which is exactly where traffic interception and a Java decompiler stop being useful. Open the app in JADX and the signing method is there in name only, declared native, with its body on the far side in ARM machine code.

This is a two-part investigation into how Shopee signs its API requests and how you reproduce that signing yourself. The strategy is the one that works on most hardened apps. You locate the native security libraries, open them in a disassembler, and turn what they do back into something you control. When a library is readable crypto, you reimplement it in Python and sign offline, at any volume, with no app in the loop. When it is a bytecode virtual machine you cannot practically rewrite, you keep the app running and drive its own signer as an oracle. We chose this route because it is the one that scales. An offline signer, or an oracle you call, runs inside your scraper on a server. A rooted phone you have to babysit does not.

This first part is the foundation. We take one of Shopee’s native libraries, libshopeeaegis.so, reverse it end to end with Ghidra and rebuild it in Python. Reading the decompiled code identifies every operation as textbook crypto, and our rebuild reproduces those algorithms byte for byte. It is the readable case, the kind you win cleanly, and the clearest worked example of the method. The second part takes on the harder library, the one that computes the per-request signature, and gets past it with the oracle approach.

What you take from each part depends on your goal. If Shopee is your target, the payoff is the full picture of its request signing across both parts. If you scrape other apps, the method matters more than the marketplace. Most apps that protect their API at all push the work into a native library, and a large share of those are plain, readable crypto you can reproduce. We work it out on Aegis here, and it is the same move on the next app you open.

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

The tools

We use four tools, each doing one job.

androguard is a Python library for static APK analysis. We use it for fast recon. It lists the native libraries an app ships and finds which classes declare native methods. It does not give you readable source. It gives you structure you can script.

JADX decompiles Dalvik bytecode back to Java. It is how you read the managed side of the app and find the exact class and method that crosses into native code. JADX stops at the native keyword, which is the handoff point to the next tool.

Ghidra is the NSA’s open source reverse engineering framework. It disassembles a .so and decompiles it to pseudo C. It is the only tool here that can read native code, and it is the one this article leans on.

Frida injects a JavaScript engine into a running process so you can hook and call functions live. We use it to run the app under instrumentation and confirm our static reading against what the app actually does.

JADX and androguard read the managed code. Ghidra reads the native code. Frida watches the code run. The native library is the one piece only Ghidra can open, so the work centers there.

Modeling the app’s defenses

Before opening anything, it helps to name the layers, because Shopee has several and only one is our target here.

The managed layer is the Java and Kotlin code. It builds requests, attaches headers, and calls into native methods. JADX reads it.

The native layer is a set of .so libraries the app loads. Pull the arm64 split out of the APK with androguard and the security-relevant ones stand out by name. They are libshopeeaegis.so, libshpssdk.so, and libBkeBizSecurity.so, plus libjnihook.so and libshook.so. The last two are a hooking framework and an anti-hook layer, which means the app actively watches for instrumentation. That matters for Frida later.

The request-signing layer sits on top. Two okhttp interceptors, com.shopee.app.network.antifraud.b and .d (they call themselves SecurityNewSapInterceptor and SecurityNewSapPostInterceptor), attach the anti-fraud headers af-ac-enc-sz-token and x-sap-ri to API requests. The values they attach come from libshpssdk.so, the Shopee Security SDK.

We target one layer, libshopeeaegis.so, a general-purpose crypto library the app calls for specific operations. The request signer in libshpssdk.so stays out of scope. It is a bytecode virtual machine, a harder problem that we handle separately, and reproducing the af-ac headers is not the promise here. The promise is that you can take libshopeeaegis.so, understand every operation it performs, and reproduce it byte for byte in Python.

One detail decides whether that promise holds. libshopeeaegis.so loads only when the app needs it, so it is not present at idle. We watched the process maps over a minute of normal browsing and the library never appeared. The crypto we are about to reverse is a toolbox the app reaches for in certain flows, not the thing running on every request.

Start your scraping journey with Byteful: 10GB New Customer Trial | Use TWSC for 15% OFF | $1.75/GB Residential Data | ISP Proxies in 15+ Countries

Claim your 10GB here

Getting the library and finding the door

We pulled Shopee PH 3.75.24 (com.shopee.ph) as an XAPK and unzipped it. The native libraries are not in base.apk. For a split bundle they live in config.arm64_v8a.apk. Listing the .so files with androguard and unzip -l, libshopeeaegis.so is a small one at 280 KB, which is a good sign. Small means little room for a heavy obfuscator.

androguard answers the first question, which library to open. It does not answer the second, how the app calls it. For that we go to JADX and find the class on the Java side. The library registers its native methods against com.shopee.sz.reinforce.Aegis. The class exposes a method fire, overloaded, declared native. Two of the overloads matter:

native byte[] fire(int mode, byte[] data)
native byte[] fire(int mode, byte[] data, byte[] key)

This is the door. The first argument is an integer mode. Then one or two byte arrays. The return is a byte array. JADX cannot show what fire does, because the body is in the .so. So we open the .so.

Check the TWSC YouTube Channel

Reading the library with Ghidra

We ran Ghidra 12.1.2 in headless mode. It runs without a GUI, it scripts cleanly, and it repeats exactly. The workflow is documented in our Ghidra tool skill you can use in Claude Code, just like I did for this test. In short, you import the .so, let auto analysis run, then run a script that decompiles functions to a file.

support/analyzeHeadless /tmp/proj aegis \
  -import config.arm64_v8a/lib/arm64-v8a/libshopeeaegis.so \
  -scriptPath ./scripts \
  -postScript DecompileExport.java out.c \
  -overwrite

Auto analysis finished in nine seconds and the decompiler produced 674 functions with zero failures. That number alone tells you this is not a packed or virtualized binary. A protected library fights the decompiler; this one did not.

The first useful function is JNI_OnLoad, which every JNI library runs at load time. Read its pseudo C and it looks up the class com/shopee/sz/reinforce/Aegis and calls RegisterNatives with two methods. That confirms the door from the Java side and tells us the native functions are registered dynamically rather than exported under Java_* names. Dynamic registration is a mild form of hiding, and it is exactly what Ghidra’s JNI handling and a RegisterNatives trace are for.

The C++ symbols survived. That is the break that makes this library readable. The class is Aegis, with methods named missileFire, missileCount, prism, snowDon, tugWar, and parse. There is a second class, TeslaModel, with model_3, model_a, model_b, model_c, model_e, model_s, model_x, model_y, and getNuremberg. The names are deliberately silly, a Tesla and military theme, but they are real symbols, and the structure is intact.

Follow the call chain from the registered native function. The dispatcher is Aegis::prism, a plain switch on the mode integer:

switch(param_1) {
case 0:  model_3(...)               // one input
case 1:  model_x(key, input, ...)   // keyed
case 2:  model_x(...); model_3(...) // keyed, then case 0
case 3:  model_y(...); model_3(...)
case 4:  model_e(...)
case 5:  model_a(...)
case 6:  model_b(...)
case 7:  model_s(...); model_3(...)
case 8:  model_c(...); model_3(...)
}

One native call selects one of nine operations by an integer, and some operations are a keyed primitive followed by model_3. To name each operation we read two things, the output size and the primitive body.

The output size comes from TeslaModel::getNuremberg(mode, len), which missileFire calls to size the output buffer before doing the work. It returns 16 for mode 4, 32 for mode 5, 64 for mode 6, and 20 for mode 8. Those are the digest sizes of MD5, SHA-256, SHA-512, and SHA-1. For mode 0 it returns the Base64 expansion of the input length. The size function alone half-names the table.

The bodies confirm the rest, and here the silly names get helpful, because the renamed primitives kept their original suffixes. model3_autopilot is a textbook Base64 encoder. It reads three bytes, writes four, and pads with 0x3d, which is the = character. modelx_autopilot_cbc is AES in CBC mode, recognizable because it XORs each 16 byte block with the previous ciphertext block before the round function. The hash contexts are renamed with a phantom and F theme but keep the gnulib _init_ctx / _process_bytes / _finish_ctx shape. phantom1 is SHA-1, phantom256 is SHA-256, InitF22 is SHA-512. And phantom1 as called by model_c is the HMAC form. It XORs the key with 0x36 for the inner pad and 0x5c for the outer pad over a 64 byte block, which is the HMAC construction.

Two of the keyed modes turned out not to be ciphers at all. model_s calls phantom256 with a key and a message and returns 32 bytes, so it is HMAC-SHA256. model_c calls phantom1 the same way and returns 20 bytes, so it is HMAC-SHA1. Reading the bodies kept us honest here. From the signatures alone we had guessed AES.

That gives the full table.

model_x pads with PKCS7 to a 16 byte boundary, then runs AES-CBC. The key length sets the variant, and a 16 byte key gives AES-128. model_y does the same but writes the IV in front of the ciphertext, the standard prepend-the-IV pattern, before the Base64 in mode 3.

One value is not in the file. The CBC IV is a fixed 16 byte constant the library keeps at a .bss address. .bss is zero-initialized on disk and filled at runtime, so the IV is set when the library initializes and you cannot read it statically. For the hash, HMAC, and Base64 modes that does not matter, because their output is fully determined by the input and key. For the three AES modes it means byte-identical output needs the real IV, which you read from the live process once the library loads.

As always, the code that will be used for the python reimplementation we’re showing now can be found in our GitHub repository reserved for paying users, inside the folder 107.SHOPEE-GHIDRA.

Read more

Now, since the rise of AI and LLMs, every office job has changed, with no exceptions for scraping professionals. For us, this means that surviving anti-bots and CAPTCHAs has become the first challenge when creating scraping pipelines. The second challenge has become the integration of scraping services and capabilities into AI pipelines and AI agents. The reason for this shift is simple: your boss or your clients don’t just want the data anymore. They want it in real-time, structured, and often packed with some insights.

Until some months ago, these pipelines required a huge amount of custom code (and time!). Luckily for us, the major struggle lasted until MCPs were first released.

In this article, you’ll learn what MCP is, how the Decodo MCP server works, and how to integrate it with Claude Desktop. You’ll also learn how to use it with two hands-on examples.

Let’s get into it!

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

What is MCP?

The Model Context Protocol (MCP) is an open standard introduced by Anthropic that defines how AI models connect to external tools, data sources, and services. Before MCP, every integration between an AI model and an external system had to be built from scratch. MCP replaces all of the complexity behind the integration with a single, shared protocol.

Basically, the MCP protocol acts as a common language for models to connect with external tools, files, and systems. For example: do you need an AI assistant to pull a file from Google Drive, query a company database, and trigger an action in an internal app? That’s exactly the kind of job MCP is built to handle!

The practical upside is composability. Developers can mix and match several MCP servers into a single AI application without writing any integration code. So, after more than a year since its introduction in the AI industry, MCP has become the standard for integrating different services and applications into a single AI software.

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

What is Decodo MCP?

The Decodo MCP Server is a web scraping layer for AI agents. It connects MCP-compatible clients to Decodo’s Web Scraping API, enabling:

• Web scraping for LLMs and AI agents without managing infrastructure: It can scrape any website, including JavaScript-heavy pages, and get real-time data without handling proxy rotation, CAPTCHA solving, or anti-bot systems. It is specifically built for RAG pipelines, AI research agents, and automation flows.

• Structured outputs for LLM workflows: It allows you to retrieve the scraped data in Markdown (LLM-ready), JSON (for structured pipelines), and screenshots (for visual context).

As of now, the Decodo MCP server exposes the following tools:

• scrape_as_markdown: Scrapes any target URL, given a target URL via prompt. Returns results in Markdown.

• screenshot: Captures a screenshot of any webpage and returns it as a PNG image.

• google_search: Scrapes Google Search for a given query, and returns parsed results.

• google_ads: Scrapes Google Ads search results.

• google_lens: Scrapes Google Lens image search results.

• google_ai_mode: Scrapes Google AI Mode (Search with AI) results.

• google_travel_hotels: Scrapes Google Travel Hotels search results.

• amazon_search: Scrapes Amazon Search for a given query, and returns parsed results.

• amazon_product: Scrapes a given Amazon Product page.

• amazon_pricing: Scrapes Amazon Product pricing information.

• amazon_sellers: Scrapes Amazon Seller information.

• amazon_bestsellers: Scrapes Amazon Bestsellers list.

• walmart_search: Scrapes Walmart Search for a given query, and returns parsed results.

• walmart_product: Scrapes Walmart Product page.

• target_search: Scrapes Target Search for a given query, and returns parsed results.

• target_product: Scrapes Target Product page.

• tiktok_post: Scrapes a TikTok post URL.

• tiktok_shop_search: Scrapes TikTok Shop Search for a given query, and returns parsed results.

• tiktok_shop_product: Scrapes TikTok Shop Product page.

• tiktok_shop_url: Scrapes TikTok Shop page by URL.

• youtube_metadata: Scrapes YouTube video metadata.

• youtube_channel: Scrapes YouTube channel videos.

• youtube_subtitles: Scrapes YouTube video subtitles.

• youtube_search: Search YouTube videos.

• reddit_post: Scrapes a specific Reddit post.

• reddit_subreddit: Scrapes Reddit subreddit results.

• reddit_user: Scrapes a Reddit user profile and their posts and comments.

• bing_search: Scrapes Bing Search results.

• chatgpt: Search and interact with ChatGPT for AI-powered responses and conversations.

• perplexity: Search and interact with Perplexity for AI-powered responses and conversations.

NOTE: Decodo is currently onboarding its MCP server onto various platforms and marketplaces. At the moment of writing this article, it can be found on the Official MCP registry, Pulse MCP, Glama AI, mcp.so, and mcpmarket.com.

How To Integrate The Decodo MCP With Claude

The prerogative of MCPs is to integrate them with LLMs. The Decodo MCP server can be integrated with several services that leverage LLMs, like Claude and Cursor. In this paragraph, you will learn how to integrate it with Claude.

Requirements

To use the Decodo MCP, your system must satisfy the following requirements:

• Claude Desktop: To integrate the Decodo MCP server with Claude, you need Claude Desktop. If you don’t have it installed yet, you can download it from their website.

• Decodo account: Create an account at dashboard.decodo.com. With a free one, you have up to 2K free requests.

• Scraping token: Get the basic authentication token. To get it, click on Web Scraping API > API playground:

  

Good. Your system is set to connect Claude Desktop with the Decodo MCP server.

Connect The Decodo MCP Server to Claude

To connect the Decodo MCP server to Claude, open Claude Desktop and click on Settings:

Then, click on Developer:

After clicking on Edit Config, the system will automatically open a folder where Claude stores all the configuration files on your local machine. Open the claude_desktop_config.json file and add the following to it:

"mcpServers": {
    "Decodo MCP Server": {
      "command": "npx",
      "args": [
        "-y",
        "@decodo/mcp-server"
      ],
      "env": {
        "SCRAPER_API_TOKEN": "<your-decodo-mcp-api-key>"
      }
    }
  } 

Replace <your-decodo-mcp-api-key> with the basic authentication token you retrieved earlier from the Decodo dashboard, and the integration is done.

Quit Claude to make the changes effective. Note that just closing the Desktop window is not sufficient. You have to quit it:

After that, when returning to Settings > Developer, you’ll see the MCP server up and running:

To be sure everything works fine, you can test it with a prompt similar to the following:

Alright, you successfully integrated the Decodo MCP server with Claude Desktop. Now it's time to test it!

Check the TWSC YouTube Channel

Running the Decodo MCP With Claude: Hands-On Examples

In this section, you will learn how to use the Decodo MCP server with two different examples:

• A basic usage taken from the documentation.

• A more advanced usage where you’ll ask Claude to retrieve some data from Amazon, returning the result in a JSON file.

Let’s get on to this!

Getting Started: Run a Google Search in Seconds

As a first and simple usage, you can test the google_search tool. The main idea behind this tool is to prompt the model with a query, and, under the hood, the MCP will use its Google search capabilities to return the result.

To just try things out, you can use the exact example reported in the Decodo MCP documentation to search for shoes on Google, reporting the top positions:

As you can see, Claude automatically tries to use the MCP server, loading its tool. As Claude is a production-ready LLM, you will be asked if you want to allow it to use the Google search from the Decodo MCP always or just for this call.

When Claude has completed its job, the result is the following:

If you want to verify that the model hasn’t hallucinated, you can search for “shoes” on Google:

Alright! You made it to your first usage example.

Need help with your scraping project?

Leveling Up: Extracting JSON-structured Data From Amazon

Having results listed in a chat can be useful for a quick overview of web research. But the actual power of MCPs is to use the capabilities of the underlying tools for specific tasks that LLMs can solve.

An example is to ask the model to retrieve some data and report the results in a JSON file. This solution provides you with data that can be further used in the second part of your pipeline—for example, for analyzing your scraped data.

For this purpose, you can use the following prompt:

Get Amazon bestsellers in electronics, extract the main info and return it in JSON format

The image below shows Amazon’s best sellers in electronics, at the time of writing this article:

Under the hood, Claude will trigger the amazon_bestsellers tool and will search the data in electronics. The chat result is the following:

Below is the (partial) JSON returned by Claude:

{
  "metadata": {
    "category": "Electronics",
    "source": "Amazon Best Sellers",
    "url": "<https://www.amazon.com/Best-Sellers/zgbs/electronics/>",
    "scraped_at": "2026-05-18",
    "total_items": 50,
    "currency": "USD"
  },
  "bestsellers": [
    {
      "rank": 1,
      "asin": "B08JHCVHTY",
      "title": "Blink Plus Plan with Monthly Auto-Renewal",
      "price": 11.99,
      "rating": 4.4,
      "ratings_count": 275779,
      "image_url": "<https://images-na.ssl-images-amazon.com/images/I/31YHGbJsldL._AC_UL300_SR300,200_.png>",
      "url": "<https://www.amazon.com/Blink-Plus-Plan-monthly-auto-renewal/dp/B08JHCVHTY>"
    },
    {
      "rank": 2,
      "asin": "B0DCH8VDXF",
      "title": "Apple EarPods Headphones with USB-C Plug",
      "price": 19,
      "rating": 4.6,
      "ratings_count": 13500,
      "image_url": "<https://images-na.ssl-images-amazon.com/images/I/513OSdW4elL._AC_UL300_SR300,200_.jpg>",
      "url": "<https://www.amazon.com/Apple-EarPods-Headphones-Built-Control/dp/B0DCH8VDXF>"
    },
    
    <Omitted for brevity>

    }
  ]
}

As you can see, the first two items correspond to the first two in the image above, taken from the Amazon best seller page. This is just to be sure the model hasn’t hallucinated.

Well done! You learned how to use the Decodo MCP server with Claude.

Conclusion

In this article, you learned what MCP is and why it has become the standard for connecting AI models to external tools and services. You also learned what the Decodo MCP server is, how to integrate it with Claude Desktop, and how to use it in practice.

So, let us know: what kind of scraping workflows are you planning to build with the Decodo MCP?

Before proceeding, let me thank NetNut, the platinum partner of the month. Their set of solutions cover all your needs for scraping.

Visit Netnut

Part of that is the cat-and-mouse game every stealth tool plays. Part of it is specific to open source. When the entire fingerprint-spoofing codebase is public, the anti-bot vendors can read it line by line and build the exact counter-signal. We made that argument in the rayobrowse review. The openness that made Camoufox popular is the same openness that let the anti-bot giants study it and catch up.

Two things changed in 2026 that make this worth a fresh look. First, the project moved. The repository at github.com/daijro/camoufox now carries a note at the top of its README:

Clover Labs is a Toronto venture studio building AI agents, listed among the project sponsors. The alpha features (per-context fingerprints, hardware spoofing) now ship first in their cloverlabs-camoufox package, and daijro’s repo became the checkpoint mirror. This is not an abandoned project; the main maintainers changed.

Second, that public repo has more than 750 forks. Open source means that when one person stops, others can pick up the work, add features on features, and keep the chase going in parallel. So the real question is not only “is Camoufox still effective”, it is “has anyone in the fork tree built something better”. This is what we tried to discover in this article.

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

The forks we actually tested

We pulled the fork list from the GitHub API and sorted it by recent pushes. Most of it is noise. Many forks share the exact pushed_at timestamp of the parent, which is the signature of mirror bots that never wrote a line of their own. Once you count how many commits each fork is ahead of daijro:main and read what those commits do, the field collapses to a handful. Many of the survivors only touch CI or rebrand the binary. Three of them touch the anti-detect surface for real.

Official Camoufox (github.com/daijro/camoufox) is the baseline. A custom Firefox build with a fingerprint database and stealth patches, driven through Playwright’s Juggler protocol. We covered how it hides Playwright’s own traces in THE LAB #65, so we will not repeat that here.

camoufox-reverse (github.com/WhiteNightShadow/camoufox-reverse) goes the other way. Instead of hiding harder, it adds a PropertyTracer at the SpiderMonkey engine layer that records which DOM properties a page reads. It is an instrument for watching the detector work, not a better scraper. That makes it the most useful tool in the set for understanding what we are up against.

LeooNic/camoufox (github.com/LeooNic/camoufox) is the most ambitious on paper. Its commits add content-aware canvas noise that claims to defeat a 2025 academic pixel-recovery attack, a sigma-lognormal humanized mouse engine, and RDPBrowser, an automation path that drives Firefox over the Remote Debugging Protocol instead of Juggler.

JWriter20/camoufox (github.com/JWriter20/camoufox) is the pragmatic one. Targeted stealth fixes, the headline being a closed WebRTC IP leak under a proxy on Firefox 146 (daijro issue #538), plus a real pytest suite, which none of the others ship.

Let’s start by using camoufox-reverse to discover something more about DataDome installed on Leboncoin.fr.

What DataDome reads, watched from inside the engine

Before testing who passes, we wanted to see what the detector looks at. We have explained the three detection layers before: behavioral, browser, and HTTP, in THE LAB #6. camoufox-reverse lets us watch the browser layer from below the JavaScript, which is a view we have never had in these pages.

The PropertyTracer is documented to be enabled via a config flag. We drove the macOS arm64 build directly with Playwright, set the trace config through the CAMOU_CONFIG environment variable, and pointed it at a DataDome-protected page. Our target throughout this article is leboncoin.fr, the French classifieds site, because it runs only DataDome. That isolates the signal we care about, with no second anti-bot muddying the result.

The full probe is in code/camoufox_fork_analysis/trace_datadome.py. The core of it sets the trace and lets DataDome’s script run:

config = {
    "propertyTrace": {
        "enabled": True,
        "logDir": str(LOG_DIR),
        "objects": [],            # empty = trace all covered getters
        "maxEventsPerSession": 200000,
    }
}
env = os.environ.copy()
env["CAMOU_CONFIG"] = json.dumps(config)
env["MOZ_DISABLE_CONTENT_SANDBOX"] = "1"  # required on macOS for the tracer

When sites get tough, skip the heavy lifting. Get clean, structured CSV datasets, ready for Excel, BI or your apps

Find your dataset

The tracer writes one JSON line per getter access, each shaped like {"o": "navigator", "p": "hardwareConcurrency", ...}. Loading the leboncoin homepage produced 140 engine-level reads across 30 distinct properties. Aggregated by object and property, the access pattern contains information useful for fingerprint creation:

 COUNT  PROPERTY
    14  window.outerWidth
    13  window.devicePixelRatio
    13  window.outerHeight
    13  navigator.plugins.indexedGetter
     9  navigator.hardwareConcurrency
     7  canvas.toDataURL
     6  window.innerWidth
     6  screen.rect
     4  navigator.platform
     4  navigator.userAgent
     4  webgl.getParameter
     4  canvas2d.getImageData
     3  navigator.maxTouchPoints
     2  offscreenCanvas.getContext

Check the TWSC YouTube Channel

This is happening entirely below the JavaScript layer. From the page’s point of view, nothing was instrumented, because the recording lives in the C++ getter, not in a JavaScript proxy. DataDome reads the screen geometry, the navigator core, the plugin and mime enumeration, and then it reaches for the canvas and WebGL. Both canvas.toDataURL and canvas2d.getImageData are in the list, alongside webgl.getParameter and offscreenCanvas.getContext.

That last detail is what connects this experiment to the rest of the article. The canvas readback is exactly the surface LeooNic’s content-aware noise patch sets out to protect, and the WebRTC and screen reads are where the other forks claim improvements. We now know the detector touches it all.

The homepage is the light version. The pages that hold the data are watched far more closely, and the tracer shows it. We pointed the same probe at a car listing (a leboncoin /ad/voitures/ URL). Those pages block direct connections, so this run went through a residential proxy, which is the setup we explain in the next section. The listing loaded its real content (the page title came back as “Alfa romeo Tonale 1.5 Ibrida 175ch Veloce TCT”), so we were tracing a passing ad page, not a challenge screen. The read pattern is a different animal: 584 engine-level reads across 35 properties, against 140 across 30 on the homepage.

 COUNT  PROPERTY (ad page)
   220  document.cookie.get
    47  window.innerWidth
    30  window.innerHeight
    26  navigator.plugins.indexedGetter
    26  screen.rect
    25  sessionStorage.setItem
    22  sessionStorage.getItem
    16  document.cookie.set
    12  performance.timing
     8  window.scrollY
     7  canvas.toDataURL
     6  webgl.getParameter
     4  canvas2d.getImageData
     3  navigator.globalPrivacyControl
     1  mediaDevices.enumerateDevices

The cookie reads jump from one on the homepage to 220 on the ad page. Session storage, which the homepage barely touched, is read and written dozens of times. New surfaces appear that the homepage never queried: window.scrollY for behavior, navigator.globalPrivacyControl, and mediaDevices.enumerateDevices. The canvas and WebGL reads are still there. This is the same DataDome, running a heavier script on the page that matters. It is the concrete reason the homepage passes a clean browser while the listings do not. It also tells you where to spend your effort. The protection you have to beat lives on the content pages, not the landing page.

Setting up a fair comparison

The shared virtual environment we’re creating already had camoufox 0.4.11, which fetches the Firefox 135 official build. We ran on an Apple M2 Max, so we pulled the macOS arm64 binaries for each build, signed them ad hoc (the cross-compiled bundles need it), and pointed the same launcher at each one with executable_path.

Two version details matter for fairness. JWriter20’s WebRTC fix targets a regression introduced in Firefox 146, so we did not compare it against the 135 cache. We pulled the official v146-hardware build (Firefox 146.0.1) as the baseline and JWriter20’s own 146.0.1 build as the patched version. Same Firefox, two builds. camoufox-reverse only ships at 135, which is fine because we used it only as a tracer, not as a contender.

Every test drives the binaries the way a real user would, through the camoufox launcher with proxy and geoip set, so the fingerprint database, the locale coherence, and the stealth patches are all active. The one exception is the WebRTC probe, explained below, where the page we run matters.

The WebRTC leak that JWriter20 actually fixes

JWriter20’s headline fix is a closed WebRTC IP leak under a proxy. We checked it on the official 146 build against the JWriter20 146 build, same launcher, same Bright Data proxy, geoip=True. The probe gathers ICE candidates from a STUN server and reports any IP that escapes (webrtc_leak_test.py).

A quick detour on what those candidates are, because the whole leak lives in them. WebRTC connects two peers directly, and to do that each side has to advertise every network address it could be reached on. Each address it offers is an ICE candidate. A candidate is an IP, a port, a protocol, and a type, and it reaches JavaScript as a string like this:

candidate:842163049 1 udp 1677729535 203.0.113.25 54321 typ srflx raddr 192.168.1.45 rport 54321

Two types matter here. A host candidate is an address of a local network interface, so it carries your LAN IP. A srflx (server-reflexive) candidate is the public address a STUN server reports back when the browser asks which IP it appears to come from, so it carries your real WAN IP. A page gathers all of this with no permission. It opens an RTCPeerConnection pointed at a STUN server, calls setLocalDescription, and reads each candidate as it arrives. The key is that STUN runs over UDP, and an HTTP proxy only tunnels TCP. The STUN request leaves from the real interface, the proxy never sees it, and the srflx candidate comes back with the real WAN IP even though every HTTP request went through the proxy.

The first version of our probe ran the RTCPeerConnection on about:blank and showed both builds leaking the real IP. That was our mistake, not a result. Camoufox’s content-level injection is not active on about:blank, so we were measuring an unprotected page. Moving the probe onto a real https origin changed everything:

official-146   HTTP exit IP (proxy): 189.173.138.17
               ICE candidates: 1  [srflx] ips=['203.0.113.25']   <- real WAN IP leaks

jwriter20-146  HTTP exit IP (proxy): 93.44.185.102
               ICE candidates: 0                                <- nothing leaks

Our real WAN IP is 203.0.113.25. The official build, behind a working proxy, still hands it to any page through the WebRTC reflexive candidate. The proxy exit IP rotates on each run, so the constant 203.0.113.25 in the candidate is unmistakably the real address, not the proxy.

The fix is real, and it is baked into the binary.

We unzipped both camoufox.cfg files to confirm. The official build sets only media.peerconnection.ice.no_host. JWriter20 adds default_address_only, proxy_only_if_behind_proxy, proxy_only_if_pbmode, and obfuscate_host_addresses. Behind a proxy that cannot carry UDP, those preferences make WebRTC gather no candidates at all, so there is nothing to leak. Reproduced across two runs.

If WebRTC leaks were your problem, JWriter20 solves them. Hold that thought, because it does not end where you would expect.

The canvas patch that we could read but not run

To see why this patch exists, you have to understand the small-arms race it sits within. The PropertyTracer run above caught DataDome calling toDataURL and getImageData. Those two calls are how a canvas fingerprint is taken. A script draws the same text and shapes into an off-screen canvas on every machine, reads the pixels back, and hashes them. The drawing commands are identical everywhere. The pixels are not, because the final image depends on your GPU, your graphics driver, and how your system rasterizes fonts. That hash is stable for your device and different from the next one, which is most of what a tracker wants.

The standard way to hide is to add noise. Camoufox, Brave, Firefox’s resist-fingerprinting mode, and a long tail of extensions all nudge a few pixels so the hash will not stay constant across sites. The weakness is in how that noise is generated. If it is a fixed per-session perturbation that depends only on a seed and the pixel position, it can be undone. A 2025 paper at The Web Conference, Breaking the Shield by Hoang Dai Nguyen and Phani Vadrevu, showed exactly that against eighteen extensions and five browsers. Their Pixel-Recovery attack paints a second canvas filled with a known solid color and reads it back. Because it knows what every pixel should have been, it solves for the perturbation and subtracts it from the real fingerprint canvas. Reload ten times, and the recovered fingerprint stays constant while the noised one keeps changing. That is the proof the noise was reversible all along.

Two changes defeat the attack, and the same paper points at both. Leave the flat regions alone, so a detector that paints a solid block and reads it back finds no tampering to measure. And make each perturbation depend on the pixel content rather than its position, so there is no single value to solve for and subtract. The second idea is what Brave’s Farbling does, deriving its noise from the canvas content so two different canvases are altered differently, and it is the one defense the Pixel-Recovery attack could not reverse.

LeooNic’s patch implements both moves, and it is the most interesting code in the whole fork tree. The rewritten ApplyCanvasNoise skips flat regions and only perturbs edges, and the comments name the attack directly:

// Content-aware + content-dependent canvas noise.
//   - Tier 1 known-pixel checks (DataDome, Castle): flat regions are skipped
//     because flat_score < FLAT_THRESHOLD. fillRect(R,G,B) is undisturbed.
//   - WWW'25 Pixel-Recovery Attack (Nguyen & Vadrevu): noise depends on the
//     pixel content AND its 4 neighbors, not just (seed, index).

FLAT_THRESHOLD is the cutoff that decides what counts as a flat region. The edge pixels that survive it get a content-dependent nudge of plus or minus one, small enough to stay invisible but enough to move the hash. The logic is sound on paper. We wanted to confirm it at runtime.

We built a probe that draws two solid blocks with one sharp boundary and counts perturbed pixels in the flat interior versus the edge (canvas_fingerprint_test.py).

First, we learned that canvas noise is off by default in every current build, which lines up with the CloverLabs “Disable Canvas Noise” commit. The noise only runs when canvas:seed is non-zero. With the seed forced on the official 146 build, the original algorithm shows its tell:

official-146 (original noise)
  interior flat pixels perturbed : 9105 / 18240   (~50%)
  boundary edge pixels perturbed : 98 / 192
  max edge delta (per channel)   : 1
  hash varies across sessions    : True

The stock algorithm perturbs roughly half of every pixel, flat fills included. That is precisely the behavior a known-pixel check catches, and precisely what LeooNic set out to fix. So the official baseline is the “before” picture, captured at runtime.

The “after” picture is what we were not able to collect. LeooNic ships only a Windows binary, so we ran it on a Windows cloud box. It would not launch under Playwright at all. Every attempt, headless or headful, with the stock launcher or LeooNic’s own 0.5.0 launcher installed from source, ended the same way:

console.error: "Warning: unrecognized command line flag" "-juggler-pipe"
Remote Settings startup changesets bundle could not be extracted (JSON.parse...)
JavaScript error: AsyncShutdown.sys.mjs, line 587: uncaught exception: undefined
<process did exit: exitCode=0>

The official 135 build launches and drives fine on the same box, so the machine and Playwright are healthy. The Firefox 149 build that LeooNic publishes aborts at startup before Juggler attaches.

This is not only our environment. LeooNic’s own issue #1 is titled “fix: port patches and build system to Firefox 149.0”, an open work in progress, and daijro carries issues #620 and #572 about Juggler failing to initialize in constrained environments. The build does run through LeooNic’s native RDP path, which is the whole point of their RDPBrowser, but even there we could not activate the canvas seed. The global config is ignored, and the per-context setCanvasSeed function the build exposes only at document start was never present on the page when driven over RDP.

So we report LeooNic honestly. The content-aware algorithm is real and well-reasoned in source, and the original algorithm’s weakness is confirmed at runtime on the official build. The published Firefox 149 binary is not something you can pick up and drive with the standard stack today. For a reader choosing a fork, that is the practical signal. The innovation lives in the code, not yet in a usable artifact you can run.

The block-rate test, and the result we did not expect

Given that, we could only test the jwriter fork compared to the original version. As always, the code that will be used for testing can be found in our GitHub repository reserved for paying users, inside the folder 106.CAMOUFOX.

Read more

Every product has its own price, usually formed by a simple and predictable formula. Proxy services follow a similar pricing logic. At first thought, multiplying the proxy price by the amount of bandwidth should result in a final cost, but in fact, many aspects must be taken into consideration. Bandwidth is not a clean, one-to-one reflection of useful work. What users pay for is not just the data they properly collect but also everything that happens around. And it’s about request failures, encountered blocks, timeouts, and suboptimal routing decisions.

Much of the traffic in scraping systems is consumed without usable output, and that’s where the gap between expected and actual costs starts to widen. To understand where the budget really goes, we need to research it deeper.

What is the math behind the scraping costs

Bandwidth is measurable, and proxy providers typically price it transparently. However, it’s not the key driver of the cost, it’s the result of a much more complex process. Each unit of bandwidth stands for a series of events, rather than merely one successful request. While some requests yield immediate data, many others can fail, be obstructed, time out, or require further attempts. Consequently, the actual cost structure is influenced more by system behavior than by the volume of traffic.

The real cost equals the total number of request cycles necessary to extract data multiplied by the cost of executing each request cycle. In simple words, it is defined by how many complete attempts the system must make before it gets that one response. The key detail is that only successful requests generate value. Each request cycle may have the initial request, retries after timeouts, proxy rotation, session management, and many other factors. All of them consume resources. Because of this, the real price increases not only when proxy prices rise but also when the system becomes less well-functioning. The same dataset can cost remarkably more to extract when the scraping pipeline is inefficient.

The cost amplification effect, known as the cumulative increase in resource usage caused by constant request cycles, is an issue in web scraping systems. One failed request can trigger multiple follow-up attempts. These extra cycles accumulate and raise the cost required to get one successful data point. For this reason, it’s accurate to evaluate scraping by cost per successful request. The fewer attempts needed, the lower the real cost.

Mistake #1 - Using the wrong proxy type

This mistake is not just widespread but can also cost you a lot. Not all targets need the same level of stability and anonymity, but numerous systems use a universal strategy. This results in either high costs or poor operation.

Each proxy type has its own balance of speed, cost, and detection resistance. The system will not work if that balance doesn’t align with the target website’s behavior. For example, mobile IPs are not just more expensive by default. They are highly trusted and harder to get blocked, so it’s logical to use them for challenging targets. When they are used on low-protection websites, they increase costs without improving the results. The approach that works is based on the right matching of proxy type to the task.

• Residential proxies are actively used in web scraping as they route traffic via IPs assigned by ISPs to real devices. By looking like it comes from the real user, these proxies ensure strong trust signals. Many proxy users notice better success rates when they switch to residential IPs.

• Mobile proxies direct traffic via carrier networks using IP addresses from mobile service providers. Since these IPs are shared, the traffic appears very authentic and is significantly more difficult for systems that rely on fingerprinting to identify.

• Datacenter proxies function on cloud or server-based infrastructure and use IP ranges that aren’t tied to real ISPs. The biggest advantage is speed. They are perfect for heavy automation and data collection tasks.

Mistake #2 - The retry loop problem

The goal of retry logic is to improve success rates by giving failed requests another attempt to return a valid request. This approach works when responses are consistent, and failures are occasional, not systematic. On targets with rate limits or unstable responses, many retries can lead to constant failures under the same conditions. Not all failures are the same. If you got a timeout, it’s worth retrying, but if it’s a 403 error or a block, there are other actions to try. For example, you can rotate proxies or fix headers.

Retries can turn into a loop where the system keeps sending more requests but isn’t getting better results. Instead of retrying everything the same way, treat different errors differently, and adjust your behaviour based on the response. You can rotate proxies after getting a certain status code or stop retrying when a request is blocked.

Mistake #3 - Misconfigured proxy rotation

Rotating proxies aggressively is not a solution. Changing IPs too often makes traffic look unnatural and can raise suspicion. On the flip side, not rotating enough can create another issue. Thus, there must be balance. Some websites tolerate frequent IP changes, while others expect a more stable session. Treating all targets the same way is not appropriate. It’s better to adjust rotation based on the context.

If the server expects the same user behavior over time, using sticky sessions may help. In this case, it’ll help maintain session consistency and not break the flow. For bulk data extraction, you can try rotating proxies more frequently. In this situation, there is no need to preserve session context. You can also refine your rotation if you use signal-based triggers instead of fixed rules. Rotate thoughtfully, and when the system detects specific conditions like a sudden drop in success rates or status codes, adapt your proxy to it.

Mistake #4 - Ignoring caching and duplicate requests

A notable portion of scraping traffic is often dedicated to retrieving data that has already been collected. This occurs when pipelines lack deduplication or clear definitions for data freshness. It leads to repeated requests for identical resources. This process consumes bandwidth and proxy capacity without providing new information.

To address this, implement a caching layer and deduplication logic. Responses can be cached based on a time-to-live (TTL) interval that aligns with the frequency of data updates. Request fingerprints can be used to identify duplicates before requests are sent. For structured data, storing IDs or hashes of processed items allows the system to skip previously captured content.

Mistake #5 - No cost-aware proxy routing

Many scraping systems process all requests through a single proxy type. This approach can simplify implementation, but it can still lead to ineffectiveness. Different endpoints often have distinct requirements, and a universal strategy may result in unnecessary costs.

For instance, using proxies with a high trust score for simple endpoints can be expensive, whereas lower-cost proxies used for protected pages may result in blocks and required retries. Without routing logic to adapt to these variables, systems often overpay or underperform. They can’t adapt.

An alternative is to implement cost-aware routing, which matches the proxy type to the difficulty of the task. This involves using more economical options for low-risk requests and escalating to higher-trust proxies only when necessary. By monitoring metrics such as status codes, latency, and success rates, the system can determine when to switch proxy pools. For example, a blocked request can be retried using a higher-trust proxy rather than repeating the request under the same conditions.

This approach creates a more structured pipeline that balances cost and performance by allocating resources based on the specific requirements of each request.

Understanding the real price of proxies

While “price per GB” is often cited as a standard industry metric, experienced engineers understand that it fails to capture the true economic reality of data scraping. In practice, failed requests consume bandwidth and incur costs despite yielding no usable data. These unsuccessful attempts represent a negative return on investment.

Furthermore, the expenses associated with automated retries add another hidden expense. We have to look beyond the basic per-GB rate and adopt the “Cost Per Successful Request” (CPSR). This metric provides a more accurate reflection of true operational expenses.

To calculate the cost of each valid data retrieval, use the following formula:

CPSR = price per GB / 1,000 * 1 / Success Rate

In this equation, the “success rate” is the percentage of requests that return an HTTP 200 OK status along with the intended data. Organizations can make better financial decisions if they start evaluating proxy services through the lens of CPSR.

DataImpulse is a reliable provider of residential, mobile, and datacenter proxies with non-expiring traffic and a pay-as-you-go model, meaning purchased traffic remains available until it is used. This vendor offers more than 90 million IPs in 195 countries. Teams usually choose DataImpulse for web scraping, ad verification, market research, SERP monitoring, and website testing.

Why is DataImpulse cheaper than other vendors?

The pricing structure is based on the proxy sourcing method. Many providers purchase traffic rights from ISPs and resell them, which includes an additional markup. DataImpulse sources IP addresses directly through its own application and SDKs, bypassing intermediaries to avoid extra costs. This operational model complies with all legal standards.

How to reduce your scraping costs by 30-60%

Cost efficiency in data collection is primarily achieved by minimizing inefficient requests and increasing the success rate of each attempt.

1. To optimize expenses, match proxy types to the specific requirements of the task. Using cost-effective proxies for straightforward targets while reserving higher-trust proxies for more challenging endpoints can reduce unnecessary spending.

2. Refining retry logic is also important. Failures should be addressed based on their specific status codes. Avoiding repeat requests under identical conditions prevents the waste of resources.

3. Proxy rotation should be managed strategically rather than randomly. Implementing sticky sessions and rotating based on indicators such as blocks or elevated failure rates can improve both stability and overall success rates.

4. Incorporating caching and deduplication techniques helps manage traffic. By avoiding redundant requests for data that has not changed, it is possible to decrease total request volume.

5. Implement a cost-aware proxy routing strategy. Prioritize lower-cost alternatives, escalating to premium options only when strictly necessary. This approach facilitates a more efficient resource allocation model, ensuring that infrastructure investments are directed toward the areas of greatest impact.

6. Lastly, pay attention to how your scraper interacts with websites. When a browser loads a page, it also pulls images, scripts, videos, and even fonts. Thus, lots of traffic is generated. Use HTTP requests for structured data and browser-based scraping when JS rendering is necessary.

These optimizations don’t require a comprehensive system overhaul. Incremental improvements in request efficiency can harvest significant cost reductions.

Start measuring your current CPSR baseline

At first sight, scraping costs look like a simple equation between proxy price and bandwidth. But the real drivers of cost lie deeper. As we’ve seen, unnecessary retries and poor rotation strategies contribute to a growing gap between expected and actual costs. A system with low success rates will always consume more resources.

The important shift is moving away from thinking in terms of raw pricing and toward thinking in terms of efficiency. It doesn’t always require major steps, simple adjustments are key. Better proxy selection, cost-aware routing, caching, and improved retry logic are among them. From factual proxy usage data from DataImpulse, we’ve seen that even small optimizations can noticeably reduce total costs. Every request should add value, so spending must be thoughtful and deliberate. Audit your scraping pipeline against these 5 mistakes today.

Rust is becoming increasingly popular thanks to its advantages in performance, stability, and security. But what about using it for web scraping?

In this post, I’ll show you what Rust brings to the table for web scraping, why it makes sense (and when it doesn’t), and how to build a web scraper in Rust.

Main Characteristics of Rust: Quick Overview

Rust stands out because it combines performance, safety, and control in a way few programming languages do. According to the 2025 Stack Overflow Developer Survey, 14.8% of respondents reported using Rust that year, making it the 14th most popular option.

Personally, what I find most compelling about Rust is its memory safety model. Thanks to ownership and borrowing, it avoids entire classes of bugs like memory leaks or race conditions. All of that, without needing a garbage collector!

Before proceeding, let me thank NetNut, the platinum partner of the month. Their set of solutions cover all your needs for scraping.

Visit Netnut

---

Here’s what Rust looks like in its simplest form:

fn main() {
    println!("Hello, world!");
}

Even in this minimal example, you can see Rust’s explicit structure and compile-time guarantees.

Remember: In Rust, println! is not a function. It’s a macro. The ! tells the compiler: “this is a macro invocation, not a normal function call.”

Performance is another big win. Rust is compiled and extremely fast, making it ideal for high-throughput parsing on heavy HTML pages or large volumes of pages (e.g., in an offline web scraping scenario). Concurrency is also first-class, helping you manage thousands of requests in parallel without the usual headaches.

On the flip side, Rust has a steeper learning curve. If you’re coming from Python or JavaScript, getting used to the syntax and strict compiler won’t be trivial. In my experience, the first steps can feel a bit unforgiving…

Why AI Has Made Rust a Solid Choice for Web Scraping

AI is changing the nature of software development, including web development. And, as you may already have noticed, not always in a “lighter” direction. Humans struggle to deal with long scripts and source code files, but machines don’t!

Thus, AI tends to produce very long and complex HTML with a lot of elements embedded in the same page. On top of that, AI makes it trivial to generate large amounts of content, which further increases HTML size. In addition, semantic HTML is more verbose than traditional HTML.

As a result, modern web pages are getting bigger and more complex. From a scraping perspective, this translates into slower and more resource-intensive parsing. What used to be lightweight DOM trees are now dense, deeply nested structures that require more CPU and memory to process.

This is exactly where Rust starts to make sense…

Sure, it may not be the easiest programming language, but Rust’s performance makes it compelling (and in some cases even necessary). Its low-level control and zero-cost abstractions allow Rust HTML parsers to process large documents in fractions of a second, even under high concurrency.

Independent benchmarks show Rust HTML parsers ranking among the fastest available. In particular, libraries like tl stand out for their exceptional speed and low overhead.

Start your scraping journey with Byteful: 10GB New Customer Trial | Use TWSC for 15% OFF | $1.75/GB Residential Data | ISP Proxies in 15+ Countries

Claim your 10GB here

Best Rust Web Scraping Libraries

How to Build a Scraper in Rust: A Step-by-Step Guide

In this section, I’ll guide you through the process of building a web scraper in Rust. The target web page will be Books to Scrape’s homepage. This is a static page, which is the ideal scenario for high-speed HTML parsing in Rust.

The end goal is to scrape all the book information and export it to a CSV file. Follow the instructions below!

Prerequisites

Make sure you have:

• Rust installed locally (the article refers to Rust 1.95.0).

• Some basic familiarity with Rust syntax and constructs.

Trusted by teams running ad verification, web scraping, SERP tracking, and market research. Ethically sourced proxies, globally accessible, and fairly priced.

Get Started With DataImpulse

Step #1: Set Up a Rust Scraping Project

Create a new Rust project for web scraping with:

cargo new books_rust_scraper

This will generate a new project called books_rust_scraper containing a basic “Hello, world!” program. Move into the project folder:

cd books_rust_scraper

You should now see the following file structure:

books_rust_scraper/
├── src/
│   └── main.rs
├── target/
├── .gitignore
├── Cargo.toml
└── Cargo.lock

Focus on the src/main.rs file:

This is the entry point of your application and currently contains a simple “Hello, world!” example. Test your Rust application with:

cargo run

The command executes the src/main.rs file, so the result will be:

Hello, world!

In that file, you’ll implement your Rust web scraping logic. Great!

Step #2: Install Required Dependencies

Run these commands to install the crates (Rust libraries) needed to build a Rust web scraper:

cargo add tokio --features full
cargo add reqwest
cargo add scraper
cargo add csv

These are the core dependencies:

• tokio: Enables asynchronous execution.

• reqwest: To send HTTP requests to retrieve HTML pages.

• scraper: To parse HTML and extract data using CSS selectors.

• csv: To export the scraped data to a CSV file.

After running the commands above, your Cargo.toml file should look similar to this:

[package]
name = "books_rust_scraper"
version = "0.1.0"
edition = "2024"

[dependencies]
csv = "1.4.0"
reqwest = "0.13.3"
scraper = "0.26.0"
tl = "0.7.8"
tokio = { version = "1.52.1", features = ["full"] }

Nice! You now have all the dependencies in place to start building your Rust scraper.

Step #3: Retrieve the Target Page

Use reqwest to fetch the target page with:

use std::error::Error;
use reqwest::Client;

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
    // Initialize the HTTP client
    let client = Client::builder()
        .build()?;

    // Retrieve the target page
    let url = "https://books.toscrape.com/";
    let response = client
    .get(url)
    .send()
    .await?;

    // Extract the HTML content from the response
    let html = response.text().await?;
  
    // Parsing logic...

    // Data export logic...

    Ok(())
}

This snippet initializes an asynchronous HTTP client using Tokio, sends a GET request to the target URL, retrieves the HTML response body, and prepares it for parsing and data extraction.

If you print html, you’ll observe:

Excellent! Get ready to apply the Rust data parsing logic.

Step #4: Implement the Parsing Logic

Before implementing the web scraping logic in Rust, study the DOM of the target page. Inspect a book HTML element in the browser:

From this structure, notice how you can select all books using the article.product_pod CSS selector. For each book element, you can retrieve:

• The title and URL from h3 a.

• The image URL from .image_container img.

• The price from .price_color.

• The rating from p.star-rating.

• The stock status from .instock.availability.

First, define a struct to store that data:

#[derive(Debug)]
struct Book {
    url: String,
    image_url: String,
    title: String,
    price: String,
    rating: String,
    in_stock: bool,
}

Next, define the parse_books() function that extracts and structures the data:

use scraper::{Html, Selector};
// ...

// ...
fn parse_books(html: &str) -> Result<Vec<Book>, Box<dyn Error>> {
    // Parse the HTML content
    let document = Html::parse_document(html);

    // Define CSS selectors for the HTML elements of interest
    let book_selector = Selector::parse("article.product_pod")?;
    let title_selector = Selector::parse("h3 a")?;
    let image_selector = Selector::parse(".image_container img")?;
    let price_selector = Selector::parse(".price_color")?;
    let rating_selector = Selector::parse("p.star-rating")?;
    let stock_selector = Selector::parse(".instock.availability")?;

    // Where to store the scraped data
    let mut books = Vec::new();

    // Iterate over each book element and extract the relevant data
    for book_el in document.select(&book_selector) {
        // Apply the parsing logic
        let title_el = book_el.select(&title_selector).next().unwrap();

        let relative_url = title_el.value().attr("href").unwrap_or("");
        let url = format!(
            "https://books.toscrape.com/catalogue/{}",
            relative_url
        );

        let image_url = book_el
            .select(&image_selector)
            .next()
            .and_then(|img| img.value().attr("src"))
            .unwrap_or("")
            .to_string();

        let image_url = format!(
            "https://books.toscrape.com/{}",
            image_url.trim_start_matches('/')
        );

        let title = title_el
            .value()
            .attr("title")
            .unwrap_or("")
            .to_string();

        let price = book_el
            .select(&price_selector)
            .next()
            .map(|e| e.text().collect::<String>())
            .unwrap_or_default();

        let rating = book_el
            .select(&rating_selector)
            .next()
            .and_then(|e| e.value().attr("class"))
            .unwrap_or("no rating")
            .replace("star-rating", "")
            .trim()
            .to_lowercase();

        let in_stock = book_el
            .select(&stock_selector)
            .next()
            .map(|e| {
                let text = e.text().collect::<String>();
                text.to_lowercase() == "in stock"
            })
            .unwrap_or(false);

        // Collect the scraped book data
        books.push(Book {
            title,
            price,
            rating,
            in_stock,
            image_url,
            url,
        });
    }

    Ok(books)
}

This function parses raw HTML into structured data using the scraper crate. Html::parse_document() creates a DOM-like representation of the page, while Selector::parse() defines CSS selectors for targeting elements.

document.select(&book_selector) iterates over each book container. Inside each element, .select() extracts nested elements, while .value().attr() retrieves attributes such as links and titles. The .text() method collects visible text content.

Finally, all extracted values are assembled into a Book struct, and each instance is stored in a vector for later export or processing.

Check the TWSC YouTube Channel

Step #5: Export the Scraped Data

Right now, the scraped data is returned by the parse_books() function as a vector of Book structs. Next, add a function that uses the csv crate to export that data into a CSV file:

use csv::Writer;
// ...

//...
fn write_csv(books: &[Book], file_path: &str) -> Result<(), Box<dyn std::error::Error>> {
    let mut wtr = Writer::from_path(file_path)?;

    // Write the header row
    wtr.write_record(&[
        "url",
        "image_url",
        "title",
        "price",
        "rating",
        "in_stock",
    ])?;

    for book in books {
        wtr.write_record(&[
            &book.url,
            &book.image_url,
            &book.title,
            &book.price,
            &book.rating,
            &book.in_stock.to_string(),
        ])?;
    }

    wtr.flush()?;
    Ok(())
}

Step #6: Put It All Together

This is the final code of your Rust web scraper:

// src/main.rs

use std::error::Error;
use reqwest::Client;
use scraper::{Html, Selector};
use csv::Writer;

#[derive(Debug)]
struct Book {
    url: String,
    image_url: String,
    title: String,
    price: String,
    rating: String,
    in_stock: bool,
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
    // Initialize the HTTP client
    let client = Client::builder()
        .build()?;

    // Retrieve the target page
    let url = "https://books.toscrape.com/";
    let response = client
    .get(url)
    .send()
    .await?;

    // Extract the HTML content from the response
    let html = response.text().await?;

    // Parse the books data from the HTML
    let books = parse_books(&html)?;

    // Export the scraped data to a CSV file
    write_csv(&books, "books.csv")?;

    Ok(())
}

fn parse_books(html: &str) -> Result<Vec<Book>, Box<dyn Error>> {
    // Parse the HTML content
    let document = Html::parse_document(html);

    // Define CSS selectors for the HTML elements of interest
    let book_selector = Selector::parse("article.product_pod")?;
    let title_selector = Selector::parse("h3 a")?;
    let image_selector = Selector::parse(".image_container img")?;
    let price_selector = Selector::parse(".price_color")?;
    let rating_selector = Selector::parse("p.star-rating")?;
    let stock_selector = Selector::parse(".instock.availability")?;

    // Where to store the scraped data
    let mut books = Vec::new();

    // Iterate over each book element and extract the relevant data
    for book_el in document.select(&book_selector) {
        // Apply the parsing logic
        let title_el = book_el.select(&title_selector).next().unwrap();

        let relative_url = title_el.value().attr("href").unwrap_or("");
        let url = format!(
            "https://books.toscrape.com/catalogue/{}",
            relative_url
        );

        let image_url = book_el
            .select(&image_selector)
            .next()
            .and_then(|img| img.value().attr("src"))
            .unwrap_or("")
            .to_string();

        let image_url = format!(
            "https://books.toscrape.com/{}",
            image_url.trim_start_matches('/')
        );

        let title = title_el
            .value()
            .attr("title")
            .unwrap_or("")
            .to_string();

        let price = book_el
            .select(&price_selector)
            .next()
            .map(|e| e.text().collect::<String>())
            .unwrap_or_default();

        let rating = book_el
            .select(&rating_selector)
            .next()
            .and_then(|e| e.value().attr("class"))
            .unwrap_or("no rating")
            .replace("star-rating", "")
            .trim()
            .to_lowercase();

        let in_stock = book_el
            .select(&stock_selector)
            .next()
            .map(|e| {
                let text = e.text().collect::<String>();
                text.to_lowercase() == "in stock"
            })
            .unwrap_or(false);

        // Collect the scraped book data
        books.push(Book {
            title,
            price,
            rating,
            in_stock,
            image_url,
            url,
        });
    }

    Ok(books)
}

fn write_csv(books: &[Book], file_path: &str) -> Result<(), Box<dyn std::error::Error>> {
    let mut wtr = Writer::from_path(file_path)?;

    // Write the header row
    wtr.write_record(&[
        "url",
        "image_url",
        "title",
        "price",
        "rating",
        "in_stock",
    ])?;

    for book in books {
        wtr.write_record(&[
            &book.url,
            &book.image_url,
            &book.title,
            &book.price,
            &book.rating,
            &book.in_stock.to_string(),
        ])?;
    }

    wtr.flush()?;
    Ok(())
}

Note how all previously defined functions are now called inside main(). Et voila! In just around 150 lines of code, you’ve built an efficient web scraper in Rust.

Run your scraper with:

cargo run

After execution, a books.csv file will be created in your project folder. Open it, and you will see:

This matches exactly the data shown on the target website, but now in a structured format. Mission complete!

Browser Automation in Rust: Does It Make Sense?

First of all, it’s worth noting that the ecosystem for browser automation in Rust is quite small compared to JavaScript or Python. Also, most libraries aren’t official, but rather community-backed ports like Playwright Rust or the Selenium bindings.

Now, from a technical standpoint, browser automation happens inside the browser itself. So, Chrome, Chromium, or Firefox do most of the heavy lifting. What you define through the library’s API simply orchestrates operations like clicking, waiting for elements, and extracting data. These commands are then translated into browser actions via WebDriver, CDP, or WebDriver BiDi.

Because of that, using a systems-level language like Rust can be more of a burden than an advantage. The main strength of Rust (i.e., raw performance) doesn’t really matter here, since the controlled browser instances are the actual bottleneck, not your automation code.

That means we lose Rust’s biggest advantage while still paying its costs. On top of that, Rust’s strict compiler and steeper learning curve can slow down development speed.

To be honest, I see Rust as excellent for the parsing and data processing layer, but I wouldn’t recommend it for browser automation…

Need help with your scraping project?

Rust for Web Scraping: Final Comment

If I had to summarize my experience with Rust for web scraping, I’d say this: it really shines when you’re parsing large HTML pages at scale or handling a high number of parsing tasks in parallel.

In those scenarios, the combination of performance, memory safety, and concurrency makes a real difference. That said, I wouldn’t recommend Rust for everyday scraping tasks… The entry barrier is just too high, the learning curve too steep, and the ecosystem around scraping too small.

On top of that, finding experienced Rust developers specifically focused on web scraping, or even just translating those skills into job opportunities, can be way more challenging than in more mainstream stacks.

So my take is pretty simple: consider Rust when performance and scale truly matter. For everything else, prefer Python or JavaScript.

👍 Pros:

• Can efficiently handle thousands of requests in parallel.

• Rust HTML parsers are extremely.

• Strict compiler checks and static guarantees lead to stable scraping pipelines.

👎 Cons:

• Slower development and prototyping speed.

• Smaller ecosystem of scraping libraries compared to Python or JavaScript.

• Not a practical choice for browser automation.

Conclusion

Here, I’ve guided you through the world of web scraping in Rust. In a world dominated by AI slops, security flaws, and neglected best practices, this programming language is gaining traction thanks to its focus on efficiency and strict compilation.

As you’ve seen, Rust is excellent for CPU-intensive or memory-intensive tasks like HTML parsing and data processing. Still, it might not be ideal for browser automation or quick prototyping. You also learned how to go from zero to scraped data in CSV format by building a Rust scraper.

I hope you found this helpful and insightful. If you have any questions, feel free to share them in the comments below!

Before proceeding, let me thank NetNut, the platinum partner of the month. Their set of solutions cover all your needs for scraping.

Visit Netnut

What the 2023 LLM hype promised, and what 2026 actually shipped

The product category that took that promise most literally is the agentic browser. A real Chromium running under an LLM that reads the page, decides what to do, and clicks. Browserbase, Hyperbrowser, Skyvern, Browser Use, BrowserOS, Owl Browser, and a long tail of proxy companies rebranding their scraping browsers as “AI-powered”. The pitch in 2024 and 2025 never changed. You hand the agent a task in plain language, and the model handles the rest, including whatever defensive challenge the site throws back.

It is 2026 now, and that prediction has not played out. CAPTCHAs are still in your pipeline. So we went looking for the answer. Can an LLM actually solve a production-grade CAPTCHA like hCaptcha? We read the code where we could, checked the default configs and the docs, and probed the public surfaces of the solver services these products lean on. The picture is consistent, and it is not the one the marketing sells.

Every major agentic browser ships a CAPTCHA-solving bullet point on its site. Open the code of the open-source agents, though, and you find a different story. Almost none of them actually use an LLM to solve a CAPTCHA. They either refuse to try, or they try and fail.

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

One thing to get straight before we open any code. The two families of CAPTCHA behave very differently, and the hype lands on them unevenly. The invisible ones, reCAPTCHA v3 and Cloudflare Turnstile, score your session in the background and rarely show a puzzle. A stealth-first browser on clean residential proxies usually walks past them without anyone seeing a challenge, which we covered for Turnstile in Cloudflare Turnstile: what is that and how it works? and THE LAB #73: How to Bypass Cloudflare in 2025. The visible image challenges, hCaptcha and reCAPTCHA v2, actually demand an answer. We went deep on reCAPTCHA v2 in Bypassing reCAPTCHAs With Open Source and Commercial Tools - Part 2. The one that matters in 2026 is hCaptcha, and that is where this article lives, because it is where most scrapers break.

So here is the question that drives everything below. When an agentic browser says it “solves” an hCaptcha, what does its code actually do?

Tool landscape

What’s the commercial offer today

There are four families of strategy in the commercial set. In none of them is the LLM in the agent layer doing the CAPTCHA work.

The first family is stealth-first. Residential proxies, fingerprint shaping, and request patterning lower the bot score so the visible challenge never triggers. The CAPTCHA is not solved. It is prevented from appearing. That gives you the cleanest legal posture in the set, because no automated solving is happening. ZenRows is the example.

The second family is the opposite of stealth. Instead of hiding that the traffic is automated, the vendor declares it openly and relies on a business arrangement with the CAPTCHA providers to be let through. Browserbase is explicit about this. Its Stealth Mode documentation says “through Browserbase’s partnerships with CAPTCHA providers, Browserbase can resolve challenges automatically so your sessions continue without interruption”, with solving “enabled by default for all sessions”. This is the verified-bot path, the same idea behind Cloudflare’s Web Bot Auth. A declared, allowlisted agent rather than a disguised one. No model solves a puzzle on either side. The provider recognizes the partner and waves it through. For the common challenge types, according to their documentation, this works without a fight.

The third family pairs a proprietary solver with documented third-party integrations. The vendor ships its own solving for some challenge types. For the rest, it documents how to wire in an external solver service. The external solver watches for the challenge and returns the response token through its extension or REST API. The agent then submits. Hyperbrowser and Skyvern Cloud both present this shape, a native or closed-source component plus a documented third-party path. Hyperbrowser advertises “Native Cloudflare Turnstile & CAPTCHA Solving” with “No external plugins” in its post on native CAPTCHA solving, then points to an external solver for the challenge types the native one does not cover.

Start your scraping journey with Byteful: 10GB New Customer Trial | Use TWSC for 15% OFF | $1.75/GB Residential Data | ISP Proxies in 15+ Countries

Claim your 10GB here

The fourth family runs an in-house solver alongside the agent. Could be a vision LLM, could be classical computer vision, could be something else. No third party is in the critical path. The vendor owns the whole stack. Bright Data, Oxylabs, and Owl Browser sit here. Owl Browser puts hard numbers on the claim: “detect and automatically solve reCAPTCHA v2 (1.2s), hCaptcha (0.8s), Turnstile (0.3s), and image CAPTCHAs.” Bright Data sells “AI-based unlocking logic” that handles “CAPTCHA solving, fingerprinting, retries, best headers, location and more” on its Web Unlocker page.

The table below maps each vendor to the mechanism its public documentation surfaces, with the source page that proves it.

Check the TWSC YouTube Channel

The open-source agents we can read

For the proprietary CAPTCHA-solving strategies we cannot see how AI is used. For the open-source ones we can. So we opened their code and read exactly how they handle CAPTCHAs. Three projects matter here.

browser-use is the most popular open-source LLM agent framework for browser automation. MIT-licensed, vendor-agnostic on the LLM side. The repo contains no LLM-driven CAPTCHA-solving logic. The one CAPTCHA-related file it ships, captcha_watchdog.py, does not solve anything either. It waits for a solver running in the BrowserUse cloud proxy and blocks the agent loop until that solver reports back. Run the library locally with your own model and no cloud proxy, and the watchdog has nothing to wait for. That makes browser-use the cleanest test of the claim that a local LLM agent solves the CAPTCHA by reading the page.

Skyvern OSS is the open core of the Skyvern Cloud product. AGPL-3.0, focused on form-filling and structured workflows, written in Python.

BrowserOS is a YC S24 open-source agent-driven browser. AGPL-3.0, 11k stars on GitHub, active development. It pairs a Chromium fork with an integrated agent runtime.

Modeling hCaptcha and reading what the open-source agents actually do

Before testing, it helps to model the target. hCaptcha embeds a widget on the host page through a script served from hcaptcha.com.

The widget renders inside an iframe whose origin is hcaptcha.com, cross-origin to the host page. When the user clicks the “I am human” checkbox, the widget decides whether to issue a challenge. If it does, a second iframe opens with the puzzle dialog. The puzzle layout varies between runs (3x3 grid, 4x3 grid, area-select with a single click on an image, bounding-box, multiple-choice prompt). When the puzzle is solved, the widget writes a response token to a hidden textarea[name="h-captcha-response"] on the host page. The host form reads that textarea on submit and posts the token along with the rest of the data. The whole solve interaction happens inside a frame the host page cannot script. The same-origin policy boundary blocks it.

That last detail decides what works and what does not. An agent driving the host page through Playwright or CDP has full control over the outer page. Inside the hcaptcha.com frame, its control is limited. A browser extension runs with cross-origin privileges. It can both observe and click inside the widget frame. That asymmetry explains most of what follows.

We started by reading three open-source agent repos to see what each one had decided to do at this boundary.

Skyvern OSS bails to the human

Skyvern’s README is candid about the scope of the OSS release: “All of the core logic powering Skyvern is available in this open source repository licensed under the AGPL-3.0 License, with the exception of anti-bot measures available in our managed cloud offering.” That single sentence puts the CAPTCHA section of every Skyvern Cloud feature page outside the repo you can read.

The repo confirms the framing. In agent_functions.py, lines 800-804, the auto_solve_captchas helper returns False unconditionally. In handler.py, lines 1149-1162, handle_solve_captcha_action does exactly one thing of substance.

async def handle_solve_captcha_action(
    action: actions.SolveCaptchaAction,
    page: Page,
    scraped_page: ScrapedPage,
    task: Task,
    step: Step,
) -> list[ActionResult]:
    LOG.warning(
        "Please solve the captcha on the page, you have 30 seconds",
        action=action,
    )
    await asyncio.sleep(30)
    return [ActionSuccess()]

Thirty seconds of asyncio.sleep and a log message asking a human to handle it. Then a success result, regardless of what the human actually did. The script-generation path in skyvern_page.py, lines 1252-1254 is even more direct. solve_captcha raises NotImplementedError. A Skyvern user opened issue #1117 asking how CAPTCHA solving was meant to work in OSS. A maintainer answered plainly: “We haven’t open sourced anything related to our captcha solver / anti-bot measures. We don’t want people abusing these things, so they must remain closed source unfortunately.”

The Skyvern OSS code does not pretend to solve CAPTCHAs. Whatever Skyvern Cloud does on top of this, the OSS release hands the problem to a human and moves on.

As always, the code that will use to bypass hCaptcha can be found in our GitHub repository reserved to paying users, inside the folder 105.HCAPTCHA.

Read more

Well, not so fast.

Open data does not automatically mean free to use, free to redistribute, or free from privacy obligations. And scraping it without thinking through the implications can land you in legal trouble, get your IP banned from a public infrastructure that was never designed to handle aggressive crawlers, or cause you to expose people’s personal information.

In this article, we will go through a complete picture of the “open data” world: what the problem actually is, how to approach it correctly, and how to implement responsible open data scrapers in Python.

Let’s dive into it!

Before proceeding, let me thank NetNut, the platinum partner of the month. Their set of solutions cover all your needs for scraping.

Visit Netnut

What “Open Data” Actually Means Legally, Ethically, and Practically

“Open” is one of the most overloaded words in the data world. Depending on the license, the jurisdiction, and the type of data involved, the same publicly accessible dataset can be freely redistributable, commercially restricted, privacy-sensitive, or legally off-limits entirely.

So, before anything else, let’s establish what you are actually dealing with.

What “Open-Source Dataset” Actually Means (and What It Doesn’t)

Where a dataset sits on the licensing spectrum determines everything: whether you can redistribute it, whether you can use it commercially, and whether collecting it at all exposes you to liability. Here is how the spectrum breaks down:

• CC0 (Creative Commons Zero): Essentially, it is a public domain dedication. The author waives all rights. You can scrape it, redistribute it, use it commercially, and modify it.

• CC-BY (Creative Commons Attribution): It requires you to credit the original source. This means you must clearly state where the data came from, who created it, and link back to the original when you publish or redistribute it. This is the most permissive license after CC0, and it is generally easy to comply with.

• CC-BY-SA (Share-Alike): This carries the same attribution requirement as CC-BY, but adds a condition: any derivative work you publish must carry the same license. In practice, this means you cannot fold a CC-BY-SA dataset into a proprietary product and lock it down.

• CC-BY-NC (Non-Commercial): It also requires attribution, but restricts commercial use entirely. You can use the data for research, journalism, or personal projects, but the moment money is involved, you need a separate agreement with the data owner.

• ODbL (Open Database License), used by OpenStreetMap: It requires both attribution and share-alike, specifically for databases. It is worth noting that ODbL distinguishes between the database itself and the contents. Basically, you can use individual facts freely, but any public use of the database as a whole must comply with the license terms.

And then there is the grey zone, which is where most scraping engineers actually operate: data that is publicly accessible but carries no explicit license. Common cases are government portals, academic repositories, open court records, and municipal datasets. This is a huge portion of what people call “open data”. And here is the thing that matters for scraping professionals: no license does not mean free to use. In most jurisdictions, the absence of a license means the default copyright law applies. Which means the creator reserves all rights.

So before you write a single line of scraper code, the first question is not “Can I access this?” but “Under what terms am I allowed to use what I access?”

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

Where the Ethical (and Legal) Risks Hide

Once you have cleared the license question, there are still several risk areas that are easy to overlook:

• License violations: This is the most obvious one. If a dataset requires attribution and you redistribute it without crediting the source, you are in breach. If it has a non-commercial clause and you use it in a commercial product, it’s the same story. These are the kind of things that generate cease-and-desist letters.

• PII embedded in “open” datasets: This is a subtler and arguably more dangerous problem than license violation. Consider open court records: they are public by design, but they contain names, addresses, and sometimes sensitive personal details. Census microdata, even when anonymized at the aggregate level, can contain individual-level records. For example, the GitHub commit history is public, but it contains email addresses, which is personal data. So, the fact that data was made public by someone else does not strip it of its privacy implications when you collect, aggregate, and store it.

• Jurisdictional complexity: A dataset hosted on a European government portal carries GDPR obligations even if you are scraping it from the United States. The GDPR applies based on where the data subjects are located, not where the scraper is running. If you are collecting data about EU residents, you are in GDPR territory regardless of your own geography.

• The aggregation problem: This is probably one of the most underappreciated risks in the scraping industry. Individually, a dataset of names, a dataset of addresses, and a dataset of employment records might each be harmless and openly licensed. But combine them, and you have created a detailed profile of real people. This is something that privacy regulations were specifically designed to prevent.

The Infrastructure Problem: Open Data Portals Are Not Built for Scrapers

Many scraping engineers come to open data with habits built on commercial targets. That experience can be misleading, because the infrastructure behind open data portals is completely different.

When you scrape a large e-commerce website or a major social media platform, you are hitting servers that are engineered to handle millions of requests per day, backed by CDNs, load balancers, and dedicated anti-bot teams. In other words, they can take a (hard) hit.

On the other hand, a municipal open data portal, a university’s research repository, or a small NGO’s dataset hosting is an entirely different story. This means that a scraper that would barely register as noise on Amazon’s servers could genuinely degrade performance for a public data portal serving thousands of researchers.

This is why scraping open data portals aggressively is arguably more unethical than doing the same to a commercial target. You are not fighting a corporation’s anti-bot system. You are potentially taking down a public resource that other people depend on.

A Four-Step Framework for Scraping Open Datasets Without Breaking Rules or Infrastructure

Every risk outlined above has a straightforward mitigation, but only if you apply it at the right point in your workflow. The mistake most scraping engineers make is treating these as afterthoughts: checking the license after the scraper is already built, thinking about PII after the data is already stored. Let’s discuss a framework that inverts this.

License-First Workflow: Read Before You Scrape

The fix for the license problem is simple in principle, even if it requires discipline in practice: make license verification the first step of your workflow.

Most well-maintained open data portals provide license information in one of these three places: a LICENSE file in the dataset’s root directory, a metadata field in the dataset’s API response, or the dataset’s documentation page. Here is a quick reference for what the licenses described above mean for your use case:

When there is no license, the safe default is not to scrape and redistribute without seeking explicit permission from the dataset owner. A short email asking for clarification is a sign of professionalism.

Check the TWSC YouTube Channel

Prefer APIs and Bulk Downloads Over Scraping

This is a rule that experienced scraping engineers sometimes forget because they are so used to reaching for their scraper toolkit: always check for an official API or bulk download endpoint before writing a scraper.

Most serious open data portals expose REST APIs or provide direct bulk download links. Using these is better in every dimension: it is faster, more reliable, more respectful of the server, and often gives you cleaner, structured data than you would get from parsing HTML.

Your workflow should be:

1. Check the portal’s documentation for an API.

2. Check for a Sitemap or structured data endpoint (as discussed in our article on robots.txt and its implications).

3. Check for bulk download links (CSV, JSON, Parquet).

4. Only fall back to HTML scraping if none of the above exist.

Scraping should be your last resort, not your first instinct.

Responsible Scraping Behavior for Open Infrastructure

When scraping is genuinely the only option, the rules of polite scraping apply. But in the case of open data portals, you should apply a higher standard than you would on a commercial target.

As covered in “best practices for ethical web scraping”, respecting rate limits, introducing delays between requests, and using a descriptive User-Agent are baseline requirements. But for open data portals, you should go further because of their weaker infrastructure. Below are additional rules you should consider:

• Respect Crawl-delay strictly: Even if major crawlers ignore it, on underfunded infrastructure, that directive is a good signal about server capacity.

• Cache responses locally: If you need to re-run your scraper for testing or debugging, you should not be hitting the server again. Cache what you have already fetched.

• Scrape during off-peak hours: For public portals serving researchers and government users, off-peak typically means nights and weekends in the portal’s local timezone.

• Scrape only what you need: This sounds obvious, but it’s easy to over-collect data “just in case”. However, for open portals, remember that every unnecessary request is a cost imposed on a public resource that stays online on an underfunded infrastructure.

Handling PII in Open Datasets

PII stands for Personally Identifiable Information. This refers to any data that can be used, alone or in combination with other data, to identify a specific individual. Think names, email addresses, phone numbers, but also subtler things like IP addresses or device IDs.

The reality is that most well-maintained open data portals go through a review process before publication, and raw PII in open datasets is not as common as you might think. The most common cases where PII can slip through are quite specific: older government datasets published before modern privacy review processes, improperly anonymized academic research deposits, or crowdsourced datasets where contributors included personal details voluntarily.

In such specific cases, the real risk for most scraping engineers is at the aggregation level. A dataset of names, a dataset of ZIP codes, and a dataset of employment records might each be perfectly clean and openly licensed in isolation. But combine them, and you have built a detailed profile of real individuals. This is something that privacy regulations like the GDPR and CPRA were specifically designed to prevent. And once you collect, store, and process that combined data, you become responsible for it, regardless of where it originally came from.

The key principle remains the usual one: identify and handle PII at collection time. Here is a schema you can use to audit the fields that are likely to contain PII:

• Direct identifiers: names, email addresses, phone numbers, national ID numbers, passport numbers, and social security numbers. These are the clearest cases as they point to a specific individual on their own, without needing to be combined with anything else. If you see these fields in a dataset, there is no ambiguity: you are dealing with PII.

• Quasi-identifiers: dates of birth, ZIP codes, job titles, gender, ethnicity, and salary ranges. None of these identify a person on their own, but they become dangerous in combination. A classic example is aggregating just three fields—say date of birth, gender, and ZIP code: this is enough to uniquely identify a great portion of the population in a country.

• Sensitive categories under GDPR: health and medical data, political opinions, religious or philosophical beliefs, biometric data, genetic data, sexual orientation, and trade union membership. This is a legally distinct class that carries stricter obligations regardless of context. In other words, you cannot process this data based on legitimate interest alone. You need explicit consent or another specific legal basis, and the bar is significantly higher than for ordinary PII.

For each PII field, decide upfront: do you need it? If not, drop it at collection time. If you do need it, apply pseudonymization (replacing the identifier with a reversible token) or anonymization (irreversible removal or generalization) before storage.

Need help with your scraping project?

Python Implementation: Putting the Full Responsible Scraping Pipeline Into Code

Principles are only useful if they translate into implementation. Below are two concrete components you can adapt for your own pipelines:

• Checking a dataset’s license before downloading anything, using CKAN’s metadata API, with a practical fallback strategy for portals that don’t run CKAN.

• Running PII detection at collection time, using field-level schema classification, with an honest discussion of where that approach has limits.

Note that the examples below omit an API-first fetch pattern and a polite scraper skeleton, even though they are covered in the framework section above. This is because those are problems with well-known, straightforward solutions that every scraping engineer should be aware of. The idea of the following sections is to provide you with lesser-known solutions, to help you get ideas to apply to your pipelines.

Checking a Dataset’s License Programmatically

Many open data portals are built on CKAN, an open-source data management system used by governments and enterprises. CKAN exposes a REST API that includes license metadata, which makes programmatic license checking straightforward.

Here is how to query a CKAN-based portal and extract license information before proceeding:

import requests

def check_dataset_license(portal_base_url: str, dataset_id: str) -> dict:
    """
    Queries a CKAN portal API to retrieve license information
    for a given dataset before any scraping begins.
    """
    api_url = f"{portal_base_url}/api/3/action/package_show"
    params = {"id": dataset_id}

    response = requests.get(api_url, params=params, timeout=10)
    response.raise_for_status()

    data = response.json()
    result = data.get("result", {})

    license_info = {
        "dataset_name": result.get("title", "Unknown"),
        "license_id": result.get("license_id", "Not specified"),
        "license_title": result.get("license_title", "Not specified"),
        "license_url": result.get("license_url", "Not specified"),
    }

    return license_info

# Example: querying the UK government's open data portal
portal = "<https://data.gov.uk>"
dataset = "road-accidents-safety-data"

license_info = check_dataset_license(portal, dataset)

print(f"Dataset: {license_info['dataset_name']}")
print(f"License: {license_info['license_title']}")
print(f"License ID: {license_info['license_id']}")
print(f"License URL: {license_info['license_url']}")

Which outputs the following:

Dataset: Road Safety Data
License: UK Open Government Licence (OGL)
License ID: uk-ogl
License URL: <https://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/>

With this information in hand, you can make an informed decision before a single byte of dataset content is downloaded. Specifically, you can directly check the government license page. The image below partially shows the license page:

But what if the portal you need to scrape doesn’t run CKAN? Not all open data portals do… Socrata (used by many US city and state governments), DKAN, and custom-built portals each have different or no metadata APIs. In those cases, your fallback options are the following:

• Check for a LICENSE or METADATA file in the dataset’s root directory or bulk download package. Many portals include one.

• Look for a <link rel=”license”> tag in the dataset’s HTML page, which some portals emit as structured metadata.

• Check the portal’s documentation or “About” page, where license terms are often stated globally for all datasets.

If none of the above yield a clear answer, treat the license as unknown and do not redistribute without seeking explicit written permission from the dataset owner. A short email asking for clarification is a professional move.

PII Detection at Scrape Time

In this case, the approach depends heavily on what you actually know about the data you need to scrape. Two situations you will encounter in practice, each calling for a different strategy:

• You know the schema: If you are retrieving structured data, field-level detection is the right approach. You know which fields are likely to carry PII, so you can target them directly. This is faster, more precise, and produces far fewer false positives than running a general NER model over free text.

• You have no schema: For unstructured data, NER-based detection is a reasonable starting point, but go in with realistic expectations. A common solution is using spaCy’s en_core_web_sm, which is a small model trained on news text, so don’t expect it to do miracles for you. Another approach, which can give way better results, is using LLMs to give a structure to unstructured text.

For the structured case, here is a field-level PII detection pipeline:

import re
import hashlib
from dataclasses import dataclass, field
from typing import Any

# Fields that are unambiguously PII on their own
DIRECT_IDENTIFIER_FIELDS = {
    "name", "full_name", "first_name", "last_name",
    "email", "email_address",
    "phone", "phone_number", "mobile",
    "ssn", "national_id", "passport_number",
    "ip_address", "device_id"
}

# Fields that are not PII alone but dangerous in combination
QUASI_IDENTIFIER_FIELDS = {
    "date_of_birth", "dob", "birth_date",
    "zip_code", "postcode", "zip",
    "gender", "sex",
    "job_title", "occupation",
    "salary", "income",
    "ethnicity", "race"
}

# Regex patterns for validating suspected PII values at the content level
EMAIL_PATTERN = re.compile(r"[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\\.[a-zA-Z0-9-.]+")
PHONE_PATTERN = re.compile(r"\\b(\\+?\\d[\\d\\s\\-().]{7,}\\d)\\b")

@dataclass
class FieldAudit:
    field_name: str
    classification: str    # "direct", "quasi", or "clean"
    original_value: Any
    processed_value: Any   # pseudonymized, generalized, or original
    action_taken: str       # "pseudonymized", "generalized", "dropped", "kept"

def pseudonymize(value: Any) -> str:
    """
    Replaces a PII value with a consistent, reversible token.
    Using a hash means the same value always produces the same token,
    which preserves referential integrity across records (e.g., you can
    still count unique users without knowing who they are).
    In production, use an HMAC with a secret key instead of plain SHA-256.
    """
    return hashlib.sha256(str(value).encode()).hexdigest()[:16]

def generalize_date(value: str) -> str:
    """
    Reduces a full date of birth to a birth year only.
    A simple but effective generalization for quasi-identifiers.
    """
    # Handles common formats: YYYY-MM-DD, DD/MM/YYYY, MM/DD/YYYY
    match = re.search(r"\\b(19|20)\\d{2}\\b", str(value))
    return match.group(0) if match else "UNKNOWN_YEAR"

def audit_record(record: dict) -> tuple[dict, list[FieldAudit]]:
    """
    Processes a single structured record field by field.
    Returns a cleaned record and a full audit trail of what was done to each field.

    Strategy:
    - Direct identifiers: pseudonymize (preserve referential integrity)
    - Quasi-identifiers: generalize where possible, pseudonymize otherwise
    - Everything else: pass through unchanged
    """
    clean_record = {}
    audit_trail = []

    for field_name, value in record.items():
        normalized = field_name.lower().strip()

        if normalized in DIRECT_IDENTIFIER_FIELDS:
            processed = pseudonymize(value)
            audit_trail.append(FieldAudit(
                field_name=field_name,
                classification="direct",
                original_value=value,
                processed_value=processed,
                action_taken="pseudonymized"
            ))
            clean_record[field_name] = processed

        elif normalized in QUASI_IDENTIFIER_FIELDS:
            # Apply field-specific generalization where we can
            if normalized in {"date_of_birth", "dob", "birth_date"}:
                processed = generalize_date(value)
                action = "generalized"
            else:
                # For other quasi-identifiers, pseudonymize as a safe default
                processed = pseudonymize(value)
                action = "pseudonymized"

            audit_trail.append(FieldAudit(
                field_name=field_name,
                classification="quasi",
                original_value=value,
                processed_value=processed,
                action_taken=action
            ))
            clean_record[field_name] = processed

        else:
            # Field is not in either PII list — pass through, but still
            # run a regex check on string values as a safety net
            if isinstance(value, str):
                if EMAIL_PATTERN.search(value) or PHONE_PATTERN.search(value):
                    # Unexpected PII in a non-PII field: flag it and pseudonymize
                    processed = pseudonymize(value)
                    audit_trail.append(FieldAudit(
                        field_name=field_name,
                        classification="direct",
                        original_value=value,
                        processed_value=processed,
                        action_taken="pseudonymized (unexpected PII in non-PII field)"
                    ))
                    clean_record[field_name] = processed
                    continue

            audit_trail.append(FieldAudit(
                field_name=field_name,
                classification="clean",
                original_value=value,
                processed_value=value,
                action_taken="kept"
            ))
            clean_record[field_name] = value

    return clean_record, audit_trail

def process_records(records: list[dict]) -> list[dict]:
    """
    Runs field-level PII detection and handling across a list of records.
    Prints an audit summary for any record where PII was found.
    """
    clean_records = []

    for i, record in enumerate(records):
        clean_record, audit_trail = audit_record(record)
        pii_fields = [a for a in audit_trail if a.classification != "clean"]

        if pii_fields:
            print(f"Record {i}: PII detected and handled in {len(pii_fields)} field(s):")
            for audit in pii_fields:
                print(f"  [{audit.classification.upper()}] {audit.field_name} "
                      f"→ {audit.action_taken}")

        clean_records.append(clean_record)

    return clean_records

# Example: a batch of records from a scraped open dataset
records = [
    {
        "record_id": "A001",
        "name": "Jane Doe",
        "date_of_birth": "1985-03-22",
        "zip_code": "SW1A 1AA",
        "incident_type": "Road accident",
        "severity": "Slight"
    },
    {
        "record_id": "A002",
        "name": "John Smith",
        "date_of_birth": "1973-11-04",
        "zip_code": "EC1A 1BB",
        "incident_type": "Road accident",
        "severity": "Serious",
        # An email that slipped into a free-text notes field
        "notes": "Witness contact: witness@example.com"
    }
]

clean = process_records(records)

The output is the following

Record 0: PII detected and handled in 3 field(s):
  [DIRECT] name → pseudonymized
  [QUASI] date_of_birth → generalized
  [QUASI] zip_code → pseudonymized
Record 1: PII detected and handled in 4 field(s):
  [DIRECT] name → pseudonymized
  [QUASI] date_of_birth → generalized
  [QUASI] zip_code → pseudonymized
  [DIRECT] notes → pseudonymized (unexpected PII in non-PII field)

A few things worth calling out in this implementation:

• Pseudonymization preserves referential integrity: Because the same input always produces the same hash token, you can still count unique individuals, join records, or track entities across datasets, without storing the raw PII. In production, replace the plain SHA-256 with an HMAC keyed on a secret, so tokens cannot be reversed by someone who also has access to the hashing algorithm.

• The regex safety net on non-PII fields: This catches the common real-world case where PII slips into a free-text or notes field that your schema classification didn’t anticipate. It is not foolproof, but it catches the obvious cases.

• The audit trail is intentional: Every field-level decision is logged. If you are ever asked to demonstrate that your collection process handled PII responsibly, you have a record of exactly what was done to each field in each record.

Conclusion

Open data is a shared resource, and how you interact with it says something about you as a professional. In this article, you learned what “open” means in the context of data scraping and how you should treat it if you want to be an ethical scraper.

So, let us know: Did we miss something? What’s your approach to handling open datasets in your scraping projects? Let’s discuss in the comments.

No wonder nearly half a trillion dollars are spent every year by financial firms on technology. Now, you probably don’t have that kind of money in the first place (and if you do, you don’t need to invest much anyway), but you might still want to collect financial data for personal use, research, academic projects, backtesting, or even just for selling it to industry giants.

No matter what you want to do with scraped financial data, there are a few pivotal tips to understand before embarking on this journey, which is exactly what I will explain here!

In this blog post, I will show why web scraping and finance are a match made in heaven and cover everything you need to know about retrieving both historical and real-time financial data from the web.

Web Scraping + Finance: A Happy Marriage

Before diving into web scraping for finance, let me explain why this is such a powerful approach and the advantages you can gain from it.

Finance Runs on (Web) Data

If there’s one thing that’s become clear over the past decade, it’s this: finance runs on data!

Financial institutions process massive volumes of market, customer, and transactional data every single day. In finance, data powers everything, from investment strategies to risk management. And the stakes are high, as bad data alone costs organizations an average of $12.9 million per year!

Data drives real-time decision-making, predictive modeling, and scenario planning. Finance teams feed that data into pipelines built around statistical analysis, machine learning, and AI to identify patterns, forecast market movements, and manage uncertainty in increasingly complex environments.

Now, here’s the central question we, web scraping enthusiasts, are all interested in: where does most of that data actually come from? A big portion of it comes from the web (not that surprising, uh?).

I’m talking about news sites, financial portals, company pages, official exchange websites, regulatory filings, institutional reports, and more. The web is essentially the largest and most dynamic data source available for financial purposes.

That’s exactly why web scraping in finance isn’t just useful. It’s foundational!

Before proceeding, let me thank NetNut, the platinum partner of the month. Their set of solutions cover all your needs for scraping.

Visit Netnut

Benefits of a Data-Driven Approach in Finance

Keep in mind that it’s not just big corporations or financial firms that benefit from data. Even individual retail investors can leverage financial data scraping to gain an edge. In particular, the main advantages include:

• Informed decisions: Access to accurate historical data supports smarter investment decisions, while real-time data enables more solid trading choices.

• Market trend insights: Spot patterns and emerging trends before the wider market does.

• Risk management: Identify potential risks early and adjust strategies proactively.

• Portfolio optimization: Fine-tune asset allocation based on backtesting and up-to-date market and company data.

• Efficiency and speed: Automate data collection, reducing time spent on manual research.

I mean, financial firms wouldn’t be spending over $495 billion a year (yeah, you read that right!) on technology (mostly built around collecting, processing, and leveraging data) if it didn’t give them a real edge!

Getting vs Selling Financial Web Data: High-Level Overview

There’s no doubt that financial firms invest billions into data. But what about you, as a web scraping expert, how can you leverage financial data for potential gain? There are two high-level approaches:

1. For yourself or your company: Build custom web scraping pipelines to gather data from multiple sources. Use it to feed investment models, AI agents, trading algorithms, or analytics pipelines. This is usually highly tailored to your strategies, risk appetite, or operational goals.

2. To sell to financial services: Collect, aggregate, and potentially enrich data from various sources to sell. You can offer broad datasets for many clients or fully customized solutions for a specific customer’s needs.

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

How to Approach Financial Data Scraping: Historical vs Real-Time

When it comes to finance, the web is packed with countless data fields and categories (e.g., news, stock prices, filings, analyst reports, and more). It’s a huge industry, and almost anything can be scraped!

At a high level, though, the key distinction for web scraping is simple: the financial data you want to collect is either historical or real-time. That’s what actually makes a difference in the approach to data scraping.

In the following chapters, I’ll dive deeper into each of the two categories of financial data. I’ll cover which fields are most interesting to scrape, where to find them, and how to collect them efficiently and effectively.

For now, start with a brief introduction to historical and real-time financial web data scraping!

Historical Financial Web Data

This includes all past financial data collected from the web, from historical stock prices to inflation rates and archived news. It’s used for analysis supporting long-term investment decisions.

👍 Pros:

• Enables backtesting of investment and trading strategies.

• Easier to scrape, as it isn’t time-sensitive.

• Data itself is stable and doesn’t change over time…

👎 Cons:

• …but the web pages displaying it (e.g., in tables and static charts) can still change, breaking your static parsing logic.

• Misses recent market shifts or breaking events.

• Data completeness varies across websites, often requiring aggregation from multiple sources.

Real-Time Financial Web Data

This includes live financial data extracted from the web, such as stock prices, market news, order books, etc. It’s employed for trading and short-term investment decisions.

👍 Pros:

• Enables fast, data-driven trading decisions.

• Captures live market movements and breaking news.

• Can be passed to AI agents and pipelines directly, as it tends to require minimal preprocessing.

👎 Cons:

• Harder to scrape reliably due to latency constraints and rate limits.

• Requires robust infrastructure for real-time ingestion and analysis, as every second counts.

• Data storage can grow rapidly because new data arrives continuously.

Mastering Historical Financial Data Scraping

As promised, let me guide you through the world of scraping historical financial data from the web.

Main Types of Historical Financial Web Data

The most important types of historical financial data you can retrieve from websites are:

• Historical stock and commodity prices: Open, high, low, close (OHLC) prices and trading volumes for stocks, ETFs, indices, and commodities, used for time-series analysis, modeling, and predictions.

• Summary info and infographics: Stock profiles, key metrics, and past indicators (e.g., P/E, EPS, moving averages), presented in dashboards or visual charts for quick insights.

• Macroeconomic indicators: Inflation, GDP, interest rates, unemployment, CPI, and PCE data, essential for understanding economic cycles and long-term market behavior.

• Financial statements: Company filings (income statements, balance sheets, cash flow), utilized for fundamental analysis and valuation models.

• News data: Archived headlines and press releases analyzed via NLP to correlate past market movements with specific events and sentiment shifts.

• ESG scores and sustainability reports: Historical environmental, social, and governance metrics employed to assess how “green” or ethical a company has been over time.

• Alternative data: Non-traditional datasets like web traffic, social media, satellite imagery (e.g., new headquarters or production plants), or credit card data for early performance signals.

Most Popular Targets

Also, if you’re interested in how to scrape historical data from the Wayback Machine, read my previous guide for this newsletter!

Scraping Techniques

Typical examples of historical financial data include lists of open, high, low, and close prices for a given stock:

Or, another example, the historical returns of a specific index (.e.g, SP500) over time:

These cases fall into the category of table-based data scraping, one of the most common web scraping scenarios. You’re probably already familiar with it, so there’s no need to go too deep here. Scraping older news and media can be slightly more challenging due to the unstructured nature of the target data, but it’s still a simple task.

At a high level, the process for getting historical finance data via web scraping follows a standard workflow:

1. Visit the target web page, either via an HTTP client or a browser automation tool.

2. Parse the page using an HTML parser, either directly or after rendering in a controlled browser.

3. Select the HTML elements of interest and extract the data.

4. Store the scraped data in your desired format (e.g., XLS, CSV, JSON) or in a database.

The main challenges involve generic anti-scraping mechanisms, such as CAPTCHAs, WAFs, IP bans, as well as browser, TLS, and device fingerprinting.

Best Practices

Based on my experience with financial web scraping, especially when focusing on historical data, these are the tips you should apply:

• Normalize and validate data: Standardize formats (dates, currencies, units) and validate across sources to catch inconsistencies early.

• Be cautious with AI parsing: Avoid using AI for automatically parsing structured data (tables, metrics, structured fields). It can introduce subtle errors and hallucinations, so prefer deterministic parsing. Harness AI mainly for retrieving unstructured text like news.

• Store raw HTML snapshots: Always keep the original page HTML. It lets you re-parse data later and extract new signals without re-scraping.

• Avoid single-source bias: When scraping news or market analysis pieces, pull data from multiple sources to reduce bias and improve reliability.

• Handle pagination properly: Many sites split historical data across pages or date ranges. Make sure your scraper fully traverses them all.

• Respect rate limits and retries: Even for historical data, implement retries and throttling to avoid blocks and incomplete datasets.

Check the TWSC YouTube Channel

Understanding Real-Time Financial Data Scraping

This is where things get a bit more interesting. Let me introduce you to real-time financial scraping!

Main Types of Real-Time Financial Web Data

The most relevant types of real-time financial web data are:

• Live price tickers: Continuously updated “last trade” prices and bid/ask spreads for stocks, crypto, and forex, used to detect breakouts and short-term trading opportunities.

• Order book and market depth: Incoming buy/sell orders, liquidity levels, and spreads, fundamental for execution strategies and high-frequency trading.

• Breaking news: Immediate updates and announcements that trigger sentiment models as soon as key figures (CEOs, central banks, governments) release information.

• Corporate event triggers: Monitoring press releases or SEC feeds for earnings surprises, M&A rumors, or sudden executive changes.

• Social media signals: Tracking ticker mentions on platforms like Reddit or X to detect retail-driven momentum, hype cycles, or panic selling in near real time.

• Institutional “whale” activity: Observing large trades or major wallet movements (especially in crypto) to identify where significant capital is flowing.

• Alternative digital signals: Web traffic spikes, app store ranking changes, or “out of stock” alerts on retail sites as proxies for real-world demand.

As you can tell, this category is more varied than historical financial data, including social media tracking and other less conventional practices. Thus, the sources to monitor for live financial web scraping can be less standardized and intuitive.

Most Popular Targets

Scraping Techniques

Imagine applying a traditional scraping pattern to real-time financial data. You send a request to a target site, extract a stock price, and repeat the operation every few seconds or even milliseconds.

The problem is latency. By the time the server responds, the page is rendered or parsed, the target data field is collected, and stored or sent to your pipeline, that piece of data is already outdated.

On top of that, this approach requires a crazy number of requests in a very short time. That increases the risk of triggering rate limiting or even IP bans. You might think proxies solve that through IP rotation, but most proxy networks introduce additional latency, often 2/3/5 seconds per request. In real-time scenarios, that delay is simply not acceptable!

Even if you switch to faster or dedicated proxies, you may end up with a smaller IP pool, which increases the likelihood of those IPs getting blocked.

A more advanced idea is to rely on browser automation and keep a page open, capturing updates as they happen. This is smarter, but still problematic. Long-lived sessions with little or no user interaction are highly suspicious and can easily trigger anti-bot systems. Plus, browser automation at scale tends to be flaky, not really reliable for persistent connections.

Long story short, scraping real-time financial data this way quickly turns into a losing game.

The solution? Stop targeting the data presentation layer in HTML and instead go directly to the data source!

API/WebSocket Scraping as The Solution

Web pages showing real-time financial data aren’t doing anything magical. Behind the scenes, they either poll APIs at regular intervals or (more commonly) maintain a persistent connection via WebSockets to receive continuous updates. The page simply renders that incoming data.

As a result, a much better approach is to intercept and replicate those data flows. You can do this through AJAX/API request inspection or WebSocket sniffing. Open the browser developer tools, go to the “Network” tab, and check where the data is coming from.

If it’s an API call, you’ll see it under the “Fetch/XHR” tab:

If it’s a WebSocket, you’ll find it under the “Socket” tab:

Once identified, replicate those API calls or connect directly to the WebSocket in your scraping script. This gives you access to near real-time financial data in a structured format (typically JSON) without the overhead of parsing HTML.

Of course, that’s not trivial. WebSockets require proper anti-bot bypass, and APIs may still enforce rate limits, tracking, and TLS fingerprinting protections. However, this approach is generally faster, more reliable, and much easier to maintain than scraping rendered pages!

And What About Live News or Social Media Scraping?

When it comes to news, if available, it makes sense to connect to public RSS feeds exposed by websites to monitor updates. This allows you to trigger scraping only when new and relevant content is published, instead of constantly polling pages unnecessarily.

Otherwise, you can build a polling mechanism that periodically checks news sites, social media platforms, and similar sources to capture fresh data. In these cases, you usually can’t rely on techniques like API or WebSocket scraping, as that’s not how those platforms fetch data.

Instead, you need a solid and robust infrastructure built around speed and efficiency: fast connections, high-quality proxies, optimized parsing, and lightweight requests. The goal is to minimize latency while maintaining reliability at scale.

Best Practices

Scraping real-time financial data is a demanding art, but it becomes easier with the following best practices:

• Prefer APIs and WebSockets over HTML parsing: Whenever possible, save time by extracting data directly from the underlying APIs or WebSocket streams utilized by web pages instead of scraping data from rendered pages.

• Choose clean, structured sources: Prioritize endpoints that return well-formatted JSON to minimize preprocessing and reduce latency.

• Stream data into pipelines immediately: Send incoming data directly to processing pipelines for real-time insights, while storing it in parallel for later analysis.

• Use specialized AI for sentiment analysis: Prefer AI/ML models tuned for finance/social media, as Reddit and X content often include slang, memes, and non-standard language.

• Optimize browser automation: Configure Playwright, Selenium, or similar browser automation tools to block images, stylesheets, and fonts. This reduces bandwidth usage and significantly speeds up rendering time.

• Design for low latency: Optimize your stack (async requests, streaming ingestion, fast JSON parsers) to minimize delays, as even milliseconds matter.

• Prefer high-quality premium proxies: Count on proxy providers with a proven track record of fast, stable connections to minimize latency and avoid disruptions.

• Time-synchronize everything: Append timestamps to all scraped data to enable time-series analysis and accurately reconstruct events.

• Build fault-tolerant systems: Expect disconnections (especially with WebSockets) and issues, so add reconnection logic and configure fallback data sources.

Need help with your scraping project?

Top 5 Open-Source Financial Web Scraping Libraries

Below is a selected set of interesting, fully open-source libraries, packages, and projects for simplified financial web scraping:

Conclusion

Here, I’ve gone through the rabbit hole of financial web scraping, the task of collecting finance-related data from the Internet. This is one of the main use cases of corporate web scraping, powering enterprise data pipelines for decision-making and market analysis.

As you’ve seen, the main difference in the approach comes down to whether you’re targeting historical or real-time data. The first follows standard web scraping practices you’re likely already familiar with. The second is trickier and requires more advanced techniques.

I hope you found this helpful and insightful. If you have questions, feel free to share them in the comments below!

This time, we wanted AWS WAF on its own, in front of a target with nothing else in front of it, and we wanted to see what changes when the 2024 Scrapy-Playwright stack is replaced with the 2026 toolbox.

The target we picked is imdb.com. It is an Amazon subsidiary, runs a standard AWS WAF deployment, and Wappalyzer confirms that there are not others antibot on the website. That makes IMDB a perfect use case for our article.

Before proceeding, let me thank NetNut, the platinum partner of the month. Their set of solutions cover all your needs for scraping.

Visit Netnut

Today we’ll test three Python HTTP clients with strong TLS fingerprint impersonation: curl_cffi, the newer httpx-curl-cffi, and Rust-backed rnet. Each one produces a TLS handshake indistinguishable from real Chrome. Is that enough to scrape an AWS WAF target without spinning up a browser? And if not, what is the smallest browser step that gets us past the gate so the rest of the work can run on a cheap HTTP client?

The tools we used

Four libraries are in scope. Three are HTTP-only, one runs a real browser.

curl_cffi is a Python binding for the curl-impersonate patched curl. It exposes a requests-like API and ships impersonation profiles for recent Chrome, Firefox, and Safari builds and works at the TLS layer. JA3 and JA4 fingerprints match the impersonated browser, along with HTTP/2 settings and header order. We tested with chrome142, the latest Chrome profile in version 0.14.0.

httpx-curl-cffi is a transport for httpx that delegates the actual HTTP work to curl_cffi. While it does not add new fingerprinting capability, it implements the httpx programming model: sync Client, async AsyncClient, event hooks, the same response object you get from the rest of an httpx-based codebase. We tested with the Chrome profile and default_headers=True.

rnet is a Rust HTTP client with Python bindings. It implements its own impersonation stack rather than wrapping curl-impersonate. The enum rnet.Impersonate exposes a wide range of Chrome, Firefox, Safari, Edge, Opera, and OkHttp profiles. We tested with Chrome137.

Scrapling is the only browser-driven tool in the set. Our Scrapling: A Complete Hands-On Guide goes through the library in depth, with Cloudflare as the test target. Its StealthyFetcher drives a stealth-patched Chromium that runs JavaScript and applies fingerprint countermeasures. The library README only advertises Cloudflare Turnstile, but the same machinery handles AWS WAF’s challenge too.

Your scraping workflows deserve a proxy infrastructure that just works. With Swiftproxy on your side, consistency is built-in.

Try Swiftproxy today

How AWS WAF protects IMDB

A quick intro of the system helps interpret the results that follow. AWS WAF is not a dedicated anti-bot platform like DataDome or Kasada. It is a general-purpose web application firewall with a bot-control module that operators can enable per rule. When the bot-control rule is in challenge mode, AWS WAF inserts a single JavaScript gate at the start of a session.

A request without a valid cookie returns HTTP 202 with x-amzn-waf-action: challenge and a short HTML body. The body contains window.gokuProps containing three base64 blobs (key, iv, context), a <script src> pointing to a customer-specific URL on *.token.awswaf.com, and a small inline script that calls AwsWafIntegration.saveReferrer(), AwsWafIntegration.checkForceRefresh(), and AwsWafIntegration.getToken(). The remote challenge.js tests the browser environment, posts a validation payload back to AWS, and on success, the response sets Set-Cookie: aws-waf-token=.... The inline script then reloads the page, and the second request, now carrying the token, gets the real content.

This works very differently from systems that score every request. Once the token is in our jar, AWS WAF lets us through with no further behavioral checks beyond IP reputation and rate limits.
What we want to discover with this article is if we’re able to bypass AWS WAF with “convincing” requests, with a proper TLS fingerprint and set of headers, or if we need a JS rendering engine.

Check the TWSC YouTube Channel

Test setup

As always, the code can be found in our GitHub repository reserved to paying users, inside the folder 104.IMDB. If you’re not able to access the repository, please use this form to request access.

The libraries we pinned at the time of writing are curl_cffi==0.14.0, httpx==0.28.1, httpx-curl-cffi==0.1.5, rnet==2.4.2, scrapling==0.4.7. Python is 3.11.

Each HTTP test creates a GET against two URLs: the IMDB home page

https://www.imdb.com/

and a title page https://www.imdb.com/title/tt0111161/. We use two URLs to confirm the challenge fires the same way on both, not only on one entry point. We do not follow redirects (follow_redirects=False) because the AWS WAF response is a 202 with content rather than a redirect, and we want to see it raw.

We capture status code, HTTP version, the full response headers, any cookies, body length, and the first 600 characters of the body, and we saved everything to JSON under aws_waf_imdb/responses/ for later inspection.

The baseline probe in probe_plain.py uses an unmodified httpx.Client(http2=True) with a generic Chrome User-Agent header and the standard Accept headers. This is the control: no TLS impersonation, no fingerprint trickery, just a normal Python HTTP client.

Read more

The web contains an extraordinary volume of information, the majority of which is in textual form. Blogs, forums, and newsletters alone generate millions of words of domain-specific knowledge every week. And they’re not the only sources of text on the web.

When you want to get insights from that kind of data, successfully extracting it from the web is only half of the battle, even now that LLMs can use vision to scrape complex visual layouts. The second part of the challenge is structuring this data to get it ready for analytics. Why? Because when you point a scraper at a news article, you get back a wall of text. But you cannot query it. You cannot aggregate it. You cannot feed it reliably into a machine learning pipeline or a database without significant preprocessing.

This article addresses the preprocessing problem of unstructured text when you scrape it from the web. It traces the evolution of solutions from classical NLP to large language models, identifies where each approach breaks down, and proposes a practical architectural solution.

Let’s get into it!

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

What “Unstructured” Really Means in Practice

Unstructured text refers to content that carries no machine-readable schema. The information exists in the data you retrieved from the web, but no field boundaries exist, no consistent labels, and no guaranteed position for any given fact.

The following schema represents the difference between unstructured and structured text (machine-readable schema):

Let’s consider three concrete scraping targets to illustrate what this costs you in practice.

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

News Articles: When Signals Are Buried in Noise

Consider you scraped a Reuters article about an ECB rate decision. The text you get back from the scraper could be something as follows:

European Central Bank decides on rates.
Listen to this article. 2 min audio. 
You might also like: Eurozone inflation hits 3-year low. 
Christine Lagarde announced Thursday a 25 basis point reduction, bringing the main 
refinancing rate to 3.40%. 
SPONSORED: Track macro events with Bloomberg Terminal. 
The decision was widely anticipated after last month's CPI print. Share this article. 
4 comments. John M. writes: this was priced in already

Your raw text contains the article body, a teaser for a related story, a sponsored insertion, and reader comments. The fact you want is buried in there: the ECB cut its main refinancing rate to 3.40% on a specific date. But your extractor gets the full content.

Such a wall of text, which, generally speaking, is way bigger than this and is useless for analytics purposes without preprocessing.

Financial Newsletters: When “Just Under Two Percent” Breaks Your Aggregation

Suppose you scrape a financial newsletter to extract an updated macroeconomic forecast. You need to capture a specific fact. Something like “Goldman Sachs revised its 2026 US GDP growth forecast down to 1.8%”. Your scraper captures the entire page output, which is similar to an article. Similarly to the previous example, the resulting raw text mixes the core facts with boilerplate and unrelated news:

Market Daily Newsletter. November 12.
Jan Hatzius (Goldman Sachs) and his team were out with a note early Tuesday.
SPONSORED: Get 50% off your trading fees today. 
They see tariffs shaving roughly 0.7 points off the baseline. Meanwhile, 
European markets rallied on ECB news. 
Read our full coverage of the Eurozone here.
The revised number now sits just under two percent for the full year.
Subscribe for premium insights.

The text distributes the target fact across the entire document. Also, the wording “just under two percent” requires numerical understanding to say that the text refers to the actual number you were searching for, that is, an exact 1.8%.

Now, imagine generalizing this after scraping hundreds of financial news and newsletters to regroup the information to summarize the numbers. Getting insight would be impossible. Why? Because some sources will give you the actual information you want (growth forecast down to 1.8%), others will use different phrasing to define the trend (”An expected growth under 2 percent”, “a slightly shrinking trend”, etc).

Without a way to create a structure for such data, you can’t get any insights from it.

Job Posting Offers: They Are Always Messier Than They Look

Consider the case when you want to scrape job offers to get an idea of what the market is paying on average for a specific position, given the expected technical skills, and considering the same day-to-day activity. Job offers can have the following ambiguities:

• A sentence might read “3+ years of experience with Python”. This establishes a floor and ignores a ceiling. Alternatively, the text might read “Senior-level candidates only”. This uses qualitative seniority as a proxy for an exact quantitative number.

• Salary breaks in a different direction. One posting can say “$120,000 - $145,000 base”. Another can be “competitive compensation commensurate with experience”. A third could be“€100,000”, which you need to convert to dollars to make an actual comparison.

• Employment type can introduce further ambiguity and difficulties. “Full-time”, “FTE”, “permanent”, and “direct hire” basically mean the same thing but are written differently. Also, the text might specify the role is “Hybrid”, which means multiple different things across companies. It could mean two days in the office. It could mean occasional travel with headquarters-optional rules.

When sites get tough, skip the heavy lifting. Get clean, structured CSV datasets, ready for Excel, BI or your apps

Find your dataset

How Classical NLP Tried to Solve This (and Where It Stopped)

Before large language models were released, the standard answer to this problem was Natural Language Processing. The classical NLP toolkit gave developers a set of tools that could, with enough effort, extract meaningful structure from text using different, but often interconnected, processes like the following:

• Named Entity Recognition (NER): NER is a process used in NLP to extract entities from text corpora. It can particularly identify spans of text as persons, organizations, locations, or dates. An NLP model trained on news corpora, for example, is able to scan an article and tag “Jane Doe” as a person and “Washington D.C.” as a geopolitical entity.

• Part-of-speech tagging: Is a process in which NLP models can identify nouns, verbs, and adjectives. This enables the downstream logic to focus on the right parts of a sentence.

• Dependency parsing: Maps grammatical relationships between words, helping to extract which subject performed which action on which object.

• Relation extraction: Identifies when two co-occurring entities have a specific relationship. For example, a person who was affiliated with an organization, or an event that occurred in a specific location.

Libraries like spaCy, Stanford NLP, and NLTK made these processes largely accessible. But they work well for well-defined, narrow tasks on consistent text domains. The problems and limitations of this solution appear quickly at the edges:

• Domain shift breaks everything: A NER model trained on news articles performs poorly on scientific abstracts. A model tuned for English financial text fails on multilingual content. In other words, every new domain requires retraining, re-labeling, and re-evaluation. These processes are very costly, both in terms of money and time.

• Context is invisible: Classical NLP models operate at the token and sentence level. They have no mechanism for understanding that “Apple” in a technology article refers to a corporation, while “apple” in a nutrition blog refers to a fruit. Disambiguation requires hand-crafted rules or separate classification layers bolted on top (which, again, is costly).

Before NLP, you could basically only use regex (with all the difficulties associated with manually cleaning data, standardizing it, and…using regex!). So, NLP was a genuine (big) step forward: it made large-scale text analysis possible in ways that pure pattern matching never could (which is a way to find patterns in scraped data using AI). But it still required substantial domain expertise, constant maintenance, and produced results that were narrow, fragile, and difficult to generalize.

Check the TWSC YouTube Channel

The Modern Solution: LLMs as Universal Structure Extractors

Large language models fundamentally changed the extraction problem. On the side of the underlying technology, a classical NLP model learns the statistical patterns inside the text. An LLM, instead, learns to understand language. This distinction matters enormously because it opened the doors to the following:

• Context disambiguation that works out of the box: Feed an LLM with a paragraph from a technology article containing the word “Apple” and it will correctly identify it as a company. Feed it with a paragraph from a recipe blog, and it will correctly identify it as a fruit. No separate disambiguation layer. The model resolves ambiguity the same way a human reader does: by reading the surrounding context.

• Semantic equivalence that is understood, not computed: An LLM knows that “$40,” “forty dollars,” “40 USD,” and “forty bucks” all express the same value. You don’t need to instruct it to understand that.

• Implicit information that becomes accessible: A sentence like “the study, conducted over three months at a Boston hospital, found no significant effect” contains a location, a duration, and a finding. An LLM can extract all three without requiring the text to follow any particular structure.

• Domain generalization that requires no retraining: The same LLM that extracts entities from political news articles can extract findings from scientific abstracts, event mentions from cultural journalism, and source attributions from investigative reporting. You just need to change the prompt, not the model.

The practical workflow becomes straightforward:

• You scrape unstructured text from the web.

• You pass the content to an LLM with a prompt that describes what you want to extract.

• The model returns a response.

• You use that response downstream.

This process works. But using LLMs alone introduces a different class of problems:

• Output format is not guaranteed: Ask an LLM to return a price, and it might return $40 in one run, 40 dollars in another, and 40 USD in a third. The model understands the value when it retrieves it from scraped content. But it does not guarantee how it expresses that value unless you explicitly constrain it.

• Required fields can go missing: If the article you extracted the content from does not mention a publication date, the model might omit the field, return null, or return "not mentioned", or invent a plausible date (which is way worse). Each behavior is different, and none of them is predictable without enforcement.

• Hallucination is a real risk: When the model is uncertain, it always generates a plausible answer. For extraction tasks, that means it can invent entity names, fabricate statistics, or fill in missing information with confident-sounding fiction. Without validation, these errors pass into your data, creating issues at the analytics level.

Generalizing all of this, you also get scalability issues because you have no consistency guaranteed. A pipeline processing 10,000 articles requires every output to follow the same schema. But a model that returns slightly different structures across runs cannot feed a database reliably without significant error handling.

In other words, LLMs provide you with the understanding that NLP lacked. But they do not, on their own, provide the structural guarantees that production pipelines require.

Need help with your scraping project?

How to Get Semantic Power and Structural Guarantees at the Same Time: A Practical Approach

One possible solution to the unpredictability of LLM outputs is to separate the two concerns that these models conflate: semantic understanding and structure enforcement.

To do so, you can:

• Use the LLM for what it does well: reading text, resolving ambiguity, extracting meaning, and normalizing inconsistent expressions.

• Use specific libraries to define schemas, enforce types, validate outputs, and reject malformed data before it enters your pipeline.

Below is how this solution works, at a high level:

Let’s see how to implement this process and how the two approaches differ in practice.

The Baseline Approach: A Direct LLM Call (and What It Gives You)

Consider the following content that can come from scraping a news article:

Fed Signals Caution as Inflation Data Disappoints

By Sarah M. Connelly | April 14, 2026 | Economics & Policy

The Federal Reserve signaled on Monday that it remains in no rush to cut interest rates,
after fresh inflation data showed consumer prices rose more than expected last month.
Jerome Powell, speaking at a conference in Washington, said the central bank needs
"greater confidence" that inflation is moving sustainably toward its two-percent target
before reducing borrowing costs.

The Consumer Price Index climbed 3.5 percent in March, up from 3.2 percent in February,
according to the Labor Department. Economists polled by Reuters had forecast a reading
of 3.3 percent. Core CPI, which strips out food and energy, rose 3.8 percent year-over-year.

Markets reacted sharply. The S&P 500 fell nearly one point five percent by midday,
while the yield on the 10-year Treasury note jumped to four point six percent.
Goldman Sachs revised its forecast, pushing back its expected first rate cut from June
to September. JPMorgan analysts said two cuts in 2026 now look "optimistic."

Powell emphasized that the Fed is not considering rate hikes at this stage, but stressed
that the path back to two percent inflation "may take longer than previously thought."
The next Fed meeting is scheduled for the first week of May

To directly pass it to a GPT model, asking it for a precise output, you can use the following code:

import os
import json
from openai import OpenAI

# Scraped content
SCRAPED_TEXT = """
Fed Signals Caution as Inflation Data Disappoints

By Sarah M. Connelly | April 14, 2026 | Economics & Policy

The Federal Reserve signaled on Monday that it remains in no rush to cut interest rates,
after fresh inflation data showed consumer prices rose more than expected last month.
Jerome Powell, speaking at a conference in Washington, said the central bank needs
"greater confidence" that inflation is moving sustainably toward its two-percent target
before reducing borrowing costs.

The Consumer Price Index climbed 3.5 percent in March, up from 3.2 percent in February,
according to the Labor Department. Economists polled by Reuters had forecast a reading
of 3.3 percent. Core CPI, which strips out food and energy, rose 3.8 percent year-over-year.

Markets reacted sharply. The S&P 500 fell nearly one point five percent by midday,
while the yield on the 10-year Treasury note jumped to four point six percent.
Goldman Sachs revised its forecast, pushing back its expected first rate cut from June
to September. JPMorgan analysts said two cuts in 2026 now look "optimistic."

Powell emphasized that the Fed is not considering rate hikes at this stage, but stressed
that the path back to two percent inflation "may take longer than previously thought."
The next Fed meeting is scheduled for the first week of May.
"""

# Define LLM client
raw_client = OpenAI(api_key=os.environ.get("YOUR_OPENAI_API_KEY"))

# Define prompt for the LLM
raw_prompt = """
Extract the following information from the article below and return it as JSON:
- title
- author
- publication_date
- mentioned_organizations
- cpi_march_value
- key_claim
- market_sentiment

Article:
""" + SCRAPED_TEXT

# Get response from LLM
raw_response = raw_client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": raw_prompt}]
)

raw_output = raw_response.choices[0].message.content

# Print results
print(raw_output)

The result will be as follows:

{
  "title": "Fed Signals Caution as Inflation Data Disappoints",
  "author": "Sarah M. Connelly",
  "publication_date": "April 14, 2026",
  "mentioned_organizations": [
    "Federal Reserve",
    "Labor Department",
    "Reuters",
    "Goldman Sachs",
    "JPMorgan"
  ],
  "cpi_march_value": "3.5 percent",
  "key_claim": "The Federal Reserve is in no rush to cut interest rates and needs greater confidence that inflation is moving sustainably toward its two-percent target before reducing borrowing costs.",
  "market_sentiment": "Negative"
}

Now, at first sight, this seems good. The prompt asked the GPT model to create a JSON file with specific values, and the model was able to do so. But two major problems affect the next steps when analyzing this data. They are:

• The publication date is reported as “April 14, 2026”. This is not represented in ISO 8601 format and will break any date parser.

• The CPI is reported as “3.5 percent”, which is a string. Not a number or a float, which is what is required for such data if you want to further analyze it (without any intermediate steps).

So, the LLM was able to give structure to an unstructured text, after being specifically prompted to do so. But it failed at providing the data in the right format. To do so, you have to provide specific guidance to the model.

What Changes When You Define The Schema

To have guarantees on the output format, you can use the following code:

import os
import json
import instructor
from openai import OpenAI
from pydantic import BaseModel, Field
from typing import Optional, Literal

# Scraped content
SCRAPED_TEXT = """
Fed Signals Caution as Inflation Data Disappoints

By Sarah M. Connelly | April 14, 2026 | Economics & Policy

The Federal Reserve signaled on Monday that it remains in no rush to cut interest rates,
after fresh inflation data showed consumer prices rose more than expected last month.
Jerome Powell, speaking at a conference in Washington, said the central bank needs
"greater confidence" that inflation is moving sustainably toward its two-percent target
before reducing borrowing costs.

The Consumer Price Index climbed 3.5 percent in March, up from 3.2 percent in February,
according to the Labor Department. Economists polled by Reuters had forecast a reading
of 3.3 percent. Core CPI, which strips out food and energy, rose 3.8 percent year-over-year.

Markets reacted sharply. The S&P 500 fell nearly one point five percent by midday,
while the yield on the 10-year Treasury note jumped to four point six percent.
Goldman Sachs revised its forecast, pushing back its expected first rate cut from June
to September. JPMorgan analysts said two cuts in 2026 now look "optimistic."

Powell emphasized that the Fed is not considering rate hikes at this stage, but stressed
that the path back to two percent inflation "may take longer than previously thought."
The next Fed meeting is scheduled for the first week of May.
"""

# Validation schema
class ArticleExtraction(BaseModel):
    title: str = Field(description="The article's headline")
    author: Optional[str] = Field(description="Full name of the author if explicitly mentioned")
    publication_date: Optional[str] = Field(description="Publication date in ISO 8601 format (YYYY-MM-DD)")
    mentioned_organizations: list[str] = Field(description="All organizations referenced in the article")
    cpi_march_value: Optional[float] = Field(description="CPI value as a float (e.g. 3.5)")
    key_claim: str = Field(description="The central argument or finding of the article in one sentence")
    market_sentiment: Literal["positive", "negative", "neutral"] = Field(
        description="Overall market sentiment expressed in the article"
    )

structured_client = instructor.from_openai(OpenAI(api_key=os.environ.get("YOUR_OPENAI_API_KEY")))

extraction = structured_client.chat.completions.create(
    model="gpt-4o",
    response_model=ArticleExtraction,
    messages=[
        {
            "role": "user",
            "content": f"Extract structured information from the following article:\\n\\n{SCRAPED_TEXT}"
        }
    ]
)

print(extraction.model_dump_json(indent=2))

print("\\n" + "=" * 60)
print("INSPECTION OUTPUT")
print("=" * 60)
for field, value in extraction.model_dump().items():
    print(f"  {field}: {repr(value)}  →  type: {type(value).__name__}")

The above code leverages two fundamental libraries:

• Pydantic: This is a Python data validation library. You define a schema as a Python class, declare the fields and their types, and Pydantic enforces that any data you put into that class matches what you declared.

• Instructor: This is the bridge between Pydantic and the LLM. The core problem it solves is that LLMs’ APIs return text, but Pydantic validates Python objects. So, something has to sit in the middle, take the LLM’s response, parse it into the structure your Pydantic model expects, and retry the call if the output doesn’t validate. That’s what Instructor does. Without Instructor, you would have to manually prompt the model to return JSON, parse that JSON yourself, handle malformed responses, write retry logic, and coerce types by hand.

By using these two libraries, the ArticleExtraction() class does the following:

• Type enforcement: Defines cpi_march_value as a float. This guarantees the model will return an actual number) instead of a string (3.5 instead of "3.5 percent" as the previous example).

• Controls formatting and vocabulary: The Literal type on market_sentiment restricts the LLM’s output to "positive", "negative", or "neutral". The model cannot invent new categories. Similarly, the description for publication_date explicitly demands the ISO 8601 format.

• Built-in prompting: The Field(description="...") parameters serve a dual purpose. First, they document the code for developers. Secondly, under the hood, the Instructor library feeds these exact descriptions to the LLM as targeted instructions. This ensures the model understands exactly what “key claim” or “publication date” means in this context.

• Graceful omissions: Wrapping fields like author in Optional[...] gives the model permission to safely return a null value if the information isn’t present in the scraped text. This highly reduces the risk of hallucinations.

The JSON output is as follows:

{
  "title": "Fed Signals Caution as Inflation Data Disappoints",
  "author": "Sarah M. Connelly",
  "publication_date": "2026-04-14",
  "mentioned_organizations": [
    "Federal Reserve",
    "Labor Department",
    "Reuters",
    "Goldman Sachs",
    "JPMorgan"
  ],
  "cpi_march_value": 3.5,
  "key_claim": "The Federal Reserve remains cautious about cutting interest rates because inflation has not yet shown sufficient progress toward its two-percent target.",
  "market_sentiment": "negative"
}

As you can see, now the CPI is a float, and the publication date is in ISO 8601.

The inspection output is the following:

============================================================
INSPECTION OUTPUT
============================================================
  title: 'Fed Signals Caution as Inflation Data Disappoints'  →  type: str
  author: 'Sarah M. Connelly'  →  type: str
  publication_date: '2026-04-14'  →  type: str
  mentioned_organizations: ['Federal Reserve', 'Labor Department', 'Reuters', 'Goldman Sachs', 'JPMorgan']  →  type: list
  cpi_march_value: 3.5  →  type: float
  key_claim: 'The Federal Reserve remains cautious about cutting interest rates because inflation has not yet shown sufficient progress toward its two-percent target.'  →  type: str
  market_sentiment: 'negative'  →  type: str

This validation helps immediately see that the data types are correct.

Conclusion

In this article, you learned what unstructured text actually costs a data pipeline. You saw how classical NLP made structured extraction possible but fragile, and how LLMs removed the domain constraints that NLP never solved. You also learned why LLMs alone are not enough and saw a practical solution to provide “guardrails” for LLMs so that their output follows a defined schema.

So, let us know: how are you managing unstructured text after you scraped it?

In this article, you’ll find out this and more. I’ll walk you through a complete guided example of how to use it, and break down its (Spoiler: undoubtedly serious) limitations.

An Introduction to the Cloudflare Crawl Endpoint

Before exploring the technical aspects behind the Cloudflare /crawl endpoint and seeing it in action, let me first give you some context!

What Is the Cloudflare /crawl Endpoint?

The /crawl endpoint is a new addition to Cloudflare’s REST APIs. Its goal is to crawl an entire website (or just a portion of it) starting from a single URL.

Note: The Crawl endpoint is currently in beta and was introduced on March 10, 2026, as highlighted in the Cloudflare changelog.

Under the hood, it automatically discovers and visits new pages, rendering them in a headless browser. It returns the discovered content as HTML, Markdown, or structured JSON, making it ideal for RAG pipelines, monitoring, or dataset creation.

As I’ll dive into later, it respects robots.txt and doesn’t bypass bot protection or captchas. Thus, it’s designed as a compliant approach to web crawling!

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

How It Works at a High Level

At a high level, the /crawl endpoint involves a two-step flow:

1. You kick off an asynchronous crawl job, passing a starting URL. Cloudflare returns a job ID.

2. You use that job ID to periodically check the job’s status or fetch results as they become available, following typical polling behavior.

Important: A crawl job can run for up to seven days! Results remain available for 14 days after completion, after which the job data is deleted.

Behind the scenes, the crawler expands outward from the starting URL. By default, the API follows a clear order:

1. The initial page.

2. Sitemap URLs.

3. Links discovered within pages.

Still, you can tweak that depending on whether you want to prioritize sitemaps, page links, or both.

Supported Use Cases

The officially promoted use cases for the Cloudflare /crawl API are just two:

• Creating knowledge bases or training AI systems (like RAG applications) using up-to-date web content.

• Collecting and analyzing content across multiple pages for research, summarization, or monitoring purposes.

Pricing

Compared to most other web crawling or discovery APIs on the market, Cloudflare’s /crawl API doesn’t charge by the number of pages. Instead, costs are based on resource usage, which depends on whether you enable the headless browser rendering feature.

If headless rendering is active, pricing follows the Browser Rendering model:

If rendering isn’t active, pricing follows the Workers model:

Yeah, I know… It’s honestly a bit confusing, and it’s almost impossible to predict the exact cost of a crawling task. The good news? You can test it for free!

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

Cloudflare Crawl Endpoints: Technical Analysis

Now that you know what Cloudflare is and what it brings to the table, it’s time to better understand its functioning, strengths, and limitations.

Endpoint Presentation

The Cloudflare Crawl API is built around two main endpoints. Both share the same base URL:

https://api.cloudflare.com/client/v4/accounts/<ACCOUNT_ID>/browser-rendering/crawl

Where <ACCOUNT_ID> is your Cloudflare account ID.

1. Initiate the Crawl Job (POST)

To start a new crawl, you need to send a POST request with the target URL (and optional parameters like depth, rendering mode, etc.) as below:

curl -X POST 'https://api.cloudflare.com/client/v4/accounts/<ACCOUNT_ID>/browser-rendering/crawl' \
  -H 'Authorization: Bearer <CLOUDFLARE_API_TOKEN>' \
  -H 'Content-Type: application/json' \
  -d '{ "url": "https://example.com" }'

Keep in mind that the endpoint supports several parameters, allowing you to greatly customize the crawling behavior, output format (JSON, HTML, or Markdown), rendering options, caching, and more. Check out the full list of supported body parameters for all available options.

Cloudflare immediately returns a job ID that you’ll use to track or retrieve results. A possible response looks like this:

{
  "success": true,
  "result": "9f1c2d3a-4b5e-6f7a-8c9d-0e1f2a3b4c5d"
}

The UUID in the result field is the Crawl job ID you’ll use to poll for updates.

2. Request Crawl Results (GET)

Once the crawl is running, make a GET request with the job ID to check the status or fetch results:

curl -X GET 'https://api.cloudflare.com/client/v4/accounts/<ACCOUNT_ID>/browser-rendering/crawl/<JOB_ID>' \
  -H 'Authorization: Bearer <CLOUDFLARE_API_TOKEN>'

Here, the <JOB_ID> placeholder is the UUID retrieved before from the result field.

The response either includes a status field like this:

{
  "result": {
    "id": "3e7a1c92-b4d8-4f6e-9a21-6c0d5b8e2f14",
    "status": "running"
    // ...
  }
}

The possible status values are: running, completed, errored, or one of several cancellation states (cancelled_due_to_timeout, cancelled_due_to_limits, cancelled_by_user).

Or, once the job is completed, calling the API returns the full results in the records field:

{
  "result": {
    "id": "3e7a1c92-b4d8-4f6e-9a21-6c0d5b8e2f14",
    "status": "completed",
    "browserSecondsUsed": 98.3,
    "total": 12,
    "finished": 12,
    "records": [
      {
        "url": "https://example.com/",
        "status": "completed",
        "markdown": "# Example Domain\nThis domain is for use in illustrative examples...",
        "metadata": {
          "status": 200,
          "title": "Example Domain",
          "url": "https://example.com/"
        }
      },
      {
        "url": "https://example.com/about",
        "status": "completed",
        "markdown": "## About\nLearn more about this example site...",
        "metadata": {
          "status": 200,
          "title": "About - Example Domain",
          "url": "https://example.com/about"
        }
      }
      // additional entries omitted for brevity...
    ],
    "cursor": 10
  },
  "success": true
}

Note that the response will vary based on the specified query parameters. For example, you can filter by specific statuses, limit the number of results, and navigate through them using a pagination system.

Features

Below is a list of the main, most relevant capabilities provided by the Cloudflare Crawl API:

• Asynchronous crawl jobs: Trigger crawling jobs and poll results when they are ready, enabling non-blocking, large-scale crawling workflows.

• Automatic URL discovery: Finds pages from the starting URL, sitemaps, and in-page links, with configurable source control.

• Flexible output formats: Returns HTML, Markdown, or structured JSON. JSON leverages Workers AI for schema-driven data extraction.

• Headless browser rendering: Control JavaScript execution with render: true or perform fast static HTML fetches with render: false.

• Fine-grained crawl control: Configure limit, depth, and URL inclusion/exclusion with the includePatterns/excludePatterns fields.

• Incremental and cache-aware crawling: Use modifiedSince and maxAge parameters to avoid re-fetching unchanged content, optimizing performance and cost.

• Advanced filtering and pagination: Retrieve results using limit, cursor, and status filters to handle large datasets efficiently.

• Authentication and custom headers: Supports HTTP auth, cookies, and custom headers for crawling protected or API-driven content.

• Dynamic content handling: Wait for JS-rendered content using gotoOptions and waitForSelector, ideal for SPAs and interactive pages.

• Resource skipping for performance: Optionally block images, media, fonts, or stylesheets to speed up crawling and reduce unnecessary bandwidth usage.

Your scraping workflows deserve a proxy infrastructure that just works. With Swiftproxy on your side, consistency is built-in.

Try Swiftproxy today

Limitations

Cloudflare’s /crawl API also comes with several important limitations, such as:

• Respects bot protection: The crawler can’t bypass CAPTCHAs (including Turnstile challenges) or Cloudflare bot protections. As a rule of thumb, sites protected via Cloudflare Bot Management or other WAFs tend to block crawling tasks entirely, limiting automated access and leading to incomplete datasets.

• Fixed User-Agent: The /crawl endpoint sets a non-customizable User-Agent value (CloudflareBrowserRenderingCrawler/1.0). You can’t change it, which may cause sites to block requests or serve different content based on the User-Agent.

• Content Signals enforcement: If a site disallows AI usage via Cloudflare Content Signals, crawl requests for those purposes are rejected with a 400 Bad Request error. Even if the site allows other uses, attempts to crawl disallowed categories will fail, limiting AI-specific data collection.

• Rate limiting and crawl pacing: Sites with strict rate limits can slow down crawling. The crawler respects the robots.txt Crawl-delay directive and implements backoff. Large crawls may need to be split into smaller jobs to avoid throttling or skipped URLs.

• Browser usage limits and job cancellation: Accounts on Workers free plans are capped at 10 minutes of browser time per day. Exceeding this limit results in a cancelled_due_to_limits status. To avoid that, you can upgrade to a paid plan.

How to Use the Cloudflare Crawl Endpoint: Step-by-Step Tutorial

In this guided section, I’ll show you how to use the Cloudflare Crawl Endpoint to crawl a website in Python. The target site will be the “Quotes to Scrape” sandbox. The goal here is to demonstrate how to use the API, rather than actually collecting relevant data.

Follow the instructions below!

Prerequisites

To follow this tutorial section, make sure you have:

• Your Cloudflare account ID at hand.

• A Cloudflare API token with the “Browser Rendering - Edit” permission.

For the sake of simplicity and to keep this tutorial concise, I’ll assume you already have a Python project set up with requests installed. That said, you can use any programming language and any HTTP client, because the high-level logic remains the same.

Step #1: Set Up the Configurations

Start by importing the required libraries and reading the necessary secrets (your Cloudflare API token and account ID). Use these secrets to prepare the Cloudflare Crawl base URL and headers. Also, specify the starting target URL as a constant.

import requests
import time
import json

# The Cloudflare secrets required for authenticating Crawl API calls
CLOUDFLARE_API_TOKEN = "<YOUR_CLOUDFLARE_API_TOKEN>"
CLOUDFLARE_ACCOUNT_ID = "<YOUR_CLOUDFLARE_ACCOUNT_ID>"

# The base URL used for all Crawl API calls
CLOUDFLARE_CRAWL_BASE_URL = f"https://api.cloudflare.com/client/v4/accounts/{CLOUDFLARE_ACCOUNT_ID}/browser-rendering/crawl"

# Common headers shared by all API calls
HEADERS = {
    "Authorization": f"Bearer {CLOUDFLARE_API_TOKEN}",
    "Content-Type": "application/json"
}

# The URL to crawl
START_URL = "https://www.ssense.com/en-us/men/product/acne-studios/silver-folded-leather-wallet/18169981"

Tip: In a production script, read the Cloudflare API token and account ID from environment variables rather than hardcoding them.

Step #2: Trigger the Crawling Job

Define a start_crawl() function to send a POST request to Cloudflare’s Crawl API:

def start_crawl(start_url):
    # Customize according to your needs
    payload = {
        "url": start_url,
        "limit": 20,
        "depth": 2,
        "formats": ["markdown"],
        "render": True
    }

    response = requests.post(CLOUDFLARE_CRAWL_BASE_URL, headers=HEADERS, json=payload)
    response.raise_for_status()

    job_id = response.json()["result"]
    return job_id

This creates a new crawling job for the target URL. Then, it returns a job_id that identifies this specific crawl.

Tip: In a production-level script, make the payload object configurable via function input arguments for greater flexibility and reusability.

Step #3: Poll Over the Job

Next, add a wait_for_completion() function to repeatedly check the job status every few seconds until the crawl finishes or times out:

def wait_for_completion(job_id, max_attempts=60, delay=5):
    status_url = f"{CLOUDFLARE_CRAWL_BASE_URL}/{job_id}?limit=1"

    for attempt in range(max_attempts):
        res = requests.get(status_url, headers=HEADERS)
        res.raise_for_status()

        status = res.json()["result"]["status"]
        print(f"Attempt {attempt+1}: {status}")

        if status != "running":
            return status

        time.sleep(delay)

    raise Exception("Timeout waiting for crawl job")

This makes GET calls to the Cloudflare /crawl endpoint. It ensures you’re waiting for the task to complete processing before fetching the crawled records.

Tip: The limit=1 query parameter is recommended to restrict the number of retrieved records, keeping the response lightweight. After all, at this stage, you’re only interested in checking the job status, not in retrieving the actual output data.

Check the TWSC YouTube Channel

Step #4: Get the Crawled Content Pages

Build a fetch_records() function to collect all crawled pages:

def fetch_records(job_id):
    print("Fetching results...")

    all_records = []
    cursor = None

    while True:
        url = f"{CLOUDFLARE_CRAWL_BASE_URL}/{job_id}"
        # Getting 10 records at a time
        params = {
            "limit": 10
        }

        if cursor:
            params["cursor"] = cursor

        response = requests.get(url, headers=HEADERS, params=params)
        response.raise_for_status()

        result = response.json()["result"]
        records = result.get("records", [])

        all_records.extend(records)
        print(f"+{len(records)} records (Total: {len(all_records)})")

        cursor = result.get("cursor")
        if not cursor:
            break

    return all_records

This handles pagination using a cursor, accessing records in batches (10 per request) until all results are returned.

Step #5: Put It All Together

Finally, in the main() function, orchestrate the workflow:

1. Start the crawl

2. Wait for completion

3. Fetch all results

Then, you can export the crawled records to a local JSON file for further use, store the retrieved data in a database, process it there, etc.

def main():
    job_id = start_crawl(START_URL)
    status = wait_for_completion(job_id)

    if status != "completed":
        raise Exception(f"Crawl failed: {status}")

    print("Crawl completed!")

    records = fetch_records(job_id)

    # Export the crawled pages to an output JSON file
    with open("records.json", "w", encoding="utf-8") as f:
        json.dump(records, f, indent=2, ensure_ascii=False)

if __name__ == "__main__":
    main()

Step #6: Complete Code

This is what your Python script for interacting with the Cloudflare Crawl API will look like:

# pip install requests

import requests
import time
import json

# The Cloudflare secrets required for authenticating Crawl API calls
CLOUDFLARE_API_TOKEN = "<YOUR_CLOUDFLARE_API_TOKEN>"
CLOUDFLARE_ACCOUNT_ID = "<YOUR_CLOUDFLARE_ACCOUNT_ID>"

# The base URL used for all Crawl API calls
CLOUDFLARE_CRAWL_BASE_URL = f"https://api.cloudflare.com/client/v4/accounts/{CLOUDFLARE_ACCOUNT_ID}/browser-rendering/crawl"

# Common headers shared by all API calls
HEADERS = {
    "Authorization": f"Bearer {CLOUDFLARE_API_TOKEN}",
    "Content-Type": "application/json"
}

# The URL to crawl
START_URL = "http://quotes.toscrape.com/"

def start_crawl(start_url):
    """
    Triggers the Cloudflare Crawl API job
    """

    # Customize according to your needs
    payload = {
        "url": start_url,
        "limit": 20,
        "depth": 2,
        "formats": ["markdown"],
        "render": True
    }

    response = requests.post(CLOUDFLARE_CRAWL_BASE_URL, headers=HEADERS, json=payload)
    response.raise_for_status()

    job_id = response.json()["result"]
    return job_id

def wait_for_completion(job_id, max_attempts=60, delay=5):
    """
    Waits for the crawling task to complete
    """

    status_url = f"{CLOUDFLARE_CRAWL_BASE_URL}/{job_id}?limit=1"

    for attempt in range(max_attempts):
        res = requests.get(status_url, headers=HEADERS)
        res.raise_for_status()

        status = res.json()["result"]["status"]
        print(f"Attempt {attempt+1}: {status}")

        if status != "running":
            return status

        time.sleep(delay)

    raise Exception("Timeout waiting for crawl job")

def fetch_records(job_id):
    """
    Collects all records from the paginated results
    """

    print("Fetching results...")

    all_records = []
    cursor = None

    while True:
        url = f"{CLOUDFLARE_CRAWL_BASE_URL}/{job_id}"
        # Getting 10 records at a time
        params = {
            "limit": 10
        }

        if cursor:
            params["cursor"] = cursor

        response = requests.get(url, headers=HEADERS, params=params)
        response.raise_for_status()

        result = response.json()["result"]
        records = result.get("records", [])

        all_records.extend(records)
        print(f"+{len(records)} records (Total: {len(all_records)})")

        cursor = result.get("cursor")
        if not cursor:
            break

    return all_records

def main():
    job_id = start_crawl(START_URL)
    status = wait_for_completion(job_id)

    if status != "completed":
        raise Exception(f"Crawl failed: {status}")

    print("Crawl completed!")

    records = fetch_records(job_id)

    # Export the crawled pages to an output JSON file
    with open("records.json", "w", encoding="utf-8") as f:
        json.dump(records, f, indent=2, ensure_ascii=False)

if __name__ == "__main__":
    main()

Step #7: Test the Script

Launch the script, and it’ll produce an output like this:

The polling mechanism required 5 attempts (~25 seconds), and the API discovered and retrieved 22 pages.

A records.json file will appear in your project directory. Open it, and you’ll see:

Notice how the “Quotes to Scrape” entries contain a markdown field with the Markdown version of the page. Instead, external links like Zyte’s homepage and Goodreads.com are skipped, since includeExternalLinks is set to false by default. In other words, the Cloudflare Crawl API doesn’t automatically attempt to fetch data from different domains than the target source URL.

Et voilà! Implementation complete.

Need help with your scraping project?

Benchmark Against Protected Websites

Cool! The Cloudflare Crawl endpoint works like a charm and is easy to use. However, I was particularly concerned about its documented limitations and wanted to verify whether they actually hold up in practice…

So, I ran tests against several well-known sites protected by common WAF and anti-bot solutions (from different providers). Here’s a summary of the results:

As you can tell, the limitations are very real. The results are quite discouraging: the Cloudflare Crawl API failed against all anti-bot–protected websites I tested.

So, is this solution reliable for web scraping? When (and how) should you actually use it? Let me break that down in a final comment!

Final Comment

In this article, I introduced you to one of the newest tools in Cloudflare’s growing ecosystem: the Crawl API! This endpoint is designed to help you crawl entire websites using distributed crawling tasks running on Cloudflare’s infrastructure.

Sure, the crawling mechanism works and is easy to launch, control, and implement. With just a few lines of code, you can get started. Still, several concerns should be raised:

1. Opaque pricing: Costs are tied to resource usage rather than the number of pages crawled, making them harder to predict.

2. Fixed User-Agent: The API doesn’t allow User-Agent customization, meaning even basic server-side checks can block it.

3. Limited effectiveness on protected sites: The API has an intended very low success rate against anti-bot–protected websites (unless you specify in Cloudflare Bot Protection settings that you allow it against your site).

4. Rate limiting constraints: It strictly respects robots.txt directives and crawl delays, which can significantly slow or limit large crawls.

In simple terms, if you want to use it for general-purpose, large-scale web crawling, I wouldn’t recommend it. The market offers more effective solutions that can actually bypass anti-bot limitations. Plus, remember that around 35% of the entire Internet is estimated to be protected against bots (i.e., you won’t be able to crawl it with this API).

Yet, if you know the target site is not protected, budget isn’t a concern, and you want to remain (overly?) ethical and compliant, the Cloudflare Crawl API can be an option.

I hope this breakdown helps you better understand this new solution and make an informed decision on whether to adopt it. Lastly, remember that the Cloudflare Crawl API is still in beta, so things may change soon. Just keep an eye on the docs for updates. Until next time!

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

The short answer is that a couple of tools finished the task, just one with any consistency. The story behind why is worth telling, because it explains what is happening at the intersection of AI agents and web data right now. We ran a similar exercise against Cloudflare earlier this year, and the conclusion is broadly the same: each anti-bot needs its own answer, and the answer changes every quarter.

From workflows to agents, and why that changes the data problem

Most code shipped under the AI banner is not really agentic. It is workflow code with an LLM dropped into a slot: generate a summary here, classify a record there, draft an email at the end. The control flow is hard-coded, and the model is one component among many.

The definition of an agent is quite different. The model decides the next action, observes the outcome, and decides again. The control flow lives inside the loop, not outside it. The agent has goals rather than scripts, and it picks tools and steps based on what it sees. That is what makes the engineering interesting, that is what makes it hard, and that is what sometimes makes it unreliable.

It also forces a different relationship with data. An agent that only sees its training corpus is stuck in the past. To make decisions worth anything, it has to read prices that change daily, stocks that move minute by minute, listings that did not exist last week. Some of that data sits behind APIs. Most of it does not. The web is still the largest and most current dataset in the world, and most of it is reachable only through a browser. So if we want our agents to act on real information, we have to give them a way to browse: opening a page, reading it, clicking a link, typing into a search bar, following a result, filling a form, all on sites that were never built for machines.

Start your scraping journey with Byteful: 10GB New Customer Trial | Use TWSC for 15% OFF | $1.75/GB Residential Data | ISP Proxies in 15+ Countries

Claim your 10GB here

This is the constraint that produced the wave of “agentic browser” launches we have seen over the last twelve months. Y Combinator alone has backed a long string of them. Hyperbrowser (S21) was an early entry: scalable cloud browser infrastructure with built-in CAPTCHA solving, proxy management, and now a multi-agent playground. The newer cohort followed the agent wave more directly: BrowserOS (S24) is an open-source agentic browser that runs the agent locally on the user’s machine; Browser Use (W25) offers an open-source agent loop on top of Playwright, plus a cloud version. Skyvern is a self-hostable browser agent that uses an LLM and computer vision instead of fixed selectors. Outside the YC pipeline, Lightpanda is doing something different again, a headless browser engine written from scratch in Zig and aimed squarely at agents and crawlers (claiming roughly 9x faster execution and 16x lower memory than Chrome). It fits the “browser built for machines” line of thought we covered in Rethinking the web browser earlier this year. Browserbase ships a managed browser plus Stagehand for natural-language automation. And the big AI labs are now in the same space: OpenAI shipped Operator and the ChatGPT Atlas browser, Anthropic shipped Computer Use, Perplexity launched Comet. Each project attacks the same problem from a slightly different angle, but the goal is identical: a browser an agent can drive without immediately tripping every detection mechanism on the other side.

Your scraping workflows deserve a proxy infrastructure that just works. With Swiftproxy on your side, consistency is built-in.

Try Swiftproxy today

The same problem scrapers have been chasing for a decade

For anyone who has worked in web data, none of this is new. The fight over whether a request looks human or automated has been going on as long as commercial scraping has existed. The product names have changed but the purpose not.

What has changed is who is selling the bypass. The companies that have spent years selling residential proxies and unblockers noticed quickly that the agentic boom is good for their business. They already have the IP networks, the fingerprint research, the bypass code, the cat-and-mouse experience. They know what TLS handshake Chrome sends in October 2025 and what it sent in October 2024. Pivoting all of that into a managed browser is a smaller leap than building one from scratch. Bright Data, Oxylabs, Rayobyte, ZenRows have all added a managed browser product alongside the proxy.

The other side of the line is moving in the opposite direction. Bot traffic has grown faster than human traffic for years, and the operators of large public sites care more about it than ever. DataDome, Cloudflare Bot Management, Akamai Bot Manager, HUMAN, Kasada: every one of them ships updates that target the exact tools we just listed. Fingerprint checks get stricter. Behavioral models get more sensitive. The JavaScript challenge changes shape every few weeks. There is no silver bullet, and there is no tool, browser, proxy, or service that bypasses every anti-bot on every site at all times. Anyone who claims otherwise is selling something that worked last quarter and might still work this week. The useful question is what works on a given target, today, at what cost.

Check the TWSC YouTube Channel

Picking a hard target

To answer that question concretely, we needed a target where the anti-bot was good and the signal was clean. We picked leroymerlin.fr, the French DIY retailer. Leroy Merlin runs DataDome standalone, with no other anti-bot layer on top, so attribution is straightforward. It also runs one of the more verbose DataDome configurations we have come across: response headers expose x-datadome-riskscore, x-datadome-protection, x-datadome-cid, and x-datadome-endpointid. Most DataDome-protected sites only show us the outcome. Here we see the score the engine assigns at every request, which is rare and very useful when comparing tools side by side.

The task we picked is small but realistic. From the homepage, the agent has to type “ampoule B22 led blanc” into the search bar, click the first product result, and add the product to the cart. Four steps. We dropped the login step on purpose: leroymerlin.fr requires an OTP to sign in, and we did not want OTP friction to confound an anti-bot test.

A run is a pass if the agent reaches the cart confirmation. Otherwise we record where it stopped and what DataDome said about it. Each tool runs ten times back to back, and we aggregate the results. Tools that support an external proxy use the same residential pool: Bright Data residential FR for the Bright Data runs, Geonode residential FR for the Geonode runs. Tools that ship their own proxy use it. The reason behind two different providers was because we wanted to diversify the IP addresses, to be sure that blocks were not a matter of IP reputation.

Need help with your scraping project?

The contestants

As you’ve seen before, the browser landscape is quite crowded and we could not cover all the tools. We picked four open-source projects and seven commercial products. Let’s start with the open source.

Camoufox is the stealth Firefox fork most people in the scraping world have already met (we introduced it on TWSC back in September 2024). It rotates real-world fingerprints, patches the obvious automation tells, and ships a Playwright-compatible API. We pair it with both Bright Data and Geonode residential proxies in France.

Pydoll takes a different route: it drives Chromium directly over CDP without WebDriver, with built-in humanized cursor movement and typing. Importantly, Pydoll implements an explicit Fetch.authRequired handler, which lets it authenticate proxies that require Basic auth.

Scrapling is a higher-level Python library. We use it in two modes. DynamicFetcher launches vanilla Playwright Chromium driven by Scrapling’s session manager. StealthyFetcher does the same, but under the hood uses an improved and customized version of patchright, a stealth-patched Playwright fork. Each gets its own row in the comparison.

RayoBrowse is the self-hosted stealth Chromium fork from Rayobyte, distributed as a Docker container that exposes a CDP endpoint on port 9222. Here we hit a wall worth flagging: for some reason RayoBrowse could not use the Bright Data residential proxy in our setup. Every navigation through that proxy failed instantly, even though the same credentials worked fine through curl from inside the same container. The same RayoBrowse setup worked fine with Geonode. We did not isolate the root cause, so we report RayoBrowse on Geonode only.

The commercial side is more crowded.

Browser Use exists in two flavors, and we tested both. The cloud version is the managed Browser Use, with its own residential proxy, its own stealth fingerprinting, and a fixed set of supported models; we drove it once in raw CDP mode (we steer it ourselves with Playwright) and once in agent mode (we hand the LLM the task in natural language and let it plan the steps).

Browserbase is a managed Chromium with optional residential proxies, Cloudflare Web Bot Auth verification, and the Stagehand agent SDK. We discovered during the test that the free tier excludes proxies entirely; without one, the session egresses from a US datacenter. We left this configuration in the test because it is what a free user would experience.

Browserless is a managed browser-as-a-service whose anti-bot story is a stealth path (/chromium/stealth) plus optional residential proxies for paid plans. The free plan caps sessions at 60 seconds, which is tight for a four-step flow. We tested it with the built-in residential proxy targeting France, and tried to test it with our external proxies via the externalProxyServer parameter; the external mode failed at connection time on every run, in the same Chromium-side authentication way that broke RayoBrowse, so we drop those configurations from the comparison.

ZenRows Scraping Browser is a managed Chromium with a built-in residential proxy network and built-in CAPTCHA solving; we connect via the WSS endpoint with proxy_country=fr to get a French exit point.

Bright Data Browser API sits at the other end of the same product category: a managed Chromium with built-in residential rotation and CAPTCHA solving, on a dedicated Browser API zone we configured on their dashboard.

As always, the code can be found in our GitHub repository reserved to paying users, inside the folder 103.BROWSERS.

What we had to fix before the numbers made sense

Read more

This means that if you route all your traffic through a single egress IP, you will be rate-limited in seconds and blacklisted in minutes. To survive at scale, you need to distribute your requests across a massive pool of IP addresses.

Traditionally, the web scraping industry has solved this issue thanks to commercial proxy providers. However, this is not the only approach. This article responds to the following question: “Is there a way to scrape at scale without burning budget on proxies?”

The answer is yes. But let’s be clear from the beginning: This approach is not a universal silver bullet. Let’s see how it works, how to build it, and what its limitations are.

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

The Typical Solution for Scraping at Scale: Proxy Provider Services

Let’s start this discussion with the typical choice for scraping at scale. IP bans and rate limits are the #1 operational problem in scraping, especially at scale. The typical solution every web scraping engineer integrates is using proxy servers, for a simple reason: proxies act as intermediaries between your scrapers and the Internet, avoiding your scrapers from getting banned. To do so, companies buy proxy IPs from proxy providers. The most common categories, both with their flaws, are the following:

• Datacenter proxies: These are cheap and fast, but their ASNs(Autonomous System Numbers) are heavily scrutinized. WAFs maintain databases of known datacenter CIDR (Classless Inter-Domain Routing) blocks, so hitting a target with a static list of 100 datacenter proxies usually results in those IPs being flagged and blocked within hours.

• Residential proxies: These route traffic through actual consumer devices. They have highly trusted IP reputations, making them excellent for bypassing anti-bot systems. However, they are priced by bandwidth, so they are very expensive, especially when scraping at scale.

The main limitation of this approach is that it is highly expensive. So, what if you need to scrape at scale but don’t have enough budget for doing so?

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

An Alternative Approach: Scraping at Scale With Dedicated Infrastructure

To escape metered billing, you can move egress back to dedicated infrastructure. But before presenting the solution, let’s first point out shortly what happens when you buy and use proxies, at the infrastructure level.

Buying Proxies Means Delegating Your Infrastructure

When you buy proxies from providers, you are delegating 100% of your infrastructure. When your scrapers make the requests, under the hood, the proxy provider connects to a gateway, which is a massive load balancer controlled entirely by the provider itself.

Let’s consider the case of residential proxies, for simplicity. Behind the gateway is a peer-to-peer (P2P) network of millions of consumer devices that the provider has acquired bandwidth from. When your request hits the gateway, their proprietary routing algorithm decides which consumer device in which country will act as your final exit node.

The second you route traffic through their gateway is the exact moment where you delegate the 100% of your scraping infrastructure.

Your scraping workflows deserve a proxy infrastructure that just works. With Swiftproxy on your side, consistency is built-in.

Try Swiftproxy today

NyxProxy: The Infrastructural Solution

NyxProxy is a self-hosted HTTP/SOCKS5 proxy server that exploits a well-known IPv6 networking trick: When a cloud provider gives you a /64 subnet, you legally own 18.4 quintillion IPv6 addresses.

Let’s explain the number and the trick around IPv6s. An IPv6 address looks like this:

 2a05:f480:1800:25db:0000:0000:0000:0001

They are 128 bits long. That gives 2^128 possible addresses. The number is so large that the designers said: “We can afford to give every organization a massive block and never worry about running out”.

Now, here is the trick. An IPv6 address is split into two halves, 64 bits each:

2a05:f480:1800:25db : 0000:0000:0000:0001
|___________________|   |_________________|
   Network prefix            Host part
   (your subnet)          (you control this)

The /64 notation means: the first 64 bits identify the network, the last 64 bits are yours to assign however you want. The last 64 bits can be any value from 0000:0000:0000:0000 to ffff:ffff:ffff:ffff: That’s 2^64 = 18.4 quintillion combinations. All valid addresses, all routable to your server.

Thanks to this trick, NyxProxy can assign a pool of those addresses to your network interface at startup, then rotate your outgoing traffic across them. This means having a fresh IP per request. The tool handles pool management, background rotation, NDP proxying via ndppd, and exposes a monitoring endpoint.

The best part is, indeed, in the NDP proxying. When your server uses a random address like 2a05:f480:1800:25db:a3f1:9922:beef:1234 as a source IP, your router upstream needs to know your server is responsible for that address. Otherwise, the response packets have nowhere to go.

IPv6 uses NDP (Neighbor Discovery Protocol) for this. The router sends an NDP query: “who has 2a05:f480:1800:25db:a3f1:9922:beef:1234?” and your server must answer.

ndppd (NDP Proxy Daemon) runs on your server and answers those queries automatically for your entire /64 subnet, essentially saying “yes, all of those addresses are mine”. Without it, your packets go out, but responses never come back.

Below is a summary schema of how this whole process works:

Provider gives you:  2a05:f480:1800:25db::/64
                     ↓
Your server can use: 2a05:f480:1800:25db:[anything]
                     ↓
NyxProxy assigns 200 random IPs to your interface
                     ↓
Each outgoing request binds to a different one
                     ↓
Target sees 200 different source IPs
                     ↓
ndppd makes sure responses route back correctly

Check the TWSC YouTube Channel

How To Use NyxProxy

Let’s now see how to use NyxProxy with a practical implementation.

Environment Setup & Prerequisites

To replicate this tutorial for deploying NyxProxy and utilizing it in your scraping scripts, you must have the following system and hardware requirements:

• Hardware: A Virtual Private Server (VPS) or bare-metal server with at least 512 MB of RAM and 100 MB of disk space. Supported architectures are amd64 or arm64.

• Subnet: A cloud provider that natively delegates a full IPv6 /64 subnet to your network interface. Note that not all the VPS providers are supported: Check out the NyxProxy documentation to learn more about supported VPSs.

• Operating system: A modern Linux distribution, specifically Ubuntu or Debian, to ensure compatibility with the automated setup scripts and sysctl kernel modifications.

• Python: Python 3.7 or higher installed on your local machine to run the scraping scripts.

To get your server ready to run the proxy daemon, you need to verify your IPv6 setup and gain root access. Ensure you are logged into your VPS via SSH as the root user, or have sudo privileges.

First, verify that your server has a globally routable IPv6 /64 subnet assigned to it. You can check this by running the following command in your server’s terminal:

ip -6 addr show | grep "scope global"

If done correctly, you should see an output similar to the following:

inet6 2a05:f480:1800:25db::1/64 scope global

If you do not see a /64 subnet, you will not be able to rotate IPs, and you must review your cloud provider’s network settings.

Next, prepare your local development environment. Suppose you call the main folder of your Python project nyxproxy_scraper/. At the end of this step, the folder will have the following structure:

nyxproxy_scraper/
    ├── main.py
    └── venv/

Where:

• main.py is the Python file that will store your proxy request logic.

• venv/ contains the standard Python virtual environment.

You can create the venv/ virtual environment directory like so:

python -m venv venv

To activate it, on Windows, run:

venv\Scripts\activate

Equivalently, on macOS and Linux, execute:

source venv/bin/activate

As a final prerequisite, install the Requests library in your activated virtual environment so your Python script can make HTTP calls:

pip install requests

Well done! You are now ready to test and use Nyxproxy.

Need help with your scraping project?

Installing and Configuring NyxProxy

NyxProxy provides a quick setup script that handles the infrastructural heavy lifting. It auto-detects your network interface, installs ndppd, tweaks the Linux kernel parameters via sysctl to allow non-local binding, and downloads the compiled Go binary.

You can launch it with the following single command:

wget <https://raw.githubusercontent.com/jannik-schroeder/nyxproxy-oss/main/scripts/quick-setup.sh> && chmod +x quick-setup.sh && sudo ./quick-setup.sh

During the setup, you will be prompted to configure your proxy credentials and set your rotation rules. Behind the scenes, the script generates a config.yaml file. Let’s look at the crucial subset of that configuration:

network:
  rotate_ipv6: true
  ipv6_subnet: "2a05:f480:1800:25db::/64"

  # The rotation mechanics:
  ipv6_pool_size: 200
  ipv6_max_usage: 100
  ipv6_max_age: 30

Below is an explanation of what these three parameters mean for your scraping pipeline:

• ipv6_pool_size: NyxProxy keeps 200 mathematically unique IPs “hot” and bound to your network interface at any given time. This keeps proxy startup times under 100ms while maintaining IP diversity.

• ipv6_max_usage: After a specific IP has been utilized for 100 requests, it is considered “burned.” NyxProxy destroys the route and spins up a fresh address to dynamically replace it.

• ipv6_max_age: If an IP hasn’t hit 100 requests but has been alive for 30 minutes, it gets forcefully rotated out. This prevents time-based algorithmic tracking by the target WAF.

Once the daemon is running as a systemd service, your VPS is officially acting as a rotating proxy gateway. When NyxProxy receives a scraper request, the underlying Go binary takes over. It looks at its internal memory, picks one of the 200 rotating IPv6 addresses in its pool, and binds to that specific address to establish the outbound connection.

The expected output is as follows:

IPv6 rotation mode: IP Pool with dynamic rotation
  Interface: enp1s0
  Subnet: 2a05:f480:1800:25db::/64
  Pool size: 200 IPs
  Rotation: Every 100 uses or 30m0s
  Initializing IP pool...
  Progress: 50/200 IPs added
  Progress: 100/200 IPs added
  Progress: 150/200 IPs added
  Progress: 200/200 IPs added
  IP pool ready with 200 addresses
  Background IP rotation started

Starting https proxy on 0.0.0.0:8080 (Protocol: IPv6)

Testing the Proxy Logic

At this point, NyxProxy has done its job. To verify it works correctly, you can use the following Python script that hits api6.ipify.org, which is an API that simply bounces back the IP address it sees:

import requests

# Point this to your VPS IP and the credentials you set during setup
proxies = {
    'http': '<http://admin:password@your-vps-ip:8080>',
    'https': '<http://admin:password@your-vps-ip:8080>'
}

# Test 5 consecutive scraping requests
for i in range(5):
    response = requests.get('<https://api6.ipify.org>', proxies=proxies)
    print(f"Request {i+1}: Target sees IP -> {response.text}")

(NOTE: If you are already familiar with ipify.org, note that the “api6” prefix can be used for IPv6 requests only.)

The result should be similar to the following:

Request 1: Target sees IP -> 2a05:f480:1800:25db:1a2b:3c4d:5e6f:7890
Request 2: Target sees IP -> 2a05:f480:1800:25db:9988:7766:5544:3322
Request 3: Target sees IP -> 2a05:f480:1800:25db:aaaa:bbbb:cccc:dddd
Request 4: Target sees IP -> 2a05:f480:1800:25db:1122:3344:5566:7788
Request 5: Target sees IP -> 2a05:f480:1800:25db:dead:beef:cafe:babe

This shows that every single HTTP request utilizes a completely different, globally routable IPv6 address generated from your subnet block. To the target server, these look like entirely distinct users connecting from across the internet.

Perfect! You have successfully built a self-healing, infinitely rotating proxy pool without handing over your budget for metered residential bandwidth.

The Illusion of Infinity: Critical Limitations of IPv6 Subnet Rotation

At this point, you may think you have found a solution to all of your budgeting problems for scraping at scale. But before you tear down your commercial proxy infrastructure, you must understand that a $5/Mo VPS and an open-source rotation daemon are not a universal silver bullet. If it were that simple, the commercial proxy industry would not exist.

This architecture has the following main limitation:

• The IPv4 compatibility wall: This entire architecture is built on one absolute prerequisite: Your target endpoint must support IPv6. If you are scraping legacy enterprise systems or platforms that haven’t migrated to dual-stack networking, this setup is useless. You cannot route an IPv6 packet to an IPv4-only server.

• Subnet-level bans (/64 prefix blocking): Enterprise WAFs are fully aware of IPv6 prefix delegation standards. They know that hosting providers allocate a /64 subnet to a single client. If their heuristics detect highly concurrent behavioral patterns (like missing browser fingerprints or anomalous TLS handshakes) originating from 2a05:f480...:1a2b, they will ban the entire /64 CIDR block. Once your /64 prefix is banned, all 18 quintillion of your “infinite” IPs are simultaneously dead. To recover, you must physically destroy the VPS and provision a new one in a different IP range.

• ASN reputation: No matter how many IPs you rotate, your traffic still originates from a Datacenter Autonomous System Number (ASN). Target firewalls assign a baseline trust score to every ASN. Traffic originating from a Datacenter ASN always starts with a highly degraded trust score compared to a Residential ASN. For highly restrictive targets, any request from a datacenter IP is instantly met with an unpassable CAPTCHA or a hard 403 Forbidden, regardless of whether it’s IPv4 or IPv6.

• nf_conntrack and hardware exhaustion: You cannot push enterprise-grade throughput on a $5, 1-vCPU server without consequence. Rotating thousands of IPv6 addresses requires the Linux kernel to aggressively maintain the nf_conntrack table and the NDP proxy table. At high concurrencies, the overhead of establishing, tracking, and tearing down thousands of TCP sockets across rotating interfaces will exhaust the memory or CPU of a low-tier VPS. The kernel will begin dropping packets natively, your latency will spike to useless levels, and your scrapers will be greeted with errors.

Conclusion

In this article, you learned how to leverage your hosting provider’s IPv6 /64 subnets to build an infinitely rotating proxy pool with NyxProxy, escaping the metered billing of residential proxy networks.

The competitive advantage of engineering your own proxy infrastructure is in your unit economics and architectural control. However, you also learned that this solution is not a universal silver bullet for every scraping scenario: It comes with trade-offs and constraints.

So, let us know: Have you already experimented with bare-metal IPv6 rotation for your scraping pipelines? What targets did it work best for? Let’s discuss in the comments!

Behind the scenes, it relies on hydration to make server-rendered pages interactive. And here’s the interesting part: the same mechanism that makes Next.js fast and popular also exposes a significant amount of structured data in the HTML sent by the server. From a scraping perspective, that’s a huge opportunity!

In this post, I’ll show you a simple trick to scrape data from virtually any Next.js website. Follow along as I break down how it works and how you can apply it yourself.

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

Next.js in Numbers

Next.js needs no introduction, but it’s worth giving some context to truly understand how popular it is (and therefore how useful the trick I’m about to present for Next.js web scraping can be):

• According to the 2025 Stack Overflow Developer Survey, 20.8% of respondents used Next.js extensively over the past year.

• Next.js is the 14th largest repository on GitHub, with over 138k stars (and still growing!).

• According to W3Techs, Next.js has a 2.9% market share among JavaScript libraries.

• Major brands such as Nike, Stripe, and Notion have chosen this full-stack framework to build their official websites.

Before Getting Started: A Bit of Context on Hydration

I know you probably just want the trick… Still, let me take a minute to explain why it works in the first place, why it’s even possible, and what kind of data you’ll actually retrieve with it!

Start your scraping journey with Byteful: 10GB New Customer Trial | Use TWSC for 15% OFF | $1.75/GB Residential Data | ISP Proxies in 15+ Countries

Claim your 10GB here

What Is Hydration?

Hydration is the process that makes a server-rendered page interactive in the browser.

Frameworks like Next.js, Remix, Nuxt, and SvelteKit employ this mechanism to combine the performance benefits of server-side rendering (SSR) with the interactivity of client-side applications.

The idea is that the server first sends fully rendered static HTML to the browser. Then, hydration happens next.

The browser downloads the JavaScript bundle, and the frontend framework reconstructs the component tree in memory, attaches event listeners, and links that virtual tree to the existing DOM instead of re-rendering it from scratch. The result is a fully interactive application built on top of server-rendered HTML.

How Does the Hydration Mechanism Work?

It’s now clear that in Next.js and similar frameworks, hydration is the process where a static, server-rendered HTML page “comes to life” and becomes fully interactive in the browser. But what’s actually happening under the hood?

At a high level, hydration is a 3-step process:

1. The server generates and sends a fully rendered HTML snapshot. The user immediately sees the content (great for First Contentful Paint). At this point, though, the page is just static HTML. Buttons, forms, and other interactive elements are visible, but they don’t work yet because no JavaScript is attached.

2. The client’s browser downloads the JavaScript bundle (which includes React and your frontend application code) and executes it.

3. React rebuilds the component tree in memory and attaches event listeners to the existing DOM nodes. Instead of discarding the HTML and re-rendering everything from scratch, React “hydrates” the existing markup, meaning it reuses it and wires it up with state and interactivity.

Once hydration completes, the page behaves like a normal single-page application: it responds to clicks, manages state, and updates dynamically.

And here’s an important detail: if the browser doesn’t support JavaScript (or it fails to load), the user still sees the server-rendered HTML. It won’t be interactive, but the core content is there. That’s great for SEO and perceived performance!

Why It Matters for Scraping Next.js (and Other Full-Stack Frameworks…)

The key insight you need to understand is simple: hydration requires data, and that data must be embedded somewhere in the HTML sent by the server!

In Next.js, when the server renders a page, it doesn’t only send markup. It also serializes the data required to rebuild the React component tree on the client. That serialized payload is embedded directly into the page’s HTML.

That’s exactly why hydration matters for scraping. Instead of parsing the DOM or simulating user interactions through browser automation, you can extract the structured data that React itself uses to hydrate the page.

In many cases, hydration data is cleaner and easier to parse than the rendered HTML. It can also contain more information than what’s visibly displayed on the page, including hidden and interesting metadata.

Keep in mind that this principle applies not only to Next.js! All other full-stack frameworks that rely on hydration, such as Remix, Nuxt, Angular Universal, and SvelteKit, tend to dehydrate state on the server and rehydrate it on the client.

So remember this simple rule. If a framework hydrates, it must serialize data. And if it serializes data into the HTML, you can scrape it.

How to Scrape Next.js Websites: 2 Approaches

The approach to scraping Next.js by targeting hydration data depends on how that data is embedded in the HTML generated on the server side.

I won’t go too deep into framework internals here (if you’re a Next.js dev, you already know things shift depending on whether you’re using the App Router or the Pages Router), but there are essentially two scenarios you’ll run into.

In this section, I’ll walk through both of them and show you exactly how I retrieve data from each!

Check the TWSC YouTube Channel

Approach #1: Target the __NEXT_DATA__ Script

As a target site, I’ll use a Nike product page as a reference:

That’s actually a great example because Nike.com is even showcased on the Next.js homepage as a real-world site built with the framework.

Now, right-click on the page and select the “Inspect” option in your browser to open the DevTools. Scroll through the DOM and get familiar with the page structure. If the Next.js site is using the Pages Router, you’ll notice a <script> tag with the id __NEXT_DATA__ containing a large JSON blob:

That JSON data is precisely the hydration data I was referring to earlier.

When a site uses the Pages Router approach in Next.js, the server embeds all the page data directly into that <script> tag. From a scraping perspective, that’s gold, as the data is already structured and ready to be captured.

Below’s a simple JavaScript snippet to extract it:

const hydartionScript = document.querySelector("#__NEXT_DATA__")
const hydrationData = JSON.parse(hydartionScript.innerHTML)
console.log(hydrationData)

What’s happening here is straightforward. The JS script:

• Selects the <script> element with id __NEXT_DATA__.

• Reads its inner HTML (which is a JSON string).

• Parses it into a JavaScript object.

• Logs it to the console.

Run this directly in the DevTools Console, and you’ll immediately see the result:

What’s interesting is how much structured data you get right away. This includes product details, images, metadata, and more. All is neatly organized, and it only took three lines of code!

If you want to store the JSON hydration object, just right-click the object in the Console and select the “Copy object” option:

From there, you can paste it wherever you need (e.g., into a local .json file, a MongoDB collection, etc.).

Need help with your scraping project?

Approach #2: Target the self.__next_f.push() Elements

Another, more complex approach to scraping Next.js involves pages built with the App Router.

Even if the App Router has been the recommended direction for a while, in my experience, it’s still not as widely adopted as the Pages Router. And honestly, that’s a bit of a gift for us (as scraping hydration data in App Router sites is definitely more complex!)

As a reference, let’s look at the “Business Overview” page on the OpenAI website, which is built with Next.js App Router:

Just like before, open DevTools and inspect the page. This time, focus on the <script> tags inside the <body>:

You’ll notice several scripts containing content like:

self.__next_f.push(<some_data>)

That “<some_data>” is serialized using the React Flight protocol for React Server Components (RSC). I won’t go too deep into the internals here (it’s a dense topic!), but what matters is that deserializing that data is not straightforward!

React Flight isn’t plain JSON. It mixes control records (HL, I, J, etc.), module references, streaming boundaries, and serialized model fragments into a transport format that React incrementally resolves at runtime.

You might think: “Why not just reuse the frontend deserialization library?” In practice, that doesn’t work well because:

• The client decoder (react-server-dom-webpack) expects a full React runtime.

• It relies on module maps and webpack IDs generated at build time.

• It resolves component references against the exact bundle that produced the stream.

• It assumes streaming semantics and internal React wiring.

Basically, outside that exact environment, you don’t have the module graph, build manifest, or hydration context. So even if you import the decoder, you can’t reconstruct the component tree the way the browser does.

There have been recent security issues in the React Flight payload deserialization system, highlighting just how sensitive and complex this layer is. For more details, refer to:

• Security Advisory: CVE-2025-66478

• Critical Security Vulnerability in React Server Components

Thus, instead of fighting the protocol, I’d simplify and accept that in this case, it’s better to extract the unparsed React Flight string data. Achieve that with the JS script below:

const nextFlightScripts = [...document.querySelectorAll("script")]
  .filter(script => script.textContent.includes("self.__next_f"))
  .map(script => script.textContent.trim())
console.log(nextFlightScripts)

This selects all <script> elements containing “self.__next_f” and builds an array of their raw contents.

Run it in the Console, and you’ll get an array of React Flight chunks:

From there, the simplest way to extract structured data is often to copy the array, feed it to an AI, and ask it to reconstruct a parsed JSON representation of the meaningful payload sections:

Is this more complicated than the __NEXT_DATA__ trick? Absolutely! Yet, it’s still a powerful way to access a large amount of page data with just a few lines of code.

Final Script to Quickly Access Data From Next.js Sites

If you combine the two approaches, you can build a production-ready script for brute-force hydration data scraping in Next.js:

// Pages Router approach (__NEXT_DATA__)
const hydrationScript = document.querySelector("#__NEXT_DATA__")
let nextData = null
if (hydrationScript) {
  try {
    nextData = JSON.parse(hydrationScript.textContent)
    console.log("__NEXT_DATA__ found:")
    console.log(nextData)
  } catch (err) {
    console.warn("Failed to parse __NEXT_DATA__:", err)
  }
} else {
  console.log("No __NEXT_DATA__ script found.")
}

// App Router approach (self.__next_f)
const nextFlightScripts = [...document.querySelectorAll("script")]
  .map(script => script.textContent.trim())
  .filter(content => content.includes("self.__next_f.push"))

if (nextFlightScripts.length > 0) {
  console.log("React Flight scripts found:")
  console.log(nextFlightScripts)
} else {
  console.log("No React Flight scripts found.")
}

To test it, just open the Console in DevTools, paste the script, and run it.

Important: The <script> components containing hydration data aren’t loaded dynamically via client-side rendering. They’re embedded directly in the HTML generated by the server.

That means you can:

1. Fetch the target Next.js-powered page with an HTTP client.

2. Parse the HTML using an HTML parsing library like Beautiful Soup or Cheerio.

3. Apply a similar version of the JavaScript script above, but adapt it to the API provided by your HTML parser.

In other words, this trick for scraping Next.js doesn’t only work in the browser DevTools. It also works perfectly in regular scraping scripts!

Pros and Cons of This Approach to Next.js Scraping

👍 Pros:

• Simple and effective, requiring only a few lines of code.

• Works on all Next.js websites (and, more generally, on most sites that rely on hydration).

• Can let you access more data than what is actually displayed on the page.

• No need for browser automation, waiting for client-side rendering, or simulating user interactions.

👎 Cons:

• You may only get partial data, meaning you might still need to complement it with a more traditional scraping approach.

• React Flight data is difficult to parse and may require custom logic or even AI-assisted parsing.

Conclusion

Here, I’ve shared a trick I personally documented years ago, and that still works to this day. It allows you to quickly scrape data from virtually any Next.js site by targeting the hydration data embedded in the HTML document generated by the server and sent to the client for rendering.

As you’ve seen, with just a few lines of JavaScript, you can extract hydration data from any Next.js-powered page. What you get back is clean, or at least almost clean, data that you can process directly in your data pipelines.

Instead of fighting the frontend, this Next.js web scraping approach helps you leverage the data the framework itself needs to function!

I hope you found this useful and insightful. If you have questions or thoughts, feel free to share them in the comments below!

But what is Polymarket, and how does it work?

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

What Polymarket is, and why it keeps making headlines

Polymarket is a prediction market, a platform where you buy and sell shares tied to the outcome of real-world events. If the event happens, your share pays $1. If it doesn’t, it pays $0. The trading price at any moment reflects what the market collectively believes the probability of that outcome is. You can bet on elections, geopolitics, sports, crypto prices, and increasingly anything else with a verifiable resolution.

It is the largest prediction market by volume, built on the Polygon blockchain. Its main competitor, Kalshi, operates as a CFTC-regulated exchange in the US. Both are attracting billions in volume, and Wall Street firms are now building dedicated trading desks around them.

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

The platform handled at least $7.5 billion in volume during 2025 (a conservative figure, since Polymarket volume is commonly double-counted due to how OrderFilled events are summed, and set a single-day record of $425 million in February 2026 when Iran-related markets resolved simultaneously. Those are not toy numbers. And with that kind of money flowing through, the headlines have followed.

In January 2026, a newly created Polymarket account invested $30,000 and walked away with $436,759 after correctly betting on Maduro’s removal from power. The account was created less than a week before the U.S. military operation, and the bulk of bids were placed hours before Trump’s announcement. In a separate case, an Israeli Air Force reservist was indicted for leaking classified details about a strike on Iran to guide Polymarket bets, netting roughly $244,000. A different trader has made nearly $1 million since 2024 from dozens of well-timed bets correctly predicting U.S. and Israeli military actions against Iran, winning 93% of five-figure wagers. These incidents triggered at least eight prediction market bills in Congress since January 2026, and federal prosecutors in Manhattan are actively exploring whether certain prediction market bets violate insider trading laws.

But insider trading is not the only way people make money on Polymarket. There is a quieter, more interesting story happening in parallel.

Start your scraping journey with Byteful: 10GB New Customer Trial | Use TWSC for 15% OFF | $1.75/GB Residential Data | ISP Proxies in 15+ Countries

Claim your 10GB here

The efficiency gap

The Efficient Market Hypothesis, formalized by Eugene Fama in the 1960s, states that asset prices reflect all available information, making it impossible to consistently beat the market. In traditional equity markets, this largely holds because massive institutional capital from hedge funds, pension funds, and proprietary trading firms constantly hunts for and eliminates mispricings. The S&P 500 trades roughly $500 billion daily. Any pricing error gets corrected in milliseconds by algorithms running in colocated data centers.

Polymarket’s individual markets often have only tens of thousands of dollars in liquidity. The ratio of “smart money” to “total market cap” is fundamentally different from equity markets, and that is why edges persist longer than they would on Wall Street. A 2025 study by IMDEA Networks Institute documented $40 million in arbitrage profits extracted from Polymarket alone between April 2024 and April 2025, analyzing 86 million bets. Arbitrage opportunities on the platform last an average of 4 seconds, with 73% of profits captured by bots, executing in under 100 milliseconds.

The institutional side is catching up. DRW is hiring dedicated prediction market traders at a $200,000 base salary. Susquehanna International Group became the first official market maker on Kalshi (a competing platform). Jump Trading is building specialized desks. But the market is not there yet. Liquidity is too thin for these firms to deploy serious capital without moving prices, leaving room for smaller, faster actors.

Check the TWSC YouTube Channel

Speed as edge: what people are building

I’ve been studying Polymarket for some months now, and I’ve probably ended up on a bubble on Instagram and other social media. I’m seeing a growing number of traders share systems designed to exploit exactly this kind of market inefficiency. The approaches vary, but the pattern is the same: faster information, faster execution, profit.

One notable case involves a trader who claims to use computer vision models processing live football match video feeds. His system watches the match in real time, detects events (goals, red cards, penalties) through frame analysis, and places bets on prediction markets seconds before the event registers on official data feeds and bookmaker odds adjust. He claims an 8-second advantage over other traders (unfortunately, I cannot find the post on Instagram about it anymore). Whether that specific claim holds up or not, this is nothing new: courtsiders have been doing this in tennis for years, attending live matches and transmitting scores faster than official data feeds reach bookmakers. In 2016, tennis umpires from Kazakhstan, Turkey, and Ukraine were banned for deliberately delaying score updates for courtside accomplices.

The same principle applies at a larger scale. Bill Benter built a multinomial logit model with over 120 variables per horse for Hong Kong racing and extracted over $1 billion between 1987 and 2001. Tony Bloom’s Starlizard syndicate employs 160 people to model Asian handicap football markets and reportedly generates 600 million GBP per year. On Polymarket itself, 14 of the top 20 most profitable wallets are bots. One bot turned $313 into $414,000 in a single month, exploiting temporal arbitrage in 15-minute crypto markets.

All of these systems share two requirements: data and speed. They need real-time access to market prices, order books, and event outcomes, and they need to act on that data faster than everyone else. All of this is possible because Polymarket provides a full set of APIs that can be used to operate programmatically on the platform.

Need help with your scraping project?

Polymarket’s API architecture

Polymarket exposes three distinct APIs, each serving a different purpose. Understanding which one to use and when is the first step toward building anything that trades or monitors this market.

Gamma API: market discovery

The Gamma API is the browsing layer. It returns human-readable market data: questions, descriptions, outcome prices, volume, liquidity, event metadata. No authentication required.

Base URL: https://gamma-api.polymarket.com

Key endpoints:

- GET /markets returns a paginated list of markets with filtering options (limit, offset, closed, tag_id)

- GET /markets/{id} returns a single market by ID or slug

- GET /events and GET /events/{id} return event-level data (events group related markets)

- GET /search?query=... performs keyword search across markets and events

A single call to /markets?limit=1&closed=false returns something like this:

{
  "id": "540816",
  "question": "Russia-Ukraine Ceasefire before GTA VI?",
  "conditionId": "0x9c1a953fe92c8357f1b646ba25d983aa83e90c525992db14fb726fa895cb5763",
  "outcomes": "[\"Yes\", \"No\"]",
  "outcomePrices": "[\"0.545\", \"0.455\"]",
  "volume": "1516211.89",
  "liquidity": "62104.61",
  "clobTokenIds": "[\"850149715...\", \"252731249...\"]"
}

The `clobTokenIds` field is the bridge to the trading layer. Each outcome (Yes/No) gets its own token ID, which is what you pass to the CLOB API to get real-time prices and order book data.

The Gamma API is rate-limited at roughly 60 requests per minute. It is useful for discovery and metadata, not for real-time price monitoring.

CLOB API: the order book

The CLOB (Central Limit Order Book) API is where trading happens. It has both public and authenticated endpoints.

Base URL: https://clob.polymarket.com

Public endpoints (no authentication):

- GET /price?token_id=X&side=BUY|SELL returns the current best price

- GET /midpoint?token_id=X returns the midpoint between best bid and ask

- GET /spread?token_id=X returns the current spread

- GET /book?token_id=X returns the full order book with all bids and asks

- GET /last-trade-price?token_id=X returns the last executed trade price

- GET /tick-size?token_id=X returns the minimum price increment

A call to /midpoint returns a minimal payload:

{"mid": "0.545"}

The /book endpoint returns the full depth:

{
  "market": "0x9c1a95...",
  "asset_id": "850149715...",
  "bids": [
    {"price": "0.54", "size": "15234.50"},
    {"price": "0.53", "size": "8920.00"}
  ],
  "asks": [
    {"price": "0.55", "size": "12100.00"},
    {"price": "0.56", "size": "6500.00"}
  ]
}

These public endpoints are what matters for price monitoring. They are lightweight, return small payloads, and have no authentication overhead.

Authenticated endpoints require a two-level authentication system:

Level 1 (L1) uses EIP-712 wallet signatures. You sign a structured message proving you control a specific Ethereum wallet address. This is a one-time operation that generates API credentials:

POST /auth/api-key
Headers: POLY_ADDRESS, POLY_SIGNATURE, POLY_TIMESTAMP, POLY_NONCE
Returns: { apiKey, secret, passphrase }

Level 2 (L2) uses HMAC-SHA256 signing on every request. Every authenticated call requires five headers: POLY_ADDRESS, POLY_SIGNATURE (computed HMAC of the request), POLY_TIMESTAMP, POLY_API_KEY, and POLY_PASSPHRASE.

Even with L2 auth, placing an order requires the user to sign the order payload locally with their private key. Three cryptographic operations total: key derivation (once), request signing (per call), order signing (per order).

The authenticated endpoints are:

- POST /order places a single order

- POST /orders places a batch of orders

- DELETE /order cancels an order

WebSocket feeds provide real-time streaming at wss://ws-subscriptions-clob.polymarket.com/ws/ for order book updates, price changes, and user-specific events.

Data API: analytics

The Data API at https://data-api.polymarket.com provides analytics-oriented data: user positions, trade history, leaderboards, and holder information. It is less documented and less stable than the other two. Some endpoints returned 404 or empty responses during our testing. Useful for research, not reliable for production.

The speed game: calling the APIs as fast as possible

If arbitrage opportunities on Polymarket last 4 seconds on average, and 73% of profits go to bots executing in under 100 milliseconds, then the speed at which you can read prices and place orders is a direct competitive advantage. We set up a benchmark to answer two questions: where should you run your code, and which language and HTTP strategy gets you there fastest?

We did our tests and chose the /midpoint endpoint for the benchmark because it requires no authentication, returns the smallest possible payload, and isolates HTTP client performance from payload parsing. Each benchmark runs 1,000 requests in two modes: sequential (one request at a time, measuring per-request latency) and concurrent (50 simultaneous workers, measuring throughput).

As always, the code can be found in our GitHub repository reserved to paying users, inside the folder 102.POLYMARKET.

Read more

In particular, your scrapers leak information at four distinct layer levels. Modern anti-bot systems, in fact, fingerprint your browser, analyze your TLS handshake, trace your network infrastructure, and track your behavioral patterns. And a single inconsistency across these layers triggers permanent blocking.

This means your scrapers aren’t competing only against rate limits anymore. Today, they are competing against machine learning models trained on billions of legitimate requests, and any deviation from the expected pattern is a signal. So, if you want to scrape at scale, your infrastructure must be indistinguishable from a real user’s browser, network stack, and behavior.

This article guides you through a systematic approach: First, understanding where leaks occur, then learning how anti-bot systems detect them, and finally building a layered defense that makes your scraper invisible.

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

Identifying the Leaks: Where Your Scraper Exposes Itself

Before fixing anything, you need to understand the complete attack surface. Modern anti-bot systems analyze your scraper at four distinct layers, and a leak at any layer can expose you.

Layer 1: The Browser Level

Headless browsers are loud by default. Launch a Puppeteer instance and check the navigator.webdriver flag. It surely returns true, and that’s a signal every major anti-bot system checks in the first 100ms of page load.

But this obvious flag is just the beginning. Anti-bot systems probe deeper:

• Error messages and stack traces: They differ between headless and headed modes. The execution context leaves fingerprints in error objects.

• Window dimensions: Properties like window.outerWidth and window.outerHeight reveal a headless operation because headless mode doesn’t render a visible window frame.

• Canvas rendering: They can produce pixel-level differences. Software rendering (headless) creates different anti-aliasing and color values than GPU-accelerated rendering (headed). Color channels can differ by 1-2 units per pixel.

• WebGL shader timing: This can vary a lot, depending on the underlying technology. GPU-accelerated browsers complete WebGL operations in microseconds. Software-rendered headless browsers take milliseconds.

• Font rendering: Headless environments often lack the full system font stack. This creates detectable layout differences when JavaScript measures text dimensions.

• Performance benchmarks: When run, they can reveal software rendering. For example, there are websites that run JavaScript stress tests, creating thousands of DOM elements, calculating layouts, and triggering reflows. In such scenarios, real browsers with GPU acceleration show consistent performance. Headless browsers, instead, show different timing patterns.

• The window.chrome object behaves differently: Real Chrome populates this object with specific properties for extension management and runtime APIs. Headless Chrome, instead, either lacks this object or provides an incomplete implementation.

Layer 2: The Network Level

Your SSL/TLS handshake identifies you before you send any application data. When your scraper connects over HTTPS, it sends a TLS Client Hello message containing supported encryption methods, protocol versions, and extensions. All in a specific order.

Here’s what makes this dangerous:

• Every browser and HTTP library has a unique TLS pattern: Real browsers send their TLS parameters in a specific sequence that matches their version and underlying platform. Python’s standard HTTP libraries send a completely different pattern. So do Node.js, Go, and any other programming language you use for coding your scrapers.

• Anti-bot systems fingerprint your TLS handshake: They capture these patterns and convert them into a fingerprint, commonly called a JA3 hash. They maintain databases of known fingerprints for every major browser and HTTP library.

• Mismatches between User-Agent and TLS fingerprint are instant red flags: When you claim to be Chrome in your User-Agent header but your TLS handshake matches Python’s urllib library, that inconsistency triggers blocking.

• Detection happens before you send any application data: The first TCP connection already identifies you as automated traffic.

• HTTP/2 fingerprinting adds another layer: Beyond TLS, the order and priority of HTTP/2 frames, settings, and window updates create additional fingerprints. Your HTTP library’s frame ordering must match your claimed browser identity.

For your scraping needs, having a reliable proxy provider like Decodo with high reputatation IPs, on your side improves the chances of success.

Try Decodo Now

Layer 3: The Infrastructure Level

Your proxy configuration can expose your real infrastructure through network-level leaks via the following main mechanisms:

• DNS leaks: They happen when your browser resolves domain names using your local DNS server instead of routing through the proxy. Your scraper might send requests through a Miami residential proxy, but if DNS queries go through your AWS datacenter in Virginia, the target site knows your real location.

• WebRTC leaks: WebRTC is a browser API designed for peer-to-peer communication. Even with a proxy configured, WebRTC will attempt to discover your real local IP and public IP through STUN servers, completely bypassing your proxy.

• IP reputation: Not all IPs are created equal. Cloudflare and similar services maintain databases of every AWS, Google Cloud, and Azure IP range. Requests from known cloud providers receive instant higher suspicion scores before any other analysis happens.

Layer 4: The Behavioral Level

Even if your browser, network, and infrastructure are perfectly disguised, your behavior patterns can still expose you:

• Timing patterns: Requesting data at fixed and precise intervals creates a perfect periodicity. No human browses with mathematical precision.

• Mouse and scroll behavior: Real humans accelerate and decelerate smoothly. Instant jumps from point A to point B are mechanically impossible.

• Session state: Stateless scrapers that never accumulate cookies or maintain persistent sessions across days look like fresh bots on every run.

• Interaction sequences: The time between page load and first click, between mouse-over and click, or the pattern of how you scroll through content. They all follow detectable human patterns.

Understanding the Detection: How Anti-Bot Systems Catch You

Now that you know where leaks occur, let’s understand how anti-bot systems actually detect them.

Fingerprint Consistency Checks

Anti-bot systems cross-reference your claimed identity with actual behavior. If your User-Agent says “Chrome 120 on Windows 10,” they verify that your JavaScript features, WebGL capabilities, canvas rendering, and TLS handshake all match Chrome 120 on Windows 10.

A single mismatch anywhere flags the entire request. You can’t be Chrome in your User-Agent, Firefox in your TLS handshake, and headless Chrome in your canvas fingerprint. Anti-bot systems create composite fingerprints combining dozens of properties, then compare them against databases of known legitimate and bot patterns.

Machine Learning Pattern Recognition

Modern anti-bot systems use ML models trained on billions of requests. They learn what “normal” looks like for each type of visitor. This means that consumer browsers from residential IPs have different behavioral patterns than datacenter scrapers.

For ML models, statistical anomalies trigger investigation. Perfect timing intervals, impossible mouse movements, or timing patterns that don’t match human variance distributions are scored as anomalous. These models adapt continuously, so when new stealth techniques emerge, the models retrain on that data. This means that what works today might fail tomorrow.

Progressive Trust Scoring

Anti-bot systems block or allow requests, but they also score. This means that lower trust scores receive degraded service: slower response times, rate limits, or CAPTCHA challenges before blocking.

Also, scores accumulate across sessions. If you leak information across multiple visits, the system builds a profile associating your various identities. In other words, one leak can poison future requests, and even fixing the leak might not restore trust if your IP or fingerprint is already marked.

Check the TWSC YouTube Channel

Building the Defense: A Layered Approach to Stealth

Building a defense from data leaks in web scraping requires addressing each layer systematically. Your stealth stack must work from the inside out: browser → network → infrastructure → behavior. Each layer must remain consistent with your claimed identity.

Defense Layer 1: Hardening the Browser

The goal at this layer is to make the browser fingerprint indistinguishable from a real user’s browser and ensure every property is consistent with your claimed identity.

Step 1: Mask Automation Signals

Start with stealth libraries that patch the most common detection vectors:

• For Puppeteer: Use puppeteer-extra-plugin-stealth to automatically override navigator.webdriver, DevTools Protocol signatures, and plugin arrays.

• For Selenium: Use undetected-chromedriver, which patches automation signals and uses real Chrome binaries instead of ChromeDriver.

• For Playwright: Leverage native evasion features that handle many detection vectors out of the box.

Additionally, disable automation flags at launch. For example, in Playwright:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(
        headless=True,
        args=[
            '--disable-blink-features=AutomationControlled',
            '--disable-dev-shm-usage',
            '--no-sandbox'
        ]
    )

But remember: Stealth libraries handle the most common 20-30 leak vectors but miss advanced fingerprinting techniques. They’re your foundation, not your complete solution.

Step 2: Spoof Hardware Signatures

Cloud server canvas and WebGL fingerprints are obvious red flags. AWS, GCP, and Azure rendering signatures are well-known to anti-bot systems.

You have two approaches for your defense here:

• Add consistent noise: Inject deterministic noise into canvas operations so the fingerprint remains stable across sessions but doesn’t match your server’s real hardware. Override canvas methods to modify pixel data slightly before it’s read back. Keep noise minimal: just enough to mask the real hardware signature without appearing obviously manipulated.

• Emulate common consumer hardware: Spoof WebGL parameters to mimic common consumer GPUs. Override vendor and renderer strings returned by WebGL APIs to match your chosen hardware profile. Use existing libraries designed for canvas fingerprint defense or implement your own parameter overrides.

Step 3: Ensure Version Consistency

This is where most scrapers fail, even with stealth libraries. Your User-Agent string must match your actual browser engine behavior precisely. Consider the following rules of thumb:

• Use real browser binaries instead of spoofing: Tools like Playwright can launch actual Chrome, ensuring perfect consistency between claimed version and actual behavior.

• If you must spoof, maintain complete version profiles: Track which JavaScript features, WebGL capabilities, and API behaviors correspond to each browser version. Every property must align.

• Never mix components from different versions: If you claim Chrome 120 on Windows 10, every single API, from JavaScript features to WebGL renderers, must behave exactly like Chrome 120 on Windows 10.

Defense Layer 2: Hardening the Network Stack

Your goal at this layer is to make your TLS handshake and HTTP traffic indistinguishable from the browser you’re claiming to be.

Step 4: Match TLS Fingerprints to Your Browser Identity

Standard HTTP libraries can’t mimic browser TLS fingerprints because they use different SSL/TLS implementations. The solution requires specialized libraries that replicate browser behavior at the protocol level:

• For Python: Use curl_cffi or similar wrappers. These libraries use libcurl compiled with BoringSSL, which is the same SSL library Chrome uses. This creates identical JA3 fingerprints to real browsers.

• For Node.js: Use cycletls or equivalent libraries that allow you to specify exact JA3 fingerprint strings matching real browsers.

Critical requirement: Your TLS fingerprint must match your User-Agent. Chrome 120’s JA3 fingerprint is different from Firefox 115’s fingerprint. The browser identity must be consistent across all layers.

Step 5: Match HTTP/2 Fingerprints

Beyond TLS, HTTP/2 frame ordering creates additional fingerprints. Libraries like curl_cffi handle this automatically when you specify a browser to impersonate, but verify that:

• Settings frames match your target browser.

• Window update sequences align.

• Priority headers follow the correct pattern.

In Python, you can do so with the following code:

response = requests.get(
    '<https://tls.peet.ws/api/all>',
    impersonate='chrome120'
)
print(response.json()['http2']['sent_frames'])

Need help with your scraping project?

Defense Layer 3: Hardening Infrastructure

Your goal at this layer is to ensure your network traffic originates from legitimate-looking IPs and doesn’t leak your real location or identity.

Step 6: Choose the Right Proxy Type

IP reputation is the first filter that anti-bot systems check. This means that your proxy choice determines your baseline trust score. Consider the following guidelines:

• Datacenter IPs = instant red flag: Requests from AWS, Google Cloud, and Azure IP ranges receive instant higher suspicion scores.

• Residential proxies = highest legitimacy: These IPs come from real ISP connections, so they look legitimate because they are legitimate consumer connections.

• Mobile proxies = premium legitimacy: These IPs originate from cellular networks (4G/5G) and receive the highest trust scores. Mobile IPs rotate naturally as devices move between cell towers, making them appear even more organic than static residential connections.

Step 7: Prevent DNS Leaks

Force all DNS resolution through your proxy tunnel. For SOCKS5 proxies, use the SOCKS5h protocol variant, which forces DNS resolution on the remote proxy server instead of locally.

For example, in Python, write the following:

import requests

proxies = {
    'http': 'socks5h://proxy.example.com:1080',
    'https': 'socks5h://proxy.example.com:1080'
}

response = requests.get('<https://example.com>', proxies=proxies)

For browser automation, configure DNS-over-HTTPS to prevent local DNS leakage. The following is an example that applies to Playwright:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(
        args=[
            '--dns-over-https-server=https://cloudflare-dns.com/dns-query'
        ]
    )

Step 8: Disable WebRTC Completely

WebRTC will expose your real IP unless you completely disable it in browser automation. For example, in Playwright, you can do so as follows:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    
    # Remove WebRTC entirely
    await page.add_init_script("""
        delete window.RTCPeerConnection;
        delete window.RTCSessionDescription;
        delete window.RTCIceCandidate;
        delete navigator.mediaDevices;
    """)

When youìve done this, verify it’s actually disabled before deploying your scraper. Visit browserleaks.com/webrtc with your scraper. You should see “WebRTC is not supported by your browser”, or only your proxy IP should be visible. Never your real IP.

Defense Layer 4: Mimicking Human Behavior

Your goal at this layer is to make your interaction patterns indistinguishable from those of real human users.

Step 9: Add Timing Jitter and Randomization

Humans are inconsistent. Perfect patterns are robotic. The solution here is not to just add randomness. You also need to match the statistical distribution of real human behavior. To do so, consider the following example in Python:

import numpy as np
import time

# Wrong example (do not use this)

# Fixed interval
time.sleep(5)  # Always 5 seconds - DETECTABLE

# Random uniform
time.sleep(random.uniform(3, 7))  # Still doesn't match human patterns

------------

# Correct example (use this!)

# Log-normal distribution (matches real human reaction times)
delay = np.random.lognormal(mean=1.5, sigma=0.5)
time.sleep(delay)

For improving randomization, model different action types with appropriate distributions. Use the following rules of thumb:

• Clicks: 0.3-2 seconds (short delays)

• Reading: 5-45 seconds (high variance)

• Scrolling: 1-8 seconds (irregular intervals)

Step 10: Implement Realistic Mouse and Scroll Behavior

High-security sites like banking, ticketing, and heavily protected e-commerce websites track interaction patterns in real-time. To defend from leaking your information on such websites, you have to define mouse movements and scrolling for your automated scripts.

For mouse movements, you can:

• Use Bezier curves to create natural arcing movements between points.

• Add slight randomness to destination coordinates.

• Include hover delays before clicking.

• Vary the number of intermediate steps based on distance.

The following is an example you can try in Python:

import numpy as np
from playwright.sync_api import sync_playwright

def bezier_curve(start, end, control_points, num_steps=20):
    """Generate points along a Bezier curve for natural mouse movement"""
    t = np.linspace(0, 1, num_steps)
    points = []
    
    # Simplified cubic Bezier
    for t_val in t:
        x = (1-t_val)**3 * start[0] + \\
            3*(1-t_val)**2*t_val * control_points[0][0] + \\
            3*(1-t_val)*t_val**2 * control_points[1][0] + \\
            t_val**3 * end[0]
        y = (1-t_val)**3 * start[1] + \\
            3*(1-t_val)**2*t_val * control_points[0][1] + \\
            3*(1-t_val)*t_val**2 * control_points[1][1] + \\
            t_val**3 * end[1]
        points.append((x, y))
    
    return points

async def human_like_click(page, selector):
    element = await page.query_selector(selector)
    box = await element.bounding_box()
    
    # Add slight randomness to destination
    target_x = box['x'] + box['width']/2 + np.random.normal(0, 2)
    target_y = box['y'] + box['height']/2 + np.random.normal(0, 2)
    
    # Move mouse along curve
    current_pos = await page.mouse.position()
    control_points = [
        (current_pos['x'] + np.random.uniform(-50, 50), 
         current_pos['y'] + np.random.uniform(-50, 50)),
        (target_x + np.random.uniform(-20, 20), 
         target_y + np.random.uniform(-20, 20))
    ]
    
    points = bezier_curve(
        (current_pos['x'], current_pos['y']), 
        (target_x, target_y), 
        control_points
    )
    
    for x, y in points:
        await page.mouse.move(x, y)
        await page.wait_for_timeout(np.random.uniform(5, 15))
    
    # Hover briefly before clicking
    await page.wait_for_timeout(np.random.uniform(100, 300))
    await page.mouse.click(target_x, target_y)

For scrolling, you can:

• Pause between scroll actions for variable amounts of time (simulating reading).

• Scroll in chunks of varying size, not uniform pixels.

• Occasionally scroll backwards (humans re-read).

• Don’t scroll in perfect increments or at constant speeds.

Use the following Python code to try such scrolling behaviour:

async def human_like_scroll(page, total_distance):
    """Scroll with human-like patterns"""
    scrolled = 0
    
    while scrolled < total_distance:
        # Vary chunk size
        chunk = np.random.randint(100, 400)
        
        await page.mouse.wheel(0, chunk)
        scrolled += chunk
        
        # Pause to simulate reading
        pause = np.random.lognormal(mean=1.2, sigma=0.8)
        await page.wait_for_timeout(pause * 1000)
        
        # Occasionally scroll backwards (humans re-read)
        if np.random.random() < 0.15:
            await page.mouse.wheel(0, -np.random.randint(50, 150))
            await page.wait_for_timeout(np.random.uniform(500, 1500))

Step 10: Maintain Persistent Session State

Stateless scrapers look like stateless bots. Real browsers, instead, accumulate state over time because:

• Cookies persist across requests and sessions.

• LocalStorage accumulates tracking data over time.

• Session IDs remain stable across days or weeks.

To mimic real browser states, you can use the following Python code:

import pickle
import requests

# Save cookies to disk after each session
session = requests.Session()

# ... perform scraping ...

with open('cookies.pkl', 'wb') as f:
    pickle.dump(session.cookies, f)

# Before next scraping session
with open('cookies.pkl', 'rb') as f:
    session.cookies.update(pickle.load(f))

In case you use a browser automation tool:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    
    # Save browser storage state
    context = browser.new_context()
    # ... perform scraping ...
    context.storage_state(path='state.json')
    
    # Reload in next session
    context = browser.new_context(storage_state='state.json')

As a final note, consider keeping sessions alive for weeks to allow third-party tracking cookies to build up. Long-lived sessions with accumulated tracking data appear more legitimate than constantly refreshed clean states.

Conclusion

In this article, you learned that, if you don’t want your data to be leaked while scraping, you have to take several defensive measures, as no single technique makes you invisible. Anti-bot systems analyze multiple signals simultaneously, and any inconsistency across layers triggers detection and blocks your scrapers.

Also, detection methods evolve. So, what works today might fail tomorrow. This means you should also monitor the defenses you implemented and test new ones.

Now, let us know: How do you prevent data leaks in your scrapers? Did we miss some technique?

Rayobyte, the proxy and web scraping solutions provider, has taken a different approach. They recently released rayobrowse, a closed‑source yet Docker‑based, self‑hostable stealth browser built for local browser automation and web scraping.

In this post, I’ll take a deep look at this solution and walk you through everything you need to know about it. By the end, you’ll understand what rayobrowse is, how its stealth browser approach works, how to set it up, and whether it’s actually worth paying attention to.

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

An Introduction to rayobrowse

Let me introduce you to the world of rayobrowse, helping you understand what it is and what makes this project special.

What is rayobrowse?

rayobrowse is a self-hosted, Chromium-based stealth browser engineered for web scraping, AI agents, and automation workflows. It’s available as a Docker image, with optional support via a Python SDK (rayobrowse on PyPI) for simplified connection. The project is developed and maintained by Rayobyte.

The stealth browser runs inside Docker and is available via the Chrome DevTools Protocol (CDP). That means tools like Playwright, Puppeteer, and Selenium (or any other tool that speaks CDP) can natively connect to it for automation purposes.

What makes it noteworthy is its approach to device fingerprinting. User agents, screen size, WebGL, fonts, timezone, and other signals are tuned so each session looks like a real browser. That way, it helps your automation avoid detection on protected websites.

Core Principles Driving the Solution

These are the core principles and goals behind the project:

1. It should run on Linux server environments without GPUs or a GUI/desktop interface.

2. It should patch Chromium at the C++ level, rather than at higher layers like CDP, which are easier for anti-bot systems to detect.

3. It should work with Playwright, a common framework in browsing automation stacks.

4. It should support both headful mode (via Xvfb) and headless mode.

5. It should emulate fingerprints from real-world devices across different regions.

6. It should be self-hostable, so you can run it locally without relying on cloud infrastructure.

7. It should be free to test and use for certain user segments.

8. It should reliably bypass major anti-bot systems and scraping targets, including complex ecommerce and SERP platforms.

Note: If you’re not familiar with Xvfb, that’s an in‑memory display server for Unix-like systems that implements the X11 display protocol without requiring a physical display or input devices. In simpler terms, it allows GUI applications to run in headless environments. rayobrowse relies on it to launch headful browser sessions even on servers without a graphical interface (that’s beneficial as headful sessions are harder to detect than purely headless ones).

Main Features for Stealth Browsing and More

Here is a list of the most relevant rayobrowse features:

• Fingerprint spoofing: Each browser session comes with a real-world realistic device fingerprint drawn from a database of thousands of profiles. Signals include user agent, OS metadata, screen resolution, fonts, WebGL, hardware concurrency, and timezone.

• Human‑like mouse movement: Optional human‑style cursor behavior (inspired by HumanCursor) makes automation appear more natural. When using standard Playwright actions like page.click() or page.mouse.move(), the library applies realistic curves and timing.

• Proxy Integration: Traffic can be routed through any HTTP proxy, including authenticated and rotating proxies.

• Headless and headful Support: rayobrowse supports both execution modes, even on GUI-less Linux servers.

• Live session viewer: A built‑in noVNC interface (available at http://localhost:6080) lets you watch browser sessions in real time directly from the browser. This is particularly useful for debugging scraping flows and visually verifying fingerprint behavior.

• Official integrations: The browser integrates with common automation frameworks, namely Playwright, Puppeteer, Selenium, and Scrapy (via scrapy-playwright), as well as emerging AI‑driven tools such as OpenClaw. As of this writing, additional integrations (e.g., Firecrawl and LangChain) are planned.

• Remote/Cloud mode: rayobrowse can run as a remote browser service. Your server requests new browser instances through a REST API, and workers connect directly to the returned CDP WebSocket endpoint. This is still a beta feature.

• API‑driven browser management: The daemon exposes REST endpoints for creating, listing, and deleting browser sessions, allowing you to orchestrate multiple browsers across a distributed scraping infrastructure.

Technical Details About the Project

Now that you know what the project is and the features it provides, you’re ready to dive into the technical aspects.

How rayobrowse Works

At a high level, rayobrowse follows these steps:

1. Chromium patching: The project tracks upstream Chromium releases and applies a focused set of patches (relying on an approach similar to Brave’s “plaster” model). These patches normalize exposed browser APIs, reduce fingerprint entropy leaks, improve automation compatibility, and preserve native Chromium behavior whenever possible.

2. Fingerprint assignment: When a browser session starts, rayobrowse assigns a realistic device fingerprint.

3. Automation integration: Browser automation libraries connect to rayobrowse through the native CDP.

Architecture

Architecturally, rayobrowse follows a clean separation between the browser runtime and the automation code.

In particular, the system runs as a Docker container that bundles three core components:

1. A daemon server that manages browser sessions.

2. A browser manager that downloads and retrieves the correct version of Chromium, a fingerprint engine that injects realistic device profiles, and a stealth browser layer containing a custom Chromium build with stealth patches.

3. A noVNC viewer, which lets you watch browser sessions in real time. This is useful for debugging and demos.

As you can see, the automation scripts don’t run inside the container. Instead, they run on the host machine and connect to the browser remotely through the Chrome DevTools Protocol.

When a new session starts, rayobrowse assigns a real-user-looking fingerprint from a large database of actual devices, containing thousands of permutations collected from websites Rayobyte owns.

Requirements

The rayobrowse project is designed to run on Linux servers without GPUs (which is a common deployment environment).

These are the required prerequisites:

• Docker, as the browser runs entirely inside a container.

• ~2GB of available RAM, as each browser instance uses ~300MB.

The main benefit of this Docker-based approach is that you don’t need to install Chromium locally, configure fonts, or set up Xvfb manually. All of those dependencies live inside the container, which keeps the host machine clean, portable, and reproducible.

It also makes the project well-suited for self-hosted environments without exposing its internal Chromium patching logic, making it much harder for anti-bot solution providers to reverse engineer how it works.

In terms of compatibility, rayobrowse works on Linux, Windows (native or WSL2), and macOS. The supported architectures are x86_64 (amd64) and ARM64 (Apple Silicon and AWS Graviton). Still, you don’t have to worry about the architecture, as Docker automatically pulls the correct image for the host machine.

Optional: If you plan to use the stealth browser through the Python SDK, an additional requirement is Python 3.10+.

Check the TWSC YouTube Channel

How to Access rayobrowse

There are two main ways you can access rayobrowse:

1. The /connect endpoint.

2. The built-in Python SDK.

Method #1: Use the /connect Endpoint

The first rayobrowse usage method involves connecting directly to the /connect endpoint. This allows any CDP‑compatible tool (including Selenium, Playwright, and Puppeteer) to open a browser session simply by pointing to a WebSocket URL like ws://localhost:9222/connect.

For instance, take a look at the Playwright connection example below:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    # Connect to rayobrowse via CDP
    browser = p.chromium.connect_over_cdp("ws://localhost:9222/connect")
    page = browser.new_context().new_page()

    # Automation logic...

    browser.close()

Keep in mind that the WebSocket browser connection URL can be customized using query parameters, as follows:

ws://localhost:9222/connect?headless=false&os=android&proxy=http://user:pass@host:port

This URL creates a rayobrowse Chromium browser session in headful mode, using Android-based fingerprints, while routing all requests through the proxy http://user:pass@host:port.

Explore all /connect query parameters in the docs.

Method #2: Use the Python SDK

You can also interact with rayobrowse through the built-in Python SDK. This exposes a create_browser() function that returns a CDP WebSocket URL for a newly created browser instance. From there, connect using Playwright or another automation framework, as shown below:

from rayobrowse import create_browser
from playwright.sync_api import sync_playwright

# Configure the rayobrowse connection to run in headful mode 
# while simulating a Windows-based fingerprint
ws_url = create_browser(headless=False, target_os="windows")

with sync_playwright() as p:
    # Connect to rayobrowse with the configured URL via CDP
    browser = p.chromium.connect_over_cdp(ws_url)
    page = browser.contexts[0].pages[0]
 
    # Automation logic...

    browser.close()

This approach gives you more control over the browser lifecycle, but it also involves more configuration and setup.

For more examples (e.g., proxy integration, multi-browser management, etc.), check out the docs.

Get Started with rayobrowse: Step-by-Step Guide

In this guided section, I’ll show you how to build a simple Playwright script that connects to rayobrowse.

For the sake of simplicity, I’ll assume you already have:

• A Unix-based system (Linux, macOS, or Windows via WSL).

• Docker installed and running on your machine.

• Git installed locally.

• A Python environment set up with Playwright installed.

Follow the instructions below!

Step #1: Clone the rayobrowse Repository

The first step is to clone the rayobrowse repository to your machine:

git clone https://github.com/rayobyte-data/rayobrowse

Then, enter the project folder with:

cd rayobrowse

The cloned folder already includes everything you need to get started, including:

• docker-compose.yml: For running the browser container.

• requirements.txt: For installing the Python SDK.

Step #2: Set Up the Environment

rayobrowse requires a .env file that contains the configuration needed to run the browser daemon. For a full list of available environment variables and what they enable, explore the official documentation.

Start by creating a .env file as a copy of the .env.example file coming with the repository:

cp .env.example .env

Then open the .env file and make sure it contains:

STEALTH_BROWSER_ACCEPT_TERMS=true

This confirms that you accept the project’s LICENSE. Without that setting, the daemon will refuse to create browser sessions.

Step #3: Start the Docker Container

Launch the rayobrowse Docker container:

docker compose up -d

Docker will automatically pull the appropriate image for your system architecture (x86_64 or ARM64). Then, it’ll start the container, as explained earlier.

Step #4: Connect via CDP and Apply the Automation Logic

You can now connect to the running rayobrowse instance through the /connect endpoint using any CDP-compatible client. In this example, I’ll use Playwright with Python:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    # Connect to the rayobrowse browser through the CDP WebSocket endpoint
    browser = p.chromium.connect_over_cdp(
        "ws://localhost:9222/connect?headless=false&os=windows"
    )

    # Create a new browser context and page
    page = browser.new_context().new_page()

    # Navigate to the target (sample) page
    page.goto("https://quotes.toscrape.com/")

    # Print the page title to verify the session is working
    print(page.title()) # Output: "Quotes to Scrape"

    # Add your scraping logic here...

    # Close the browser session
    browser.close()

At this point, write your scraping or automation logic, which will run inside the stealth Chromium browser provided by rayobrowse.

For debugging, you can watch the browser session live through noVNC at http://localhost:6080/vnc.html. While the script is running, you should see a headful Chromium session opening and navigating to the target page specified in the script:

As you can tell, the server creates a headful Chromium session (due to the headless=false query parameter) and connects it to the page requested by the script.

Optional: If you want more control over the browser lifecycle, install the Python SDK with:

pip install -r requirements.txt

Take a look at the official examples in the repository for more guidance.

Need help with your scraping project?

Pricing and Limitations

This is how the rayobrowse pricing model works:

What matters most for us, developers, is that you can run rayobrowse for free via self‑hosting. In practice, the only real cost comes from proxies, which are necessary for scaling scraping workloads and avoiding IP bans (something that’s standard in most production scraping setups).

The main thing to keep in mind is that rayobrowse is still in beta. Rayobyte already uses it to scrape millions of pages per day, but results can vary depending on the target site and configuration.

Fingerprint coverage is currently strongest for Windows and Android, while macOS and Linux profiles are less mature. In addition, Canvas and WebGL fingerprinting are still evolving, which means some websites may detect the current implementation.

Benchmarks and Final Comment

To put rayobrowse to the test, I ran a simple script against a single page for each of the most popular anti‑bot detection systems. These are the results I obtained:

Note: These tests were performed on my local machine using my ISP’s IP address.

As you can see, in this simple experiment rayobrowse achieved a 100% success rate, while Playwright failed consistently in headless mode and even struggled in some headful scenarios.

This suggests that the project is definitely worth keeping an eye on, especially thanks to its self‑hosted nature.

To be honest, and this is just my personal opinion as an expert who works in this field, I don’t usually get very excited about projects like this…. In my experience, many libraries of this type either get cracked down on or simply don’t receive the long‑term support they deserve. In this case, however, things are a bit different. The project is closed‑source and backed by a well‑known company in the industry, which makes the expectations for its future understandably much higher!

Here, I covered what the project is about, what it offers, how it works, and how to use it. As always, remember to use rayobrowse only for legal and ethical web scraping. Until next time!

FAQ

Why is rayobrowse based on Chromium and not Chrome?

rayobrowse is based on Chromium simply because Chrome is closed-source. Plus, tests performed on difficult websites show no meaningful difference in detection rates between Chrome and Chromium. Using Chromium also avoids false positives and reflects the broader ecosystem of Chromium-based browsers like Brave, Edge, and Samsung Internet.

Is rayobrowse open source?

rayobrowse isn’t open-source to prevent anti-bot companies from reverse-engineering it. Similar projects, like Camoufox, were quickly studied and countered once their code became public. Rayobyte decided to keep the project closed-source to help maintain its effectiveness and reliability over the long term.

Can everyone use rayobrowse?

No, not all companies can use rayobrowse. Its license prohibits organizations listed in Rayobyte’s restricted list from using the software. For everyone else, the project is free to download and run locally.

Does rayobrowse support proxy integration?

Yes, Rayobrowse fully supports proxy integration. You can route traffic through any HTTP proxy using the proxy query parameter on the /connect endpoint or via the proxy option exposed by the create_browser() function from the Python SDK. The proxy support includes authentication and rotating proxies.

We have seen this pattern in every scraping operation we have worked with. The knowledge exists. It is just scattered across wikis, Slack threads, internal repos, and people’s heads. The real problem is not documentation; it is retrieval. People write things down. They just cannot find them when it matters.

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

In THE LAB #77, we explored the concept of RAG (Retrieval-Augmented Generation) applied to scraped data and showed how to build a basic knowledge assistant using FAISS. That was a proof of concept. This time we are going deeper. We are showing the production system we actually built and use daily, and we are explaining the reasoning behind each design choice: why markdown, how embeddings work, which chunking strategy actually performs better, and what role auto-tagging plays in retrieval.

After reading this article, we hope you will understand the mechanics well enough to build the same system for your team.

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

What we are building and why

At TWSC, we have published around 300 articles over the past four years. Tutorials, reverse-engineering deep dives, tool comparisons, anti-bot analysis. When we sit down to write a new article, we need to remember what we have already covered, find previous work to link to, and check whether a technique we are about to describe was already explained in a past issue. Doing this by memory or by searching Substack’s archive stops working after the first hundred articles.

We also follow what the broader community publishes. Projects like Crawl4AI, which appeared on Hacker News, show that the need to ingest web content into structured, LLM-ready knowledge bases is shared across the industry. The tools for crawling and extracting content keep getting better, but the retrieval side, finding the right piece of information in a growing archive, still requires a purpose-built system.

So we built one. Here is what the complete pipeline looks like:

Sources                                  Processing              Storage & Retrieval
─────────                                ──────────              ───────────────────
Substack articles                   ──┐
                                      ├──> HTML-to-Markdown ──> Frontmatter + Tagging ──> Markdown files
Hacker News and other sources       ──┘

Markdown files ──> Chunker ──> Embedder (e5-large-v2) ──> PostgreSQL + pgvector

Search query ──> Query embedding ──> Cosine similarity search ──> Ranked results

Three stages, each independent and replaceable. You scrape content from your sources. You process and embed it. You search it.

If your team writes in Confluence instead of Substack, you swap the scraper. If you prefer Qdrant over pgvector, you swap the vector store. The architecture remains the same.

And here’s the hardware used for most of the steps, from embedding to the storage and retrieval: my DGX Spark.

Yes, I know, probably an overkill.

Check the TWSC YouTube Channel

The tools

Playwright handles browser-based scraping for our own Substack articles. Substack serves content dynamically and requires authentication for premium posts, so a plain HTTP client is not an option.

Algolia API (via Hacker News) provides structured search over HN stories. No scraping needed: HN exposes its full search index through public endpoints.

ScrapegraphAI and Firecrawl convert external article URLs into clean markdown. ScrapegraphAI is the primary extractor, Firecrawl is the fallback.

sentence-transformers with the intfloat/e5-large-v2 model generates 1024-dimensional embeddings. We will explain why we chose this model later in the article.

PostgreSQL with pgvector stores embeddings and handles similarity search. We chose it over dedicated vector databases because we already need PostgreSQL for metadata, and pgvector with HNSW indexing handles our scale without adding infrastructure.

Docker Compose ties everything together as three containers: PostgreSQL, the API server, and the indexer.

You can find the code in our GitHub repository reserved to paying users, inside the folder 101.KNOWLEDGE_BASE.

Need help with your scraping project?

Why markdown as the universal format

The first design choice we had to make was what format our knowledge base would store. We had content from Substack (HTML), Hacker News links (various formats), and potentially Confluence, Google Docs, or Slack in the future. We needed a common representation.

We chose markdown for three reasons.

First, markdown preserves document structure without carrying rendering noise. An HTML page contains navigation bars, ad slots, JavaScript, CSS classes, and layout dividers. None of that is content. When you convert to markdown, you keep headings, paragraphs, code blocks, links, and lists. Everything the embedding model needs, nothing it would choke on.

Second, markdown is readable by humans and machines alike. When something goes wrong in the pipeline, you can open a markdown file and immediately see what the system is working with. Try doing that with a serialized HTML DOM or a JSON blob from an API response.

Third, YAML frontmatter is a natural fit for markdown and gives us a structured metadata header without mixing it into the content. Each file gets an `id`, `type`, `title`, `publish_date`, `topics`, and `visibility` field. This metadata drives filtering at search time and never enters the embedding model. The separation is important: embeddings capture meaning, frontmatter captures facts.

There are two paths to get content into markdown. You can build your own converter using open-source libraries, or you can use commercial services that handle extraction and conversion for you. In this article we show both approaches deliberately. For our own Substack articles, we built a converter from scratch with BeautifulSoup and markdownify. It costs nothing, we control every detail, and it works because we know the source HTML structure intimately. For external content discovered on Hacker News, we use commercial services like ScrapegraphAI and Firecrawl instead, because every URL leads to a different site with a different HTML structure. Building custom converters for thousands of unknown domains would be impractical. The trade-off is clear: when you control the source, build your own; when you are scraping the open web, commercial extraction services save an enormous amount of development time.

Our Substack HTML-to-markdown converter is deliberately simple. It strips scripts, styles, buttons, navigation, and footers, then converts the remaining HTML:

def html_to_markdown(html: str) -> str:
    soup = BeautifulSoup(html, "lxml")
    for tag in soup.find_all(["script", "style", "button", "form", "nav", "footer"]):
        tag.decompose()

    md = markdownify(
        str(soup),
        heading_style="ATX",
        bullets="-",
        strip=["script", "style", "button", "form", "nav"],
    )
    md = re.sub(r"\n{4,}", "\n\n\n", md)
    return md.strip()

The final output for each document looks like this:

---
id: a1b2c3d4e5f6...
type: twsc_article
title: "THE LAB #94: Using Cookies and Session Persistence"
slug: the-lab-94-using-cookies-and-session
canonical_url: https://substack.thewebscraping.club/p/the-lab-94-using-cookies-and-session
publish_date: 2025-11-15
visibility: premium
topics:
  - browser-automation
  - cloudflare
  - scraping-infra
---

[article body in markdown]

Scraping your own content

The first source we built was a scraper for our own Substack articles. The pattern applies to any CMS: discover URLs, authenticate if needed, extract content, convert to markdown with frontmatter.

URL discovery and authentication

Most publishing platforms expose a sitemap. We fetch it, filter for article URLs (Substack uses /p/ in the path), and track the lastmod date to detect changes:

def fetch_sitemap(sitemap_url: str) -> list[dict]:
    req = Request(sitemap_url)
    req.add_header("User-Agent", "Mozilla/5.0 ...")
    with urlopen(req) as response:
        content = response.read()

    root = ET.fromstring(content)
    ns = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}

    articles = []
    for url_elem in root.findall("sm:url", ns):
        loc = url_elem.find("sm:loc", ns)
        lastmod = url_elem.find("sm:lastmod", ns)
        if loc is not None and "/p/" in loc.text:
            articles.append({"url": loc.text.strip(), "lastmod": lastmod.text or ""})
    return articles

Substack gates premium content behind authentication. We handle this with a persistent Playwright browser context that stores cookies across runs. On the first run you log in manually; after that, the saved session keeps you authenticated. For cron jobs, we verify the session by loading a known premium article and checking if the full content appears.

We try multiple CSS selectors for extraction because Substack has changed its DOM structure over time. The extracted HTML goes through the markdown converter we showed earlier.

Ingesting external sources: Hacker News

Read more

The traditional way to answer that question involved surveys, focus groups, and expensive reports from firms that charge you a fortune for data that’s already a few months old by the time you read it. Today, the data you need is sitting on public web pages: You just need to collect it.

In this article, we’ll discuss how to scrape data for market research, what sources actually matter, how to build a pipeline that doesn’t fall apart after a week, and where the legal lines are.

Let’s dive into it!

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

What “Market Research” Actually Means Web Scraping Professionals

Market research needs to answer three questions:

• “What are our competitors doing?”

• “What are our customers saying?”

• “How is the market moving?”

That’s it. Everything else is a variation of those three. And if you think about it, the web gives you access to all three, if you know where to look.

In practice, scraped market intelligence sits on three pillars:

• Competitive data: Pricing, product catalogs, feature changes, hiring signals. This is the “what are they doing?” pillar.

• Customer sentiment: Reviews, forum discussions, social media posts. This is the “what are people saying?” pillar.

• Market signals: Job postings, regulatory filings, trend volumes, new product launches. This is the “where is the market going?” pillar.

Now, why scraping instead of traditional research? Because scraping is real-time, it’s continuous, and it doesn’t depend on people filling out forms. A survey tells you what 500 people said last month. A scraper tells you what thousands of customers are saying right now, every single day, without anyone having to opt in.

That’s the competitive advantage. And it’s a big one.

For your scraping activity, you need IPs with good reputation. For this reason, we’re using a proxy provider like our partner Ping Proxies, that’s sharing with TWSC readers this offer.

💰 - Use TWSC for 15% OFF | $1.75/GB Residential Bandwidth | ISP Proxies in 15+ Countries

Where to Scrape: Sources That Actually Matter

Not all sources are worth your time. You could scrape the entire Internet and still end up with nothing useful if you’re not targeting the right places. Below is a list of high-value targets for market research and what you can extract from each:

• Competitor websites: Pricing pages, product pages, feature matrices, changelog, and blog posts. This is your primary source for understanding what competitors are offering and how they position themselves. Pricing pages, in particular, are gold. They change more often than you’d think, and tracking those changes over time tells you a lot about a competitor’s strategy.

• Review platforms (G2, Trustpilot, Amazon, Yelp): Customer pain points, feature requests, sentiment shifts. Reviews are unfiltered customer feedback. Nobody writes a G2 review because they were asked nicely in a survey. They write it because they feel strongly about something—and that’s exactly the kind of signal you want.

• Job boards (LinkedIn, Indeed): Hiring patterns reveal where a company is investing. If a competitor suddenly posts 20 machine learning engineer roles, that tells you something no press release will. Job postings are one of the most underrated market research signals out there.

• Social media and forums (Reddit, X, niche communities): Unfiltered opinions, emerging trends, early complaints about products. Reddit threads and niche forums are where people say what they actually think, not what they’d say in a focus group.

• Government and public data portals: SEC filings, patent databases, import/export records. These are slower-moving signals, but they’re authoritative. A patent filing can tell you what a competitor is building 18 months before it ships.

Here’s the key question to ask yourself before adding a source to your scraper: “Does this data answer a specific research question, or am I just hoarding?”. If you can’t tie a source to a concrete insight, skip it. You’ll save yourself storage costs, maintenance headaches, and potential legal issues.

Building the Pipeline: From Raw HTML to Market Intelligence

A market research scraper is not a one-off script you run from your terminal. It’s a pipeline. And pipelines need structure. If you treat it like a quick script, you’ll end up with a mess of cron jobs, inconsistent data formats, and no idea whether your data is fresh or stale. So, build it properly from the start.

A scraping for market intelligence pipeline should have four stages:

1. Collection: Fetch the pages, extract the fields you need, throw the rest away. Don’t store raw HTML “just in case” (you’ll learn why in the legal section of this article).

2. Storage: Store facts and metadata (source URL, timestamp, extracted fields). Use a structure that makes deduplication and versioning easy. In practice, this means designing your schema around a composite key (for example: source + entity ID + scraped timestamp) so you can track how a data point changes over time without overwriting previous records.

3. Transformation: Normalize the data across sources, deduplicate records, and enrich with additional context (geocoding, industry classification, entity linking).

4. Analysis: Turn rows into insights. This is where the actual market research happens. And to be clear: “Analysis” doesn’t mean opening a CSV and scrolling through it. The goal is to turn your pipeline’s output into dashboards, scheduled reports, or Slack alerts that reach the people who make decisions. If the data sits in a database and nobody looks at it, the whole pipeline is wasted effort.

Scheduling Matters More Than You Think

Different data types have different freshness requirements. Getting this wrong means either wasting resources or working with stale data. The main ideas to consider when engineering the triggering times are the following:

• Price tracking: Daily or hourly, depending on the market. Consider that e-commerce prices can change multiple times a day. SaaS pricing pages, instead, change less often. But when they do, it’s significant.

• Review monitoring: Monitoring reviews daily is usually enough. Reviews don’t appear in real-time, and sentiment trends are measured in weeks, not minutes.

• Job postings: A weekly schedule works for trend analysis of the job market. Remember that you’re looking for patterns, not individual listings.

• Social media: This depends on your use case. If you’re tracking a product launch or a PR crisis, you might need near-real-time. For general trend analysis, daily or even weekly batches work fine.

Tools That Work Well for Market Research Scraping

You don’t need to reinvent the wheel. The software industry already provides you with the best tools for your market research scraping pipeline. Here’s a solid stack for a market research pipeline:

• Scrapy for structured crawling. Scrapy’s architecture is designed for exactly this kind of work: You define spiders per source, plug in middleware for proxy rotation and retry logic, and use item pipelines to clean and store data as it flows through. For market research specifically, Scrapy’s built-in feed exports let you dump results straight to JSON, CSV, or even S3 without writing custom I/O code. And if you need to coordinate multiple spiders (say, one per competitor), Scrapy’s project structure keeps things organized as your source list grows.

• Playwright or Puppeteer for JS-heavy pages. The key difference from Scrapy is that you’re running a real browser, which means you can handle dynamic content, infinite scroll, and client-side rendering. The trade-off is resource cost: Each browser instance eats memory and CPU, so you don’t want to use this for targets that serve static HTML.

• A task queue for scheduling and orchestration. This is what turns a collection of scrapers into an actual pipeline. Instead of running scripts manually or relying on cron jobs, a task queue lets you schedule scrapes per source at different intervals, retry failed jobs automatically, and control concurrency so you’re not overwhelming a target site with parallel requests. It also gives you visibility: you can see what’s queued, what’s running, what failed, and why.

• PostgreSQL for structured market data that needs querying and versioning. Relational databases shine here because market research data is inherently relational: competitors have products, products have prices, prices change over time.

The point is this: Pick tools that let you build a maintainable system, not just a working script. Every tool in this stack solves a specific problem, and none of them requires you to build infrastructure from scratch. The best market research pipeline is the one that’s boring to operate, because boring means reliable.

Check the TWSC YouTube Channel

Scaling Without Getting Blocked

If you’re scraping one competitor once a week, you don’t need this section. If you’re tracking 50 competitors daily across thousands of pages, you do.

Here’s the reality: The moment you start scraping at scale, you become visible. But sites don’t like bots, even polite ones. So you need to be smart about how you scale. Consider the following rules of thumb to avoid getting blocked:

• Proxy rotation: Residential proxies for sensitive targets (sites with aggressive anti-bot systems), datacenter proxies for everything else. Rotate per request or per session, depending on the site’s detection mechanisms. The key is to not send thousands of requests from the same IP in an hour.

• Rate limiting and backoff: Be a good citizen. If you hammer a site with concurrent requests, you’ll get blocked, and you’ll deserve it. Implement exponential backoff on failures, and set reasonable delays between requests. A 2-3 second delay between requests is a good starting point for most sites.

• Fingerprint management: Headers, TLS fingerprint, and browser-level signals matter on sites with serious anti-bot systems. Make sure your request headers look consistent and realistic.

• CAPTCHAs: If you’re hitting CAPTCHAs regularly, your approach is too aggressive. Fix the root cause (rate, fingerprint, proxy quality) before reaching for solver services. CAPTCHA solvers are a band-aid, not a solution.

The general principle is simple: Scrape at a pace that doesn’t degrade the target site’s performance.

Need help with your scraping project?

Turning Scraped Data into Actual Market Insights

Let’s be clear about something: Raw scraped data is not market research. It’s just data. A CSV with 50’000 rows of competitor prices is not an insight. A chart showing that competitor X has dropped their enterprise tier price by 15% over three months: That’s an insight.

Here’s where the value gets created:

• Price tracking and competitive benchmarking: Track changes over time, visualize trends, and set alerts for significant moves. The goal is not to know what a competitor charges today. It’s to understand their pricing trajectory. Are they moving upmarket? Are they running more frequent discounts? Are they simplifying their tier structure? This is where predictive analytics meets scraped data with the goal of predicting future moves from your competitors.

• Sentiment analysis on reviews: Use NLP to extract themes from customer reviews. This is powerful for product teams who want to understand what customers love and hate about competitors. But remember: You’re analyzing the data internally, not republishing the reviews.

• Hiring signal analysis: Aggregate job postings by role type, department, and location. A competitor suddenly posting 15 ML engineer roles tells you they’re investing in AI. A wave of sales hiring in EMEA tells you they’re expanding geographically. This is a signal that’s almost impossible to get from any other source.

• Trend detection: Time-series analysis on product launches, feature changes, pricing moves, or social media mentions. The goal is to spot patterns or anomalies before they become obvious. If three competitors all add the same feature within two months, that’s a market trend, not a coincidence.

Overall, the output of your scraping pipeline should be dashboards, reports, or automated alerts, not a database dump that someone has to manually dig through. If the insights don’t reach decision-makers in a usable format, the whole pipeline is wasted effort.

Legal and Ethical Considerations: Don’t Skip This Section

I know, I know. You’re a developer, not a lawyer. But here’s a thing I’m sure you know: Most legal problems in scraping are self-inflicted. They happen because someone scraped “everything on the page,” stored it “for later,” and only then asked: “Wait, can we actually use this?”

As discussed in detail in “How to Avoid Copyright Violations While Scraping”, let’s go through the key legal and ethical principles of web scraping shortly:

• Scrape facts, not expression: Copyright protects expression, not facts. Prices, SKUs, dates, availability, and job titles are facts. No one owns the fact that a SaaS product costs $49/month. On the other hand, product descriptions, review text, and blog posts are creative expressions.

• Don’t store raw pages by default: Storing the HTML of entire pages means creating copies of copyrighted content. Instead, parse in-memory, extract only the fields you need, and discard the rest. If you need to debug, store a small sample with short retention.

• Respect robots.txt: The robots.txt file is not the law, but ignoring it is evidence of bad faith if things go sideways. In disputes, it can be used to show that you knew you were unwelcome and kept going anyway.

• Terms of Service matter: If the ToS explicitly forbids scraping and you scrape anyway, you may have a breach-of-contract problem. This is often easier for the site owner to prove than copyright infringement, because the argument is straightforward: you agreed to a contract, then you violated it.

• Don’t scrape behind a login: Once you log in, you’ve affirmatively agreed to a contract. Breaking that contract to scrape is a fast track to legal trouble. If your plan requires authenticated access, treat it as a licensing problem, not an engineering challenge.

• GDPR/CCPA: If you’re scraping anything that could be personal data (usernames, reviewer names, profile information), you need to know which privacy laws apply. This is especially relevant for review scraping and social media monitoring.

Here’s the mental model that works: A price comparison tool that shows prices and links back to the source? Generally safe. A product catalog that copies descriptions, images, and reviews so users never need to visit the original site? That’s where you get into trouble, even if you don’t publicly display the results because you use them for internal analysis.

Keeping Your Scrapers Alive: Monitoring and Maintenance

Scrapers in production break for several reasons. Sites change layouts, add anti-bot measures, restructure their URLs, or just go down for maintenance. If you don’t monitor your scrapers, your data goes stale silently, and you won’t know until someone asks why the pricing dashboard hasn’t updated in three weeks.

Here’s a breakdown of what you need:

• Dead selector detection: Alert when a CSS selector or XPath returns empty across multiple consecutive runs. A selector that worked yesterday and returns nothing today means the site changed its HTML structure. The keyword here is “multiple consecutive runs”. A single empty result could be a transient issue, so consider not triggering alerts on the first failure. Instead, set a threshold, like three consecutive empty results, before flagging it. When it does fire, you need to inspect the current page structure and update your selectors. Alternatively, try to go beyond the DOM using AI and LLMs, to make your extraction more resilient to layout changes in the first place.

• HTTP status monitoring: A spike in 403s means you’re getting blocked. A spike in 429s means you’re hitting rate limits. A spike in 404s means URLs have changed. Each of these requires a different response. For 403s, check your proxy pool and rotation logic: You might need fresher IPs or a lower request rate. For 429s, back off and increase your delays between requests; the site is telling you exactly what the problem is. For 404s, the target has likely restructured its URL patterns, which means you need to update your URL generation logic, not just retry the same broken links. Log these status codes per source and per run so you can spot trends early. A gradual increase in 403s over a week is a warning sign that your current setup is losing effectiveness, even if individual runs still return some data.

• Data quality checks: Row counts, null rates, value distributions. If your price tracker suddenly shows all prices as $0 or your review scraper returns empty text fields, you want to know immediately. Build quality checks into your pipeline as a post-scrape validation step, not as something you run manually. Compare each run’s output against baseline expectations: If you normally get 200 rows from a source and today you got 12, something is wrong, even if those 12 rows look fine individually.

• Automated tests against fixture HTML: Save sample HTML pages from your targets and write tests against them. When a test fails, you know the site has changed before your production scraper breaks. Treat your scrapers like production code, because they are. In practice, this means saving a snapshot of a relevant section in the target page as a local HTML file. Then, write unit tests that run your extraction logic against that fixture and assert expected outputs. Store these fixtures in version control alongside your scraper code. When a site changes and your production scraper breaks, update the fixture with the new HTML. This gives you a repeatable workflow for handling site changes instead of scrambling every time something breaks.

The goal is simple: You should know when something breaks before your stakeholders do. A Slack alert that says “Competitor X pricing scraper returned 0 results” is infinitely better than a product manager asking why the dashboard is empty.

Conclusion

In this article, you learned that market research scraping is about building a reliable pipeline that collects the right facts, transforms them into insights, and doesn’t get you in legal trouble.

The competitive advantage of scraping for market research is in what you do with the data. Anyone can code a scraper. But building a system that delivers reliable, actionable market intelligence week after week? That’s where the real value is!

So, let us know: Are you using web scraping for market research? What sources have you found most valuable? How did you structure your scraping pipeline? Let’s discuss in the comments!