In particular, your scrapers leak information at four distinct layer levels. Modern anti-bot systems, in fact, fingerprint your browser, analyze your TLS handshake, trace your network infrastructure, and track your behavioral patterns. And a single inconsistency across these layers triggers permanent blocking.

This means your scrapers aren’t competing only against rate limits anymore. Today, they are competing against machine learning models trained on billions of legitimate requests, and any deviation from the expected pattern is a signal. So, if you want to scrape at scale, your infrastructure must be indistinguishable from a real user’s browser, network stack, and behavior.

This article guides you through a systematic approach: First, understanding where leaks occur, then learning how anti-bot systems detect them, and finally building a layered defense that makes your scraper invisible.

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

Identifying the Leaks: Where Your Scraper Exposes Itself

Before fixing anything, you need to understand the complete attack surface. Modern anti-bot systems analyze your scraper at four distinct layers, and a leak at any layer can expose you.

Layer 1: The Browser Level

Headless browsers are loud by default. Launch a Puppeteer instance and check the navigator.webdriver flag. It surely returns true, and that’s a signal every major anti-bot system checks in the first 100ms of page load.

But this obvious flag is just the beginning. Anti-bot systems probe deeper:

• Error messages and stack traces: They differ between headless and headed modes. The execution context leaves fingerprints in error objects.

• Window dimensions: Properties like window.outerWidth and window.outerHeight reveal a headless operation because headless mode doesn’t render a visible window frame.

• Canvas rendering: They can produce pixel-level differences. Software rendering (headless) creates different anti-aliasing and color values than GPU-accelerated rendering (headed). Color channels can differ by 1-2 units per pixel.

• WebGL shader timing: This can vary a lot, depending on the underlying technology. GPU-accelerated browsers complete WebGL operations in microseconds. Software-rendered headless browsers take milliseconds.

• Font rendering: Headless environments often lack the full system font stack. This creates detectable layout differences when JavaScript measures text dimensions.

• Performance benchmarks: When run, they can reveal software rendering. For example, there are websites that run JavaScript stress tests, creating thousands of DOM elements, calculating layouts, and triggering reflows. In such scenarios, real browsers with GPU acceleration show consistent performance. Headless browsers, instead, show different timing patterns.

• The window.chrome object behaves differently: Real Chrome populates this object with specific properties for extension management and runtime APIs. Headless Chrome, instead, either lacks this object or provides an incomplete implementation.

Layer 2: The Network Level

Your SSL/TLS handshake identifies you before you send any application data. When your scraper connects over HTTPS, it sends a TLS Client Hello message containing supported encryption methods, protocol versions, and extensions. All in a specific order.

Here’s what makes this dangerous:

• Every browser and HTTP library has a unique TLS pattern: Real browsers send their TLS parameters in a specific sequence that matches their version and underlying platform. Python’s standard HTTP libraries send a completely different pattern. So do Node.js, Go, and any other programming language you use for coding your scrapers.

• Anti-bot systems fingerprint your TLS handshake: They capture these patterns and convert them into a fingerprint, commonly called a JA3 hash. They maintain databases of known fingerprints for every major browser and HTTP library.

• Mismatches between User-Agent and TLS fingerprint are instant red flags: When you claim to be Chrome in your User-Agent header but your TLS handshake matches Python’s urllib library, that inconsistency triggers blocking.

• Detection happens before you send any application data: The first TCP connection already identifies you as automated traffic.

• HTTP/2 fingerprinting adds another layer: Beyond TLS, the order and priority of HTTP/2 frames, settings, and window updates create additional fingerprints. Your HTTP library’s frame ordering must match your claimed browser identity.

For your scraping needs, having a reliable proxy provider like Decodo with high reputatation IPs, on your side improves the chances of success.

Try Decodo Now

Layer 3: The Infrastructure Level

Your proxy configuration can expose your real infrastructure through network-level leaks via the following main mechanisms:

• DNS leaks: They happen when your browser resolves domain names using your local DNS server instead of routing through the proxy. Your scraper might send requests through a Miami residential proxy, but if DNS queries go through your AWS datacenter in Virginia, the target site knows your real location.

• WebRTC leaks: WebRTC is a browser API designed for peer-to-peer communication. Even with a proxy configured, WebRTC will attempt to discover your real local IP and public IP through STUN servers, completely bypassing your proxy.

• IP reputation: Not all IPs are created equal. Cloudflare and similar services maintain databases of every AWS, Google Cloud, and Azure IP range. Requests from known cloud providers receive instant higher suspicion scores before any other analysis happens.

Layer 4: The Behavioral Level

Even if your browser, network, and infrastructure are perfectly disguised, your behavior patterns can still expose you:

• Timing patterns: Requesting data at fixed and precise intervals creates a perfect periodicity. No human browses with mathematical precision.

• Mouse and scroll behavior: Real humans accelerate and decelerate smoothly. Instant jumps from point A to point B are mechanically impossible.

• Session state: Stateless scrapers that never accumulate cookies or maintain persistent sessions across days look like fresh bots on every run.

• Interaction sequences: The time between page load and first click, between mouse-over and click, or the pattern of how you scroll through content. They all follow detectable human patterns.

Understanding the Detection: How Anti-Bot Systems Catch You

Now that you know where leaks occur, let’s understand how anti-bot systems actually detect them.

Fingerprint Consistency Checks

Anti-bot systems cross-reference your claimed identity with actual behavior. If your User-Agent says “Chrome 120 on Windows 10,” they verify that your JavaScript features, WebGL capabilities, canvas rendering, and TLS handshake all match Chrome 120 on Windows 10.

A single mismatch anywhere flags the entire request. You can’t be Chrome in your User-Agent, Firefox in your TLS handshake, and headless Chrome in your canvas fingerprint. Anti-bot systems create composite fingerprints combining dozens of properties, then compare them against databases of known legitimate and bot patterns.

Machine Learning Pattern Recognition

Modern anti-bot systems use ML models trained on billions of requests. They learn what “normal” looks like for each type of visitor. This means that consumer browsers from residential IPs have different behavioral patterns than datacenter scrapers.

For ML models, statistical anomalies trigger investigation. Perfect timing intervals, impossible mouse movements, or timing patterns that don’t match human variance distributions are scored as anomalous. These models adapt continuously, so when new stealth techniques emerge, the models retrain on that data. This means that what works today might fail tomorrow.

Progressive Trust Scoring

Anti-bot systems block or allow requests, but they also score. This means that lower trust scores receive degraded service: slower response times, rate limits, or CAPTCHA challenges before blocking.

Also, scores accumulate across sessions. If you leak information across multiple visits, the system builds a profile associating your various identities. In other words, one leak can poison future requests, and even fixing the leak might not restore trust if your IP or fingerprint is already marked.

Check the TWSC YouTube Channel

Building the Defense: A Layered Approach to Stealth

Building a defense from data leaks in web scraping requires addressing each layer systematically. Your stealth stack must work from the inside out: browser → network → infrastructure → behavior. Each layer must remain consistent with your claimed identity.

Defense Layer 1: Hardening the Browser

The goal at this layer is to make the browser fingerprint indistinguishable from a real user’s browser and ensure every property is consistent with your claimed identity.

Step 1: Mask Automation Signals

Start with stealth libraries that patch the most common detection vectors:

• For Puppeteer: Use puppeteer-extra-plugin-stealth to automatically override navigator.webdriver, DevTools Protocol signatures, and plugin arrays.

• For Selenium: Use undetected-chromedriver, which patches automation signals and uses real Chrome binaries instead of ChromeDriver.

• For Playwright: Leverage native evasion features that handle many detection vectors out of the box.

Additionally, disable automation flags at launch. For example, in Playwright:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(
        headless=True,
        args=[
            '--disable-blink-features=AutomationControlled',
            '--disable-dev-shm-usage',
            '--no-sandbox'
        ]
    )

But remember: Stealth libraries handle the most common 20-30 leak vectors but miss advanced fingerprinting techniques. They’re your foundation, not your complete solution.

Step 2: Spoof Hardware Signatures

Cloud server canvas and WebGL fingerprints are obvious red flags. AWS, GCP, and Azure rendering signatures are well-known to anti-bot systems.

You have two approaches for your defense here:

• Add consistent noise: Inject deterministic noise into canvas operations so the fingerprint remains stable across sessions but doesn’t match your server’s real hardware. Override canvas methods to modify pixel data slightly before it’s read back. Keep noise minimal: just enough to mask the real hardware signature without appearing obviously manipulated.

• Emulate common consumer hardware: Spoof WebGL parameters to mimic common consumer GPUs. Override vendor and renderer strings returned by WebGL APIs to match your chosen hardware profile. Use existing libraries designed for canvas fingerprint defense or implement your own parameter overrides.

Step 3: Ensure Version Consistency

This is where most scrapers fail, even with stealth libraries. Your User-Agent string must match your actual browser engine behavior precisely. Consider the following rules of thumb:

• Use real browser binaries instead of spoofing: Tools like Playwright can launch actual Chrome, ensuring perfect consistency between claimed version and actual behavior.

• If you must spoof, maintain complete version profiles: Track which JavaScript features, WebGL capabilities, and API behaviors correspond to each browser version. Every property must align.

• Never mix components from different versions: If you claim Chrome 120 on Windows 10, every single API, from JavaScript features to WebGL renderers, must behave exactly like Chrome 120 on Windows 10.

Defense Layer 2: Hardening the Network Stack

Your goal at this layer is to make your TLS handshake and HTTP traffic indistinguishable from the browser you’re claiming to be.

Step 4: Match TLS Fingerprints to Your Browser Identity

Standard HTTP libraries can’t mimic browser TLS fingerprints because they use different SSL/TLS implementations. The solution requires specialized libraries that replicate browser behavior at the protocol level:

• For Python: Use curl_cffi or similar wrappers. These libraries use libcurl compiled with BoringSSL, which is the same SSL library Chrome uses. This creates identical JA3 fingerprints to real browsers.

• For Node.js: Use cycletls or equivalent libraries that allow you to specify exact JA3 fingerprint strings matching real browsers.

Critical requirement: Your TLS fingerprint must match your User-Agent. Chrome 120’s JA3 fingerprint is different from Firefox 115’s fingerprint. The browser identity must be consistent across all layers.

Step 5: Match HTTP/2 Fingerprints

Beyond TLS, HTTP/2 frame ordering creates additional fingerprints. Libraries like curl_cffi handle this automatically when you specify a browser to impersonate, but verify that:

• Settings frames match your target browser.

• Window update sequences align.

• Priority headers follow the correct pattern.

In Python, you can do so with the following code:

response = requests.get(
    '<https://tls.peet.ws/api/all>',
    impersonate='chrome120'
)
print(response.json()['http2']['sent_frames'])

Need help with your scraping project?

Defense Layer 3: Hardening Infrastructure

Your goal at this layer is to ensure your network traffic originates from legitimate-looking IPs and doesn’t leak your real location or identity.

Step 6: Choose the Right Proxy Type

IP reputation is the first filter that anti-bot systems check. This means that your proxy choice determines your baseline trust score. Consider the following guidelines:

• Datacenter IPs = instant red flag: Requests from AWS, Google Cloud, and Azure IP ranges receive instant higher suspicion scores.

• Residential proxies = highest legitimacy: These IPs come from real ISP connections, so they look legitimate because they are legitimate consumer connections.

• Mobile proxies = premium legitimacy: These IPs originate from cellular networks (4G/5G) and receive the highest trust scores. Mobile IPs rotate naturally as devices move between cell towers, making them appear even more organic than static residential connections.

Step 7: Prevent DNS Leaks

Force all DNS resolution through your proxy tunnel. For SOCKS5 proxies, use the SOCKS5h protocol variant, which forces DNS resolution on the remote proxy server instead of locally.

For example, in Python, write the following:

import requests

proxies = {
    'http': 'socks5h://proxy.example.com:1080',
    'https': 'socks5h://proxy.example.com:1080'
}

response = requests.get('<https://example.com>', proxies=proxies)

For browser automation, configure DNS-over-HTTPS to prevent local DNS leakage. The following is an example that applies to Playwright:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(
        args=[
            '--dns-over-https-server=https://cloudflare-dns.com/dns-query'
        ]
    )

Step 8: Disable WebRTC Completely

WebRTC will expose your real IP unless you completely disable it in browser automation. For example, in Playwright, you can do so as follows:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    
    # Remove WebRTC entirely
    await page.add_init_script("""
        delete window.RTCPeerConnection;
        delete window.RTCSessionDescription;
        delete window.RTCIceCandidate;
        delete navigator.mediaDevices;
    """)

When youìve done this, verify it’s actually disabled before deploying your scraper. Visit browserleaks.com/webrtc with your scraper. You should see “WebRTC is not supported by your browser”, or only your proxy IP should be visible. Never your real IP.

Defense Layer 4: Mimicking Human Behavior

Your goal at this layer is to make your interaction patterns indistinguishable from those of real human users.

Step 9: Add Timing Jitter and Randomization

Humans are inconsistent. Perfect patterns are robotic. The solution here is not to just add randomness. You also need to match the statistical distribution of real human behavior. To do so, consider the following example in Python:

import numpy as np
import time

# Wrong example (do not use this)

# Fixed interval
time.sleep(5)  # Always 5 seconds - DETECTABLE

# Random uniform
time.sleep(random.uniform(3, 7))  # Still doesn't match human patterns

------------

# Correct example (use this!)

# Log-normal distribution (matches real human reaction times)
delay = np.random.lognormal(mean=1.5, sigma=0.5)
time.sleep(delay)

For improving randomization, model different action types with appropriate distributions. Use the following rules of thumb:

• Clicks: 0.3-2 seconds (short delays)

• Reading: 5-45 seconds (high variance)

• Scrolling: 1-8 seconds (irregular intervals)

Step 10: Implement Realistic Mouse and Scroll Behavior

High-security sites like banking, ticketing, and heavily protected e-commerce websites track interaction patterns in real-time. To defend from leaking your information on such websites, you have to define mouse movements and scrolling for your automated scripts.

For mouse movements, you can:

• Use Bezier curves to create natural arcing movements between points.

• Add slight randomness to destination coordinates.

• Include hover delays before clicking.

• Vary the number of intermediate steps based on distance.

The following is an example you can try in Python:

import numpy as np
from playwright.sync_api import sync_playwright

def bezier_curve(start, end, control_points, num_steps=20):
    """Generate points along a Bezier curve for natural mouse movement"""
    t = np.linspace(0, 1, num_steps)
    points = []
    
    # Simplified cubic Bezier
    for t_val in t:
        x = (1-t_val)**3 * start[0] + \\
            3*(1-t_val)**2*t_val * control_points[0][0] + \\
            3*(1-t_val)*t_val**2 * control_points[1][0] + \\
            t_val**3 * end[0]
        y = (1-t_val)**3 * start[1] + \\
            3*(1-t_val)**2*t_val * control_points[0][1] + \\
            3*(1-t_val)*t_val**2 * control_points[1][1] + \\
            t_val**3 * end[1]
        points.append((x, y))
    
    return points

async def human_like_click(page, selector):
    element = await page.query_selector(selector)
    box = await element.bounding_box()
    
    # Add slight randomness to destination
    target_x = box['x'] + box['width']/2 + np.random.normal(0, 2)
    target_y = box['y'] + box['height']/2 + np.random.normal(0, 2)
    
    # Move mouse along curve
    current_pos = await page.mouse.position()
    control_points = [
        (current_pos['x'] + np.random.uniform(-50, 50), 
         current_pos['y'] + np.random.uniform(-50, 50)),
        (target_x + np.random.uniform(-20, 20), 
         target_y + np.random.uniform(-20, 20))
    ]
    
    points = bezier_curve(
        (current_pos['x'], current_pos['y']), 
        (target_x, target_y), 
        control_points
    )
    
    for x, y in points:
        await page.mouse.move(x, y)
        await page.wait_for_timeout(np.random.uniform(5, 15))
    
    # Hover briefly before clicking
    await page.wait_for_timeout(np.random.uniform(100, 300))
    await page.mouse.click(target_x, target_y)

For scrolling, you can:

• Pause between scroll actions for variable amounts of time (simulating reading).

• Scroll in chunks of varying size, not uniform pixels.

• Occasionally scroll backwards (humans re-read).

• Don’t scroll in perfect increments or at constant speeds.

Use the following Python code to try such scrolling behaviour:

async def human_like_scroll(page, total_distance):
    """Scroll with human-like patterns"""
    scrolled = 0
    
    while scrolled < total_distance:
        # Vary chunk size
        chunk = np.random.randint(100, 400)
        
        await page.mouse.wheel(0, chunk)
        scrolled += chunk
        
        # Pause to simulate reading
        pause = np.random.lognormal(mean=1.2, sigma=0.8)
        await page.wait_for_timeout(pause * 1000)
        
        # Occasionally scroll backwards (humans re-read)
        if np.random.random() < 0.15:
            await page.mouse.wheel(0, -np.random.randint(50, 150))
            await page.wait_for_timeout(np.random.uniform(500, 1500))

Step 10: Maintain Persistent Session State

Stateless scrapers look like stateless bots. Real browsers, instead, accumulate state over time because:

• Cookies persist across requests and sessions.

• LocalStorage accumulates tracking data over time.

• Session IDs remain stable across days or weeks.

To mimic real browser states, you can use the following Python code:

import pickle
import requests

# Save cookies to disk after each session
session = requests.Session()

# ... perform scraping ...

with open('cookies.pkl', 'wb') as f:
    pickle.dump(session.cookies, f)

# Before next scraping session
with open('cookies.pkl', 'rb') as f:
    session.cookies.update(pickle.load(f))

In case you use a browser automation tool:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    
    # Save browser storage state
    context = browser.new_context()
    # ... perform scraping ...
    context.storage_state(path='state.json')
    
    # Reload in next session
    context = browser.new_context(storage_state='state.json')

As a final note, consider keeping sessions alive for weeks to allow third-party tracking cookies to build up. Long-lived sessions with accumulated tracking data appear more legitimate than constantly refreshed clean states.

Conclusion

In this article, you learned that, if you don’t want your data to be leaked while scraping, you have to take several defensive measures, as no single technique makes you invisible. Anti-bot systems analyze multiple signals simultaneously, and any inconsistency across layers triggers detection and blocks your scrapers.

Also, detection methods evolve. So, what works today might fail tomorrow. This means you should also monitor the defenses you implemented and test new ones.

Now, let us know: How do you prevent data leaks in your scrapers? Did we miss some technique?

Rayobyte, the proxy and web scraping solutions provider, has taken a different approach. They recently released rayobrowse, a closed‑source yet Docker‑based, self‑hostable stealth browser built for local browser automation and web scraping.

In this post, I’ll take a deep look at this solution and walk you through everything you need to know about it. By the end, you’ll understand what rayobrowse is, how its stealth browser approach works, how to set it up, and whether it’s actually worth paying attention to.

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

An Introduction to rayobrowse

Let me introduce you to the world of rayobrowse, helping you understand what it is and what makes this project special.

What is rayobrowse?

rayobrowse is a self-hosted, Chromium-based stealth browser engineered for web scraping, AI agents, and automation workflows. It’s available as a Docker image, with optional support via a Python SDK (rayobrowse on PyPI) for simplified connection. The project is developed and maintained by Rayobyte.

The stealth browser runs inside Docker and is available via the Chrome DevTools Protocol (CDP). That means tools like Playwright, Puppeteer, and Selenium (or any other tool that speaks CDP) can natively connect to it for automation purposes.

What makes it noteworthy is its approach to device fingerprinting. User agents, screen size, WebGL, fonts, timezone, and other signals are tuned so each session looks like a real browser. That way, it helps your automation avoid detection on protected websites.

Core Principles Driving the Solution

These are the core principles and goals behind the project:

1. It should run on Linux server environments without GPUs or a GUI/desktop interface.

2. It should patch Chromium at the C++ level, rather than at higher layers like CDP, which are easier for anti-bot systems to detect.

3. It should work with Playwright, a common framework in browsing automation stacks.

4. It should support both headful mode (via Xvfb) and headless mode.

5. It should emulate fingerprints from real-world devices across different regions.

6. It should be self-hostable, so you can run it locally without relying on cloud infrastructure.

7. It should be free to test and use for certain user segments.

8. It should reliably bypass major anti-bot systems and scraping targets, including complex ecommerce and SERP platforms.

Note: If you’re not familiar with Xvfb, that’s an in‑memory display server for Unix-like systems that implements the X11 display protocol without requiring a physical display or input devices. In simpler terms, it allows GUI applications to run in headless environments. rayobrowse relies on it to launch headful browser sessions even on servers without a graphical interface (that’s beneficial as headful sessions are harder to detect than purely headless ones).

Main Features for Stealth Browsing and More

Here is a list of the most relevant rayobrowse features:

• Fingerprint spoofing: Each browser session comes with a real-world realistic device fingerprint drawn from a database of thousands of profiles. Signals include user agent, OS metadata, screen resolution, fonts, WebGL, hardware concurrency, and timezone.

• Human‑like mouse movement: Optional human‑style cursor behavior (inspired by HumanCursor) makes automation appear more natural. When using standard Playwright actions like page.click() or page.mouse.move(), the library applies realistic curves and timing.

• Proxy Integration: Traffic can be routed through any HTTP proxy, including authenticated and rotating proxies.

• Headless and headful Support: rayobrowse supports both execution modes, even on GUI-less Linux servers.

• Live session viewer: A built‑in noVNC interface (available at http://localhost:6080) lets you watch browser sessions in real time directly from the browser. This is particularly useful for debugging scraping flows and visually verifying fingerprint behavior.

• Official integrations: The browser integrates with common automation frameworks, namely Playwright, Puppeteer, Selenium, and Scrapy (via scrapy-playwright), as well as emerging AI‑driven tools such as OpenClaw. As of this writing, additional integrations (e.g., Firecrawl and LangChain) are planned.

• Remote/Cloud mode: rayobrowse can run as a remote browser service. Your server requests new browser instances through a REST API, and workers connect directly to the returned CDP WebSocket endpoint. This is still a beta feature.

• API‑driven browser management: The daemon exposes REST endpoints for creating, listing, and deleting browser sessions, allowing you to orchestrate multiple browsers across a distributed scraping infrastructure.

Technical Details About the Project

Now that you know what the project is and the features it provides, you’re ready to dive into the technical aspects.

How rayobrowse Works

At a high level, rayobrowse follows these steps:

1. Chromium patching: The project tracks upstream Chromium releases and applies a focused set of patches (relying on an approach similar to Brave’s “plaster” model). These patches normalize exposed browser APIs, reduce fingerprint entropy leaks, improve automation compatibility, and preserve native Chromium behavior whenever possible.

2. Fingerprint assignment: When a browser session starts, rayobrowse assigns a realistic device fingerprint.

3. Automation integration: Browser automation libraries connect to rayobrowse through the native CDP.

Architecture

Architecturally, rayobrowse follows a clean separation between the browser runtime and the automation code.

In particular, the system runs as a Docker container that bundles three core components:

1. A daemon server that manages browser sessions.

2. A browser manager that downloads and retrieves the correct version of Chromium, a fingerprint engine that injects realistic device profiles, and a stealth browser layer containing a custom Chromium build with stealth patches.

3. A noVNC viewer, which lets you watch browser sessions in real time. This is useful for debugging and demos.

As you can see, the automation scripts don’t run inside the container. Instead, they run on the host machine and connect to the browser remotely through the Chrome DevTools Protocol.

When a new session starts, rayobrowse assigns a real-user-looking fingerprint from a large database of actual devices, containing thousands of permutations collected from websites Rayobyte owns.

Requirements

The rayobrowse project is designed to run on Linux servers without GPUs (which is a common deployment environment).

These are the required prerequisites:

• Docker, as the browser runs entirely inside a container.

• ~2GB of available RAM, as each browser instance uses ~300MB.

The main benefit of this Docker-based approach is that you don’t need to install Chromium locally, configure fonts, or set up Xvfb manually. All of those dependencies live inside the container, which keeps the host machine clean, portable, and reproducible.

It also makes the project well-suited for self-hosted environments without exposing its internal Chromium patching logic, making it much harder for anti-bot solution providers to reverse engineer how it works.

In terms of compatibility, rayobrowse works on Linux, Windows (native or WSL2), and macOS. The supported architectures are x86_64 (amd64) and ARM64 (Apple Silicon and AWS Graviton). Still, you don’t have to worry about the architecture, as Docker automatically pulls the correct image for the host machine.

Optional: If you plan to use the stealth browser through the Python SDK, an additional requirement is Python 3.10+.

Check the TWSC YouTube Channel

How to Access rayobrowse

There are two main ways you can access rayobrowse:

1. The /connect endpoint.

2. The built-in Python SDK.

Method #1: Use the /connect Endpoint

The first rayobrowse usage method involves connecting directly to the /connect endpoint. This allows any CDP‑compatible tool (including Selenium, Playwright, and Puppeteer) to open a browser session simply by pointing to a WebSocket URL like ws://localhost:9222/connect.

For instance, take a look at the Playwright connection example below:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    # Connect to rayobrowse via CDP
    browser = p.chromium.connect_over_cdp("ws://localhost:9222/connect")
    page = browser.new_context().new_page()

    # Automation logic...

    browser.close()

Keep in mind that the WebSocket browser connection URL can be customized using query parameters, as follows:

ws://localhost:9222/connect?headless=false&os=android&proxy=http://user:pass@host:port

This URL creates a rayobrowse Chromium browser session in headful mode, using Android-based fingerprints, while routing all requests through the proxy http://user:pass@host:port.

Explore all /connect query parameters in the docs.

Method #2: Use the Python SDK

You can also interact with rayobrowse through the built-in Python SDK. This exposes a create_browser() function that returns a CDP WebSocket URL for a newly created browser instance. From there, connect using Playwright or another automation framework, as shown below:

from rayobrowse import create_browser
from playwright.sync_api import sync_playwright

# Configure the rayobrowse connection to run in headful mode 
# while simulating a Windows-based fingerprint
ws_url = create_browser(headless=False, target_os="windows")

with sync_playwright() as p:
    # Connect to rayobrowse with the configured URL via CDP
    browser = p.chromium.connect_over_cdp(ws_url)
    page = browser.contexts[0].pages[0]
 
    # Automation logic...

    browser.close()

This approach gives you more control over the browser lifecycle, but it also involves more configuration and setup.

For more examples (e.g., proxy integration, multi-browser management, etc.), check out the docs.

Get Started with rayobrowse: Step-by-Step Guide

In this guided section, I’ll show you how to build a simple Playwright script that connects to rayobrowse.

For the sake of simplicity, I’ll assume you already have:

• A Unix-based system (Linux, macOS, or Windows via WSL).

• Docker installed and running on your machine.

• Git installed locally.

• A Python environment set up with Playwright installed.

Follow the instructions below!

Step #1: Clone the rayobrowse Repository

The first step is to clone the rayobrowse repository to your machine:

git clone https://github.com/rayobyte-data/rayobrowse

Then, enter the project folder with:

cd rayobrowse

The cloned folder already includes everything you need to get started, including:

• docker-compose.yml: For running the browser container.

• requirements.txt: For installing the Python SDK.

Step #2: Set Up the Environment

rayobrowse requires a .env file that contains the configuration needed to run the browser daemon. For a full list of available environment variables and what they enable, explore the official documentation.

Start by creating a .env file as a copy of the .env.example file coming with the repository:

cp .env.example .env

Then open the .env file and make sure it contains:

STEALTH_BROWSER_ACCEPT_TERMS=true

This confirms that you accept the project’s LICENSE. Without that setting, the daemon will refuse to create browser sessions.

Step #3: Start the Docker Container

Launch the rayobrowse Docker container:

docker compose up -d

Docker will automatically pull the appropriate image for your system architecture (x86_64 or ARM64). Then, it’ll start the container, as explained earlier.

Step #4: Connect via CDP and Apply the Automation Logic

You can now connect to the running rayobrowse instance through the /connect endpoint using any CDP-compatible client. In this example, I’ll use Playwright with Python:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    # Connect to the rayobrowse browser through the CDP WebSocket endpoint
    browser = p.chromium.connect_over_cdp(
        "ws://localhost:9222/connect?headless=false&os=windows"
    )

    # Create a new browser context and page
    page = browser.new_context().new_page()

    # Navigate to the target (sample) page
    page.goto("https://quotes.toscrape.com/")

    # Print the page title to verify the session is working
    print(page.title()) # Output: "Quotes to Scrape"

    # Add your scraping logic here...

    # Close the browser session
    browser.close()

At this point, write your scraping or automation logic, which will run inside the stealth Chromium browser provided by rayobrowse.

For debugging, you can watch the browser session live through noVNC at http://localhost:6080/vnc.html. While the script is running, you should see a headful Chromium session opening and navigating to the target page specified in the script:

As you can tell, the server creates a headful Chromium session (due to the headless=false query parameter) and connects it to the page requested by the script.

Optional: If you want more control over the browser lifecycle, install the Python SDK with:

pip install -r requirements.txt

Take a look at the official examples in the repository for more guidance.

Need help with your scraping project?

Pricing and Limitations

This is how the rayobrowse pricing model works:

What matters most for us, developers, is that you can run rayobrowse for free via self‑hosting. In practice, the only real cost comes from proxies, which are necessary for scaling scraping workloads and avoiding IP bans (something that’s standard in most production scraping setups).

The main thing to keep in mind is that rayobrowse is still in beta. Rayobyte already uses it to scrape millions of pages per day, but results can vary depending on the target site and configuration.

Fingerprint coverage is currently strongest for Windows and Android, while macOS and Linux profiles are less mature. In addition, Canvas and WebGL fingerprinting are still evolving, which means some websites may detect the current implementation.

Benchmarks and Final Comment

To put rayobrowse to the test, I ran a simple script against a single page for each of the most popular anti‑bot detection systems. These are the results I obtained:

Note: These tests were performed on my local machine using my ISP’s IP address.

As you can see, in this simple experiment rayobrowse achieved a 100% success rate, while Playwright failed consistently in headless mode and even struggled in some headful scenarios.

This suggests that the project is definitely worth keeping an eye on, especially thanks to its self‑hosted nature.

To be honest, and this is just my personal opinion as an expert who works in this field, I don’t usually get very excited about projects like this…. In my experience, many libraries of this type either get cracked down on or simply don’t receive the long‑term support they deserve. In this case, however, things are a bit different. The project is closed‑source and backed by a well‑known company in the industry, which makes the expectations for its future understandably much higher!

Here, I covered what the project is about, what it offers, how it works, and how to use it. As always, remember to use rayobrowse only for legal and ethical web scraping. Until next time!

FAQ

Why is rayobrowse based on Chromium and not Chrome?

rayobrowse is based on Chromium simply because Chrome is closed-source. Plus, tests performed on difficult websites show no meaningful difference in detection rates between Chrome and Chromium. Using Chromium also avoids false positives and reflects the broader ecosystem of Chromium-based browsers like Brave, Edge, and Samsung Internet.

Is rayobrowse open source?

rayobrowse isn’t open-source to prevent anti-bot companies from reverse-engineering it. Similar projects, like Camoufox, were quickly studied and countered once their code became public. Rayobyte decided to keep the project closed-source to help maintain its effectiveness and reliability over the long term.

Can everyone use rayobrowse?

No, not all companies can use rayobrowse. Its license prohibits organizations listed in Rayobyte’s restricted list from using the software. For everyone else, the project is free to download and run locally.

Does rayobrowse support proxy integration?

Yes, Rayobrowse fully supports proxy integration. You can route traffic through any HTTP proxy using the proxy query parameter on the /connect endpoint or via the proxy option exposed by the create_browser() function from the Python SDK. The proxy support includes authentication and rotating proxies.

We have seen this pattern in every scraping operation we have worked with. The knowledge exists. It is just scattered across wikis, Slack threads, internal repos, and people’s heads. The real problem is not documentation; it is retrieval. People write things down. They just cannot find them when it matters.

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

In THE LAB #77, we explored the concept of RAG (Retrieval-Augmented Generation) applied to scraped data and showed how to build a basic knowledge assistant using FAISS. That was a proof of concept. This time we are going deeper. We are showing the production system we actually built and use daily, and we are explaining the reasoning behind each design choice: why markdown, how embeddings work, which chunking strategy actually performs better, and what role auto-tagging plays in retrieval.

After reading this article, we hope you will understand the mechanics well enough to build the same system for your team.

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

What we are building and why

At TWSC, we have published around 300 articles over the past four years. Tutorials, reverse-engineering deep dives, tool comparisons, anti-bot analysis. When we sit down to write a new article, we need to remember what we have already covered, find previous work to link to, and check whether a technique we are about to describe was already explained in a past issue. Doing this by memory or by searching Substack’s archive stops working after the first hundred articles.

We also follow what the broader community publishes. Projects like Crawl4AI, which appeared on Hacker News, show that the need to ingest web content into structured, LLM-ready knowledge bases is shared across the industry. The tools for crawling and extracting content keep getting better, but the retrieval side, finding the right piece of information in a growing archive, still requires a purpose-built system.

So we built one. Here is what the complete pipeline looks like:

Sources                                  Processing              Storage & Retrieval
─────────                                ──────────              ───────────────────
Substack articles                   ──┐
                                      ├──> HTML-to-Markdown ──> Frontmatter + Tagging ──> Markdown files
Hacker News and other sources       ──┘

Markdown files ──> Chunker ──> Embedder (e5-large-v2) ──> PostgreSQL + pgvector

Search query ──> Query embedding ──> Cosine similarity search ──> Ranked results

Three stages, each independent and replaceable. You scrape content from your sources. You process and embed it. You search it.

If your team writes in Confluence instead of Substack, you swap the scraper. If you prefer Qdrant over pgvector, you swap the vector store. The architecture remains the same.

And here’s the hardware used for most of the steps, from embedding to the storage and retrieval: my DGX Spark.

Yes, I know, probably an overkill.

Check the TWSC YouTube Channel

The tools

Playwright handles browser-based scraping for our own Substack articles. Substack serves content dynamically and requires authentication for premium posts, so a plain HTTP client is not an option.

Algolia API (via Hacker News) provides structured search over HN stories. No scraping needed: HN exposes its full search index through public endpoints.

ScrapegraphAI and Firecrawl convert external article URLs into clean markdown. ScrapegraphAI is the primary extractor, Firecrawl is the fallback.

sentence-transformers with the intfloat/e5-large-v2 model generates 1024-dimensional embeddings. We will explain why we chose this model later in the article.

PostgreSQL with pgvector stores embeddings and handles similarity search. We chose it over dedicated vector databases because we already need PostgreSQL for metadata, and pgvector with HNSW indexing handles our scale without adding infrastructure.

Docker Compose ties everything together as three containers: PostgreSQL, the API server, and the indexer.

You can find the code in our GitHub repository reserved to paying users, inside the folder 101.KNOWLEDGE_BASE.

Need help with your scraping project?

Why markdown as the universal format

The first design choice we had to make was what format our knowledge base would store. We had content from Substack (HTML), Hacker News links (various formats), and potentially Confluence, Google Docs, or Slack in the future. We needed a common representation.

We chose markdown for three reasons.

First, markdown preserves document structure without carrying rendering noise. An HTML page contains navigation bars, ad slots, JavaScript, CSS classes, and layout dividers. None of that is content. When you convert to markdown, you keep headings, paragraphs, code blocks, links, and lists. Everything the embedding model needs, nothing it would choke on.

Second, markdown is readable by humans and machines alike. When something goes wrong in the pipeline, you can open a markdown file and immediately see what the system is working with. Try doing that with a serialized HTML DOM or a JSON blob from an API response.

Third, YAML frontmatter is a natural fit for markdown and gives us a structured metadata header without mixing it into the content. Each file gets an `id`, `type`, `title`, `publish_date`, `topics`, and `visibility` field. This metadata drives filtering at search time and never enters the embedding model. The separation is important: embeddings capture meaning, frontmatter captures facts.

There are two paths to get content into markdown. You can build your own converter using open-source libraries, or you can use commercial services that handle extraction and conversion for you. In this article we show both approaches deliberately. For our own Substack articles, we built a converter from scratch with BeautifulSoup and markdownify. It costs nothing, we control every detail, and it works because we know the source HTML structure intimately. For external content discovered on Hacker News, we use commercial services like ScrapegraphAI and Firecrawl instead, because every URL leads to a different site with a different HTML structure. Building custom converters for thousands of unknown domains would be impractical. The trade-off is clear: when you control the source, build your own; when you are scraping the open web, commercial extraction services save an enormous amount of development time.

Our Substack HTML-to-markdown converter is deliberately simple. It strips scripts, styles, buttons, navigation, and footers, then converts the remaining HTML:

def html_to_markdown(html: str) -> str:
    soup = BeautifulSoup(html, "lxml")
    for tag in soup.find_all(["script", "style", "button", "form", "nav", "footer"]):
        tag.decompose()

    md = markdownify(
        str(soup),
        heading_style="ATX",
        bullets="-",
        strip=["script", "style", "button", "form", "nav"],
    )
    md = re.sub(r"\n{4,}", "\n\n\n", md)
    return md.strip()

The final output for each document looks like this:

---
id: a1b2c3d4e5f6...
type: twsc_article
title: "THE LAB #94: Using Cookies and Session Persistence"
slug: the-lab-94-using-cookies-and-session
canonical_url: https://substack.thewebscraping.club/p/the-lab-94-using-cookies-and-session
publish_date: 2025-11-15
visibility: premium
topics:
  - browser-automation
  - cloudflare
  - scraping-infra
---

[article body in markdown]

Scraping your own content

The first source we built was a scraper for our own Substack articles. The pattern applies to any CMS: discover URLs, authenticate if needed, extract content, convert to markdown with frontmatter.

URL discovery and authentication

Most publishing platforms expose a sitemap. We fetch it, filter for article URLs (Substack uses /p/ in the path), and track the lastmod date to detect changes:

def fetch_sitemap(sitemap_url: str) -> list[dict]:
    req = Request(sitemap_url)
    req.add_header("User-Agent", "Mozilla/5.0 ...")
    with urlopen(req) as response:
        content = response.read()

    root = ET.fromstring(content)
    ns = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}

    articles = []
    for url_elem in root.findall("sm:url", ns):
        loc = url_elem.find("sm:loc", ns)
        lastmod = url_elem.find("sm:lastmod", ns)
        if loc is not None and "/p/" in loc.text:
            articles.append({"url": loc.text.strip(), "lastmod": lastmod.text or ""})
    return articles

Substack gates premium content behind authentication. We handle this with a persistent Playwright browser context that stores cookies across runs. On the first run you log in manually; after that, the saved session keeps you authenticated. For cron jobs, we verify the session by loading a known premium article and checking if the full content appears.

We try multiple CSS selectors for extraction because Substack has changed its DOM structure over time. The extracted HTML goes through the markdown converter we showed earlier.

Ingesting external sources: Hacker News

Read more

The traditional way to answer that question involved surveys, focus groups, and expensive reports from firms that charge you a fortune for data that’s already a few months old by the time you read it. Today, the data you need is sitting on public web pages: You just need to collect it.

In this article, we’ll discuss how to scrape data for market research, what sources actually matter, how to build a pipeline that doesn’t fall apart after a week, and where the legal lines are.

Let’s dive into it!

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

What “Market Research” Actually Means Web Scraping Professionals

Market research needs to answer three questions:

• “What are our competitors doing?”

• “What are our customers saying?”

• “How is the market moving?”

That’s it. Everything else is a variation of those three. And if you think about it, the web gives you access to all three, if you know where to look.

In practice, scraped market intelligence sits on three pillars:

• Competitive data: Pricing, product catalogs, feature changes, hiring signals. This is the “what are they doing?” pillar.

• Customer sentiment: Reviews, forum discussions, social media posts. This is the “what are people saying?” pillar.

• Market signals: Job postings, regulatory filings, trend volumes, new product launches. This is the “where is the market going?” pillar.

Now, why scraping instead of traditional research? Because scraping is real-time, it’s continuous, and it doesn’t depend on people filling out forms. A survey tells you what 500 people said last month. A scraper tells you what thousands of customers are saying right now, every single day, without anyone having to opt in.

That’s the competitive advantage. And it’s a big one.

For your scraping activity, you need IPs with good reputation. For this reason, we’re using a proxy provider like our partner Ping Proxies, that’s sharing with TWSC readers this offer.

💰 - Use TWSC for 15% OFF | $1.75/GB Residential Bandwidth | ISP Proxies in 15+ Countries

Where to Scrape: Sources That Actually Matter

Not all sources are worth your time. You could scrape the entire Internet and still end up with nothing useful if you’re not targeting the right places. Below is a list of high-value targets for market research and what you can extract from each:

• Competitor websites: Pricing pages, product pages, feature matrices, changelog, and blog posts. This is your primary source for understanding what competitors are offering and how they position themselves. Pricing pages, in particular, are gold. They change more often than you’d think, and tracking those changes over time tells you a lot about a competitor’s strategy.

• Review platforms (G2, Trustpilot, Amazon, Yelp): Customer pain points, feature requests, sentiment shifts. Reviews are unfiltered customer feedback. Nobody writes a G2 review because they were asked nicely in a survey. They write it because they feel strongly about something—and that’s exactly the kind of signal you want.

• Job boards (LinkedIn, Indeed): Hiring patterns reveal where a company is investing. If a competitor suddenly posts 20 machine learning engineer roles, that tells you something no press release will. Job postings are one of the most underrated market research signals out there.

• Social media and forums (Reddit, X, niche communities): Unfiltered opinions, emerging trends, early complaints about products. Reddit threads and niche forums are where people say what they actually think, not what they’d say in a focus group.

• Government and public data portals: SEC filings, patent databases, import/export records. These are slower-moving signals, but they’re authoritative. A patent filing can tell you what a competitor is building 18 months before it ships.

Here’s the key question to ask yourself before adding a source to your scraper: “Does this data answer a specific research question, or am I just hoarding?”. If you can’t tie a source to a concrete insight, skip it. You’ll save yourself storage costs, maintenance headaches, and potential legal issues.

Building the Pipeline: From Raw HTML to Market Intelligence

A market research scraper is not a one-off script you run from your terminal. It’s a pipeline. And pipelines need structure. If you treat it like a quick script, you’ll end up with a mess of cron jobs, inconsistent data formats, and no idea whether your data is fresh or stale. So, build it properly from the start.

A scraping for market intelligence pipeline should have four stages:

1. Collection: Fetch the pages, extract the fields you need, throw the rest away. Don’t store raw HTML “just in case” (you’ll learn why in the legal section of this article).

2. Storage: Store facts and metadata (source URL, timestamp, extracted fields). Use a structure that makes deduplication and versioning easy. In practice, this means designing your schema around a composite key (for example: source + entity ID + scraped timestamp) so you can track how a data point changes over time without overwriting previous records.

3. Transformation: Normalize the data across sources, deduplicate records, and enrich with additional context (geocoding, industry classification, entity linking).

4. Analysis: Turn rows into insights. This is where the actual market research happens. And to be clear: “Analysis” doesn’t mean opening a CSV and scrolling through it. The goal is to turn your pipeline’s output into dashboards, scheduled reports, or Slack alerts that reach the people who make decisions. If the data sits in a database and nobody looks at it, the whole pipeline is wasted effort.

Scheduling Matters More Than You Think

Different data types have different freshness requirements. Getting this wrong means either wasting resources or working with stale data. The main ideas to consider when engineering the triggering times are the following:

• Price tracking: Daily or hourly, depending on the market. Consider that e-commerce prices can change multiple times a day. SaaS pricing pages, instead, change less often. But when they do, it’s significant.

• Review monitoring: Monitoring reviews daily is usually enough. Reviews don’t appear in real-time, and sentiment trends are measured in weeks, not minutes.

• Job postings: A weekly schedule works for trend analysis of the job market. Remember that you’re looking for patterns, not individual listings.

• Social media: This depends on your use case. If you’re tracking a product launch or a PR crisis, you might need near-real-time. For general trend analysis, daily or even weekly batches work fine.

Tools That Work Well for Market Research Scraping

You don’t need to reinvent the wheel. The software industry already provides you with the best tools for your market research scraping pipeline. Here’s a solid stack for a market research pipeline:

• Scrapy for structured crawling. Scrapy’s architecture is designed for exactly this kind of work: You define spiders per source, plug in middleware for proxy rotation and retry logic, and use item pipelines to clean and store data as it flows through. For market research specifically, Scrapy’s built-in feed exports let you dump results straight to JSON, CSV, or even S3 without writing custom I/O code. And if you need to coordinate multiple spiders (say, one per competitor), Scrapy’s project structure keeps things organized as your source list grows.

• Playwright or Puppeteer for JS-heavy pages. The key difference from Scrapy is that you’re running a real browser, which means you can handle dynamic content, infinite scroll, and client-side rendering. The trade-off is resource cost: Each browser instance eats memory and CPU, so you don’t want to use this for targets that serve static HTML.

• A task queue for scheduling and orchestration. This is what turns a collection of scrapers into an actual pipeline. Instead of running scripts manually or relying on cron jobs, a task queue lets you schedule scrapes per source at different intervals, retry failed jobs automatically, and control concurrency so you’re not overwhelming a target site with parallel requests. It also gives you visibility: you can see what’s queued, what’s running, what failed, and why.

• PostgreSQL for structured market data that needs querying and versioning. Relational databases shine here because market research data is inherently relational: competitors have products, products have prices, prices change over time.

The point is this: Pick tools that let you build a maintainable system, not just a working script. Every tool in this stack solves a specific problem, and none of them requires you to build infrastructure from scratch. The best market research pipeline is the one that’s boring to operate, because boring means reliable.

Check the TWSC YouTube Channel

Scaling Without Getting Blocked

If you’re scraping one competitor once a week, you don’t need this section. If you’re tracking 50 competitors daily across thousands of pages, you do.

Here’s the reality: The moment you start scraping at scale, you become visible. But sites don’t like bots, even polite ones. So you need to be smart about how you scale. Consider the following rules of thumb to avoid getting blocked:

• Proxy rotation: Residential proxies for sensitive targets (sites with aggressive anti-bot systems), datacenter proxies for everything else. Rotate per request or per session, depending on the site’s detection mechanisms. The key is to not send thousands of requests from the same IP in an hour.

• Rate limiting and backoff: Be a good citizen. If you hammer a site with concurrent requests, you’ll get blocked, and you’ll deserve it. Implement exponential backoff on failures, and set reasonable delays between requests. A 2-3 second delay between requests is a good starting point for most sites.

• Fingerprint management: Headers, TLS fingerprint, and browser-level signals matter on sites with serious anti-bot systems. Make sure your request headers look consistent and realistic.

• CAPTCHAs: If you’re hitting CAPTCHAs regularly, your approach is too aggressive. Fix the root cause (rate, fingerprint, proxy quality) before reaching for solver services. CAPTCHA solvers are a band-aid, not a solution.

The general principle is simple: Scrape at a pace that doesn’t degrade the target site’s performance.

Need help with your scraping project?

Turning Scraped Data into Actual Market Insights

Let’s be clear about something: Raw scraped data is not market research. It’s just data. A CSV with 50’000 rows of competitor prices is not an insight. A chart showing that competitor X has dropped their enterprise tier price by 15% over three months: That’s an insight.

Here’s where the value gets created:

• Price tracking and competitive benchmarking: Track changes over time, visualize trends, and set alerts for significant moves. The goal is not to know what a competitor charges today. It’s to understand their pricing trajectory. Are they moving upmarket? Are they running more frequent discounts? Are they simplifying their tier structure? This is where predictive analytics meets scraped data with the goal of predicting future moves from your competitors.

• Sentiment analysis on reviews: Use NLP to extract themes from customer reviews. This is powerful for product teams who want to understand what customers love and hate about competitors. But remember: You’re analyzing the data internally, not republishing the reviews.

• Hiring signal analysis: Aggregate job postings by role type, department, and location. A competitor suddenly posting 15 ML engineer roles tells you they’re investing in AI. A wave of sales hiring in EMEA tells you they’re expanding geographically. This is a signal that’s almost impossible to get from any other source.

• Trend detection: Time-series analysis on product launches, feature changes, pricing moves, or social media mentions. The goal is to spot patterns or anomalies before they become obvious. If three competitors all add the same feature within two months, that’s a market trend, not a coincidence.

Overall, the output of your scraping pipeline should be dashboards, reports, or automated alerts, not a database dump that someone has to manually dig through. If the insights don’t reach decision-makers in a usable format, the whole pipeline is wasted effort.

Legal and Ethical Considerations: Don’t Skip This Section

I know, I know. You’re a developer, not a lawyer. But here’s a thing I’m sure you know: Most legal problems in scraping are self-inflicted. They happen because someone scraped “everything on the page,” stored it “for later,” and only then asked: “Wait, can we actually use this?”

As discussed in detail in “How to Avoid Copyright Violations While Scraping”, let’s go through the key legal and ethical principles of web scraping shortly:

• Scrape facts, not expression: Copyright protects expression, not facts. Prices, SKUs, dates, availability, and job titles are facts. No one owns the fact that a SaaS product costs $49/month. On the other hand, product descriptions, review text, and blog posts are creative expressions.

• Don’t store raw pages by default: Storing the HTML of entire pages means creating copies of copyrighted content. Instead, parse in-memory, extract only the fields you need, and discard the rest. If you need to debug, store a small sample with short retention.

• Respect robots.txt: The robots.txt file is not the law, but ignoring it is evidence of bad faith if things go sideways. In disputes, it can be used to show that you knew you were unwelcome and kept going anyway.

• Terms of Service matter: If the ToS explicitly forbids scraping and you scrape anyway, you may have a breach-of-contract problem. This is often easier for the site owner to prove than copyright infringement, because the argument is straightforward: you agreed to a contract, then you violated it.

• Don’t scrape behind a login: Once you log in, you’ve affirmatively agreed to a contract. Breaking that contract to scrape is a fast track to legal trouble. If your plan requires authenticated access, treat it as a licensing problem, not an engineering challenge.

• GDPR/CCPA: If you’re scraping anything that could be personal data (usernames, reviewer names, profile information), you need to know which privacy laws apply. This is especially relevant for review scraping and social media monitoring.

Here’s the mental model that works: A price comparison tool that shows prices and links back to the source? Generally safe. A product catalog that copies descriptions, images, and reviews so users never need to visit the original site? That’s where you get into trouble, even if you don’t publicly display the results because you use them for internal analysis.

Keeping Your Scrapers Alive: Monitoring and Maintenance

Scrapers in production break for several reasons. Sites change layouts, add anti-bot measures, restructure their URLs, or just go down for maintenance. If you don’t monitor your scrapers, your data goes stale silently, and you won’t know until someone asks why the pricing dashboard hasn’t updated in three weeks.

Here’s a breakdown of what you need:

• Dead selector detection: Alert when a CSS selector or XPath returns empty across multiple consecutive runs. A selector that worked yesterday and returns nothing today means the site changed its HTML structure. The keyword here is “multiple consecutive runs”. A single empty result could be a transient issue, so consider not triggering alerts on the first failure. Instead, set a threshold, like three consecutive empty results, before flagging it. When it does fire, you need to inspect the current page structure and update your selectors. Alternatively, try to go beyond the DOM using AI and LLMs, to make your extraction more resilient to layout changes in the first place.

• HTTP status monitoring: A spike in 403s means you’re getting blocked. A spike in 429s means you’re hitting rate limits. A spike in 404s means URLs have changed. Each of these requires a different response. For 403s, check your proxy pool and rotation logic: You might need fresher IPs or a lower request rate. For 429s, back off and increase your delays between requests; the site is telling you exactly what the problem is. For 404s, the target has likely restructured its URL patterns, which means you need to update your URL generation logic, not just retry the same broken links. Log these status codes per source and per run so you can spot trends early. A gradual increase in 403s over a week is a warning sign that your current setup is losing effectiveness, even if individual runs still return some data.

• Data quality checks: Row counts, null rates, value distributions. If your price tracker suddenly shows all prices as $0 or your review scraper returns empty text fields, you want to know immediately. Build quality checks into your pipeline as a post-scrape validation step, not as something you run manually. Compare each run’s output against baseline expectations: If you normally get 200 rows from a source and today you got 12, something is wrong, even if those 12 rows look fine individually.

• Automated tests against fixture HTML: Save sample HTML pages from your targets and write tests against them. When a test fails, you know the site has changed before your production scraper breaks. Treat your scrapers like production code, because they are. In practice, this means saving a snapshot of a relevant section in the target page as a local HTML file. Then, write unit tests that run your extraction logic against that fixture and assert expected outputs. Store these fixtures in version control alongside your scraper code. When a site changes and your production scraper breaks, update the fixture with the new HTML. This gives you a repeatable workflow for handling site changes instead of scrambling every time something breaks.

The goal is simple: You should know when something breaks before your stakeholders do. A Slack alert that says “Competitor X pricing scraper returned 0 results” is infinitely better than a product manager asking why the dashboard is empty.

Conclusion

In this article, you learned that market research scraping is about building a reliable pipeline that collects the right facts, transforms them into insights, and doesn’t get you in legal trouble.

The competitive advantage of scraping for market research is in what you do with the data. Anyone can code a scraper. But building a system that delivers reliable, actionable market intelligence week after week? That’s where the real value is!

So, let us know: Are you using web scraping for market research? What sources have you found most valuable? How did you structure your scraping pipeline? Let’s discuss in the comments!

The Proxy Price Benchmark is now updated weekly

If you haven't checked it yet, the Proxy Price Benchmark is the tool I built to answer a simple but important question: how much should you actually be paying for your proxies?

Every week, I (or better, my fleet of agents) update the pricing data directly from the vendors, so you always have a reliable reference to compare offers or negotiate with your current provider.

This week, we added two new vendors: Dataimpulse and AnyIP, bringing the total number of monitored providers to 27.

Check the latest prices

If you use proxies at scale and would find API access to this data useful, I am considering a paid API plan. If you are interested, join the waitlist and tell me about your use case. I want to understand demand before I build it.

This week on Scraping News: stealth browsers are getting serious

The Scraping News feed has been tracking an interesting trend this week: two new stealth browser projects worth watching.

Owl Browser is a purpose-built browser engine for automation at scale. Not a Playwright wrapper but a full engine built on Chromium (CEF) with a custom C99 HTTP server, 256 parallel contexts, and sub-12ms cold start. Self-hosted, Docker-ready, with Python and TypeScript SDKs. If you are running high-volume scraping and hitting the limits of standard headless setups, this is worth a closer look.

Rayobrowse is Rayobyte's open-source stealth Chromium browser, released from their production scraping infrastructure. It handles fingerprint randomization at the browser level (user agent, WebGL, fonts, screen resolution, timezone) and connects via CDP, so it works with Playwright, Puppeteer, Selenium, or any custom script. Runs on headless Linux with no GPU required.

Both address the same problem from different angles: standard headless Chromium is detected, and the solution is now moving from patch-level evasion to full browser-level stealth. We will be covering both in depth on TWSC soon.

See all the latest news on Scraping News

Keep in mind that both the Proxy Price Benchmark tools and Scraping News are in an early version; feel free to suggest improvements and bug fixes.

You might think that, in a web scraping scenario, the solution is simply to connect directly to the WebSocket channels. Sure, that’s possible, but there are a few obstacles along the way. The main ones are WebSocket anti-bot techniques and bot detection measures.

In this post, I’ll walk through the most common ones, explain how they work, and share proven tips and tricks to help you avoid them.

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

A Quick Intro to WebSockets

Before diving into WebSocket bot detection, let me first provide some context about WebSocket as a protocol and its role in web scraping.

What Is the WebSocket Protocol?

WebSocket, also abbreviated as WS for short, is a web protocol standardized in RFC 6455 that enables full-duplex, bidirectional communication between clients and servers over a single, persistent TCP connection.

Unlike HTTP, which is stateless and request-driven, WebSockets establish a long-lived connection through an initial HTTP handshake. After the handshake, both client and server can send messages independently, with data transmitted in frames that can be text, binary, or control frames (ping, pong, close).

WebSockets support fragmentation, masking, and optional compression via extensions like per-message-deflate, while newer HTTP/2 and HTTP/3 mechanisms allow multiplexing, reduced latency, and better proxy traversal.

For your ethical scraping activity, you need IPs with good reputation. For this reason, we’re using a proxy provider like our partner Ping Proxies, that’s sharing with TWSC readers this offer.

💰 - Use TWSC for 15% OFF | $1.75/GB Residential Bandwidth | ISP Proxies in 15+ Countries

Why and When Web Pages Use WebSockets

The WebSocket protocol opens the door to live, bidirectional web communication. Unlike HTTP’s request-response model, it lets servers and clients exchange data continuously over a single, persistent connection.

In general, WebSockets are essential for any application where low latency and frequent updates are required. Common use cases include:

• Live streaming: YouTube Live, TikTok LIVE, Kick, Twitch, and similar platforms.

• Chat applications: Slack, Discord, and other messaging services.

• Collaboration tools: Google Docs, Figma, and online whiteboards.

• Gaming and multiplayer experiences: Browser-based MMO games, turn-based games, and PvP games.

• Financial data feeds: Stock tickers, cryptocurrency price updates, and trading dashboards.

• IoT and telemetry: Sensor updates, home automation, and device monitoring.

• Notifications and alerts: Push updates for social networks, dashboards, or monitoring systems.

In short, WebSocket comes into play wherever instant, continuous communication is necessary (and standard HTTP polling would be too slow or resource-intensive).

Main Challenges of Scraping Data from WebSockets

Connecting to a WebSocket server for collecting data isn’t as straightforward as spoofing API requests for web scraping. In particular, the main challenges of scraping data straight from WebSockets include:

• Finding the right client implementation: You must use a WebSocket client (and there are way fewer than HTTP clients…) that supports the correct protocol version and any negotiated extensions, such as compression or subprotocols.

• Limited documentation and examples: WebSocket scraping is less common than API scraping, so there are fewer guides, tools, and community resources available.

• Proxy integration complexity: Not all clients support proxy integrations, making IP rotation a challenge.

• No request–response model: You can’t simply send a request and receive a response, as with API scraping. Instead, you must send the right messages and then listen to a continuous stream of events.

• Real-time data handling: You require a system to collect, process, and store messages in real time, often dealing with high-frequency updates.

Main WebSocket Anti-Bot Techniques and Solutions

Now you’re ready to discover the most important WebSocket-specific bot detection techniques, along with practical tips to avoid and bypass them. The idea here is to target a WebSocket server from an automated script, relying on a WS client in Python, Node.js, or another programming language of your choice.

WebSocket Handshake Issues

The WebSocket handshake is a transition phase in which an HTTP connection is upgraded to a persistent WebSocket connection. During this step, both the client and the server negotiate the connection parameters, and either side can abort the process if the conditions aren’t acceptable.

Because the handshake is where the protocol upgrade happens, it’s also a pivotal security and bot-detection point. The server must carefully validate everything the client requests. Otherwise, protocol misuse or security issues may occur.

In detail, during the handshake, a WebSocket client must send a valid HTTP/1.1 GET request with specific headers, for example:

GET /live-data HTTP/1.1
Host: example.com:9000
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: JKjeFfYU8mti9re0prPQrw==
Sec-WebSocket-Protocol: chat, superchat
Sec-WebSocket-Version: 13

In practice, browsers also include additional headers such as Origin, User-Agent, Referer, Cookie, as well as authentication headers (e.g., Authorization). While these HTTP headers aren’t strictly required by the WebSocket specification, they are extremely valuable for fingerprinting and bot detection.

Now, the server should respond with 400 Bad Request and immediately close the connection if it encounters:

• An unknown or malformed header.

• An invalid Sec-WebSocket-Key.

• An unsupported WebSocket version.

Instead, if the WebSocket version is unsupported, the server should return a Sec-WebSocket-Version header listing the versions it supports (most modern servers only accept version 13).

In practice, repeated handshake failures or non-browser-like handshake patterns are often treated as a bot indicator. Those may result in blocking, particularly after repeated handshake attempts from the same IP or when fingerprinting enables identification even across IP changes.

📌 Tips:

• Always send a valid Origin header: All major browsers include it, and many servers automatically reject WebSocket requests without one.

• Replicate real browser handshakes as closely as possible: Inspect the WebSocket request made by a real browser and match all headers (e.g., User-Agent and similar extra headers).

• Avoid excessive handshake attempts from the same machine: Too many connection attempts in a short time window are a common bot signal.

• Use IP rotation carefully: Rotation can help avoid rate-based blocks, but it doesn’t protect against fingerprint-based detection if the handshake remains identical.

Honeypot WebSocket Events and Channels

If you’re familiar with common anti-bot techniques, you’ve probably heard of honeypots. A honeypot is a decoy mechanism designed to attract bots by exposing fake or hidden resources, allowing systems to detect automated behavior when those resources are accessed or interacted with (e.g., invisible links or fake pages created to study bots).

In the context of WebSockets, honeypot events are a possible anti-bot technique to detect automated clients. With this approach, the server deliberately sends fake, misleading, or non-actionable events over the WebSocket connection. Similarly, the server might expose channels that aren’t meant to be accessed by regular clients.

Yet, automated scraping bots may react incorrectly to WebSocket honeypots by:

• Processing incoming data that is fake or intentionally invalid.

• Requesting access to or subscribing to channels they aren’t supposed to use.

📌 Tips:

• Study real browser behavior carefully: Inspect WebSocket traffic in your browser’s DevTools (“Network” → “Socket”) and observe which server messages actually trigger data flow or UI updates.

• Avoid assuming every message is meaningful: Remember that reacting to every event can lead to detection.

Connection Lifecycle Anomalies and Patterns

Since WebSocket channels are stateful (unlike stateless HTTP requests), servers can detect bots by analyzing connection behavior over time. Scraping bots tend to prioritize speed over realistic user behavior, which can produce identifiable patterns.

In this regard, popular bot-like indicators include:

• Very short-lived connections: Opening and closing sockets rapidly to collect data.

• Immediate reconnections after closure: Reconnecting instantly without human-like delays.

• High connection churn per IP: Multiple connections from the same IP within a short period.

• Missing browser events: Typical browser WebSocket clients trigger events like proper socket closure, whereas bots often skip them.

• Unnatural latency patterns: Servers use ping frames as heartbeats to check responsiveness. Real users on home Wi-Fi or mobile networks exhibit variable latency (jitter), while automated scripts deployed on data centers generally show extremely stable, low-latency responses.

📌 Tips:

• Introduce some randomness: Introduce realistic delays between connections and reconnections.

• Replicate intended behavior: Emulate browser close events if testing automated clients.

• Add latency variation: Consider latency variation when sending and receiving frames to mimic real-world network jitter.

• Rotate connection IPs: Use proxies to distribute WebSocket connections across multiple IPs.

WebSocket Binary Data Transmission

WebSocket servers sometimes choose to send binary data instead of plain text or JSON. The main technical reasons for this are:

• Reduced bandwidth: Binary messages omit field names and whitespace, making packets smaller than JSON strings and supporting high-frequency updates.

• Faster parsing: Binary data can be read as typed arrays or fixed-size fields, avoiding JSON parsing overhead.

• Custom protocols: Web apps can define their own compact binary format for predictable, high-frequency data.

• Efficient number storage: Numeric values can be stored in 1–4 bytes rather than as multi-character strings, saving space.

For instance, TikTok LIVE pages use WebSockets to stream updates (e.g., chat messages, view counters, and other statistics) in binary format:

Sure, binary data can be converted to text. So, you may think that’s not a problem…

Well, keep in mind that most web applications using binary data implementations include some form of compression or encryption. This adds significant complexity!

Reverse-engineering these systems is technically possible by inspecting browser WebSocket clients, analyzing request headers for compression hints, or trial-and-error with common compression methods. Still, that’s time-consuming and error-prone. Plus, encryption keys, salts, or other details can easily change with each deployment.

📌 Tips:

This time, the only piece of advice I have is to look for alternative data sources. Many WebSocket-based pages, including TikTok LIVE, use regular HTTP APIs to retrieve initial data.

Note: Why aren’t these APIs called server-side when the HTML page is generated? In the case of live data, it’s more reliable to fetch it on the client, because even a single second of latency could result in outdated or inconsistent information.

Thus, polling over those RESTful APIs instead of the WebSocket data streams can allow you to retrieve the information of interest without dealing with binary encoding, compression, or encryption challenges.

Check the TWSC YouTube Channel

WebSocket-Based Bot Detection Measures

The WebSocket protocol is built on top of HTTP, so they inherit many anti-bot techniques commonly used for HTTP requests. At the same time, due to its stateful and persistent nature, anti-bot solutions like WAF (Web Application Firewalls) can leverage WebSockets to detect automated behavior even more effectively…

As a result, WebSocket-based anti-bot measures are not only relevant when connecting directly to WS servers, but also when interacting with web pages through browser automation tools like Playwright and Selenium. That’s why you must know them!

Advanced TLS Fingerprinting

Traditional HTTP fingerprinting checks headers and TLS details. WebSockets extend this by combining the TLS handshake with WebSocket-specific framing, which is much harder to spoof. Signals include JA3/JA4 fingerprints, unusual cipher suite ordering, frame fragmentation patterns, and incorrect masking behavior.

Need help with your scraping project?

Continuous Device Fingerprinting

HTTP allows basic fingerprinting on a per-request basis, but it can’t verify whether the client’s environment remains consistent. The stateful nature of WebSockets enables servers to continuously validate device fingerprints over time. For example, servers can request Canvas/WebGL renders, available fonts, and other browser characteristics repeatedly. Any inconsistency can lead to an immediate block.

Real-Time User Behavior Monitoring

WebSockets allow live streaming of mouse, keyboard, and scrolling events back to the server. This enables a much deeper level of user behavior analysis compared to static HTTP requests.

After all, most browser automation scripts produce perfectly straight mouse movements or instantaneous clicks, while human interactions naturally include slight jitter, variable speed, and reaction delays. These differences make automated clients easier to detect when behavior is constantly monitored over a WebSocket connection.

Conclusion

Here, I introduced the WebSocket protocol and explained why and when it comes in handy. Specifically, you learned that it powers live data updates on web applications. Want to access that data? Well, it’s not as straightforward as you might think due to WebSocket anti-bot techniques.

In this post, I explored the most relevant WS bot detection methods, along with useful advice for bypassing them successfully. You also saw how WebSocket’s stateful, continuous data streaming can be used by WAFs and other advanced anti-bot systems for enhanced detection.

I hope you found this helpful and informative. If you have any questions or comments, drop them below. Until next time!

But a browser does everything the hard way. It downloads the full page, parses HTML, executes JavaScript, renders the DOM, loads images, fonts, and stylesheets. For each request, it allocates hundreds of megabytes of RAM and takes seconds to complete what an HTTP client could do in milliseconds. When a pipeline needs to scrape ten pages, this overhead is irrelevant. When it needs to scrape ten thousand pages, the browser becomes the bottleneck.

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

Consider a concrete scenario: we need to monitor the wishlist of an e-commerce account, pulling product data, stock levels, and price changes every hour across hundreds of items. Running Camoufox for every single API call would mean spinning up a full browser instance, navigating to each page, waiting for JavaScript to execute, extracting the data, and closing. For a hundred items, that is minutes of execution time and gigabytes of memory. The same API calls through an HTTP client would complete in seconds using a fraction of the resources.

As we measured in THE LAB #96, HTTP clients with TLS impersonation can be 27x faster than browsers on the same target. The difference is not marginal. It is the difference between a pipeline that runs on a single machine and one that requires a cluster.

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

The problem is that these two approaches are usually treated as mutually exclusive. Either you use a browser for everything, accepting the overhead, or you try an HTTP client and hope the anti-bot system does not block it. But many websites only need a browser at the gate: for the login, the initial challenge, or the session establishment. Everything after that is plain API calls.

If we can use a browser to earn a valid session and then hand it off to an HTTP client, we get the reliability of browser automation where it matters and the speed of HTTP everywhere else. That is the pattern we want to build. But the handoff is not as simple as copying a few cookies, and the traps along the way are worth understanding before building a pipeline around this idea.

The hybrid pattern

The idea is simple in principle. Many websites require a browser only at the gate: the login flow, the initial anti-bot challenge, or the session establishment. Once that gate is passed, subsequent requests are plain API calls or page fetches that do not require JavaScript execution. If we can extract the session state from the browser and replay it through an HTTP client, we skip the browser for 99% of the work.

The session state, in practice, means cookies. An authentication flow sets session cookies that the server trusts for subsequent requests. If we transfer those cookies from the browser to an HTTP client, the server should treat the HTTP client as the same authenticated user.

But cookies alone are often not enough. Modern anti-bot systems like Akamai do not just check whether you have the right cookies. They also check whether the client presenting those cookies looks like the same client that earned them.

This is where TLS fingerprinting enters the picture: if the browser that logged in was Firefox, but the HTTP client that reuses the cookies presents a Python TLS fingerprint, the server may reject the request or simply drop the connection without responding.

So the real challenge is not just transferring cookies. It is maintaining continuity across two different execution models: the browser and the HTTP client must look like the same entity to the server.

Check the TWSC YouTube Channel

Tool landscape

For this experiment, we used two tools.

Camoufox is a custom Firefox build designed for stealth. It spoofs fingerprints (WebGL, canvas, audio, navigator properties), patches headless detection vectors, and uses Playwright’s Juggler protocol for automation. We covered it extensively in THE LAB #65: Scraping Datadome-protected websites with Camoufox. Its role here is limited to one thing: logging in.

curl_cffi is a Python binding for curl-impersonate, a modified version of curl that mimics the TLS and HTTP/2 fingerprint of real browsers. It supports impersonating Chrome and Firefox at specific versions, which means it can present the same TLS fingerprint as the browser that established the session. Unlike a browser, it uses negligible resources per request and can process thousands of pages per minute.

The key property that makes this pairing work: Camoufox is Firefox-based, and curl_cffi can impersonate Firefox’s TLS fingerprint. The server sees a consistent Firefox identity across both steps.

You can find the code in our GitHub repository reserved to paying users, inside the folder 100.HYBRID_SCRAPING.

Need help with your scraping project?

The target: Net-a-Porter

We chose Net-a-Porter as our target. It is a luxury e-commerce platform protected by Akamai Bot Manager, with authenticated features (wishlists, account details) exposed through internal JSON APIs. This gives us a clean test case: the login requires a real browser (Akamai blocks automation tools at the login endpoint), but the authenticated API calls are plain HTTP requests that return structured JSON.

Please keep in mind that this is an experiment for study purposes, and we’re not inciting you to scrape Net-a-Porter or any other website, especially the part behind a login.

Before diving into code, we need to understand what we’re dealing with. Net-a-Porter’s architecture has three layers relevant to us:

Akamai Bot Manager sits in front of everything. It sets a cluster of tracking cookies (_abck, bm_sz, bm_s, ak_bmsc, and others) that are generated through JavaScript execution on the client side. These cookies prove that a real browser visited the page. Without them, API calls either fail or hang indefinitely.

The login API at /api/nap/wcs/resources/store/nap_il/loginidentity/v2 accepts a JSON payload with email and password. On success, it returns a 201 status with an Ubertoken in the response body. This token is the key to all authenticated endpoints.

Authenticated API endpoints like the wishlist API at /api/nap/wcs/resources/store/nap_il/wishlist/v2/{id} require both the session cookies and the Ubertoken passed as an x-ubertoken header. They return clean JSON with product details, stock levels, and metadata.

The experiment: what worked and what did not

We did not arrive at the final solution directly. The investigation path itself reveals the constraints of session handoff, so it is worth walking through each attempt.

Read more

Lots of tools have emerged to solve the issue of browser fingerprinting, which, for example, is one of the primary reasons for blocks even when using high-quality residential proxies. So, the need companies have for stable, scalable data collection makes anti-detect solutions essential for survival in the current status of the industry.

In this article, you’ll discover Dolphin{anty}: A powerful anti-detect browser that lets you orchestrate hundreds of unique, isolated browser profiles. You’ll learn its strengths, why you should consider it for your scraping or multi-accounting projects, and how it works with a practical guide.

Let’s dive into it!

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

What is an Anti-detect Browser?

An antidetect browser is a specialized web browsing tool designed to mask a user’s digital fingerprint, allowing them to appear as a distinct, unique visitor to websites and tracking systems. Standard browsers like Chrome or Firefox broadcast a user’s hardware and software data. An anti-detect browser, instead, enables users to customize and spoof these parameters for every session.

In the context of web scraping, web scraping professionals use this technology to bypass anti-bot measures that rely on browser fingerprinting to identify and block automated traffic. Anti-detect browsers can also be used in “multi-accounting” strategies. You can use them to create isolated browser profiles, each with its own unique fingerprint, cookies, and proxy IP. The common use case is that a single user can manage hundreds of social media, e-commerce, or ad accounts simultaneously without triggering security flags that would normally link the accounts together and lead to mass bans.

A successful data pipeline is made not only by the right tool to use, but also from the right IP address. Proxy providers like Decodo help you achieving your scraping goals.

Try Decodo Now

What is Dolphin{anty} and Why Consider it for Your Web Scraping Projects?

Dolphin{anty} is an anti-detect browser that allows you to manage hundreds of unique, isolated browser profiles for web scraping and multi-accounting. You can use it via its desktop application or programmatically, as it provides a flexible API for deep integration with your scripts.

The best part of using it is that you can orchestrate wide scraping operations without worrying about browser fingerprinting. Forget about immediate IP bans, CAPTCHAs triggered by suspicious metadata, or complex cookie management. Dolphin{anty} handles the masking of your digital identity for you very simply. Also, thanks to its built-in “Scenarios” builder and synchronizer, it can automatically replicate human-like actions across multiple profiles simultaneously. So, say goodbye to manual warm-up routines and the fear of losing accounts to anti-fraud systems.

The top reasons why you should consider it for your projects are the following:

• Advanced anti-detect capabilities: If you’ve been scraping for a while, you know that standard headless browsers often leak metadata that triggers anti-bot defenses. Dolphin{anty} solves this by providing real, unique digital fingerprints for every profile. It mimics user behaviors at a granular level, allowing you to bypass sophisticated detection systems without the constant headache of being blocked.

• Mass profile management: Managing a few accounts is easy, but scaling to hundreds or thousands is a different beast. Dolphin{anty} is built for scale. It allows you to orchestrate hundreds of isolated browser profiles from a single interface. Whether you are managing a massive farm of accounts for data collection or need to segment your scraping tasks, the tool provides the infrastructure to keep everything organized and efficient.

• Flexible API integration: For those who prefer code, Dolphin{anty} offers a robust API that integrates deeply with your existing Python or Node.js pipelines. This allows you to automate profile creation, launch browsers programmatically, and integrate the anti-detect capabilities directly into your custom scraping infrastructure.

For your ethical scraping activity, you need IPs with good reputation. For this reason, we’re using a proxy provider like our partner Ping Proxies, that’s sharing with TWSC readers this offer.

💰 - Use TWSC for 15% OFF | $1.75/GB Residential Bandwidth | ISP Proxies in 15+ Countries

Dolphin{anty}’s Main Features

Dolphin{anty} is packed with features designed to make multi-accounting and scraping easier. The main features you should know about it are the following:

• Real fingerprint generation: The core of Dolphin{anty} is its ability to provide genuine device fingerprints. Instead of just blocking trackers, it creates a unique digital identity for every profile you run. In practice, it manages over 20 parameters—from WebRTC to Canvas—so your scrapers look exactly like real users on real devices.

• Built-in Automation: You don’t always need to be a coding wizard to automate tasks. Dolphin{anty} offers a “Scenarios” builder that lets you create automated workflows visually. Whether it’s warming up accounts or parsing data, you can set these scripts to run automatically. And for those who prefer code, the flexible API allows you to integrate these profiles directly into your existing scripts.

• Profile synchronizer: This is a game-changer if you need to perform the same action across multiple accounts. The Synchronizer allows you to perform an action in a “master” profile, and the tool automatically repeats that exact action across all other selected profiles in real-time. This saves you a massive amount of time on routine interactions.

• Team collaboration: If you work in a team, you know that sharing browser sessions and cookies can be a nightmare. Dolphin{anty} simplifies this by allowing you to transfer profiles, cookies, and proxies to colleagues in just a few clicks. You can also manage permissions, ensuring that team members only have access to the functionality they need.

• Smart profile management: When you are dealing with hundreds of profiles, organization is key. The tool provides a highly intuitive interface where you can use tags, statuses, and notes to sort and find your profiles instantly. It’s built to help you navigate a large farm of accounts without getting lost in the chaos.

Check the TWSC YouTube Channel

Hands-on Dolphin{anty}: Step-by-step Scraping Tutorial

In this section, you will see how easy and fast it is to use Dolphin{anty}. Get ready for the tutorial!

Setting Up Dolphin{anty}

First of all, you need to create a new login. After creating a new account on Dolphin{anty}, the system will ask you to download the software. As you can see from the image below, it supports all the major Operating Systems:

Below is how Dolphin{anty}’s interface appears after you installed it on your machine:

Good. Everything is set up. Time to create new profiles!

Need help with your scraping project?

Create New Profiles

Before using Dolphin{anty}, you have to create a new profile. To do so, click on CREATE PROFILE and fill in the fields:

Profiles are the core of Dolphin{anty}. This is where, for example, you can change the fingerprinting for your anti-detect strategies. To do so, you only need to click on NEW FINGERPRINT, and the tool will change all the fingerprinting data for you. And if the standard fingerprinting is not sufficient, you can manage advanced configurations:

Also, if your use case needs to use a specific social media like Facebook, you can set Facebook’s URL as the starting page and the credentials to log in to a profile you need to manage:

When everything is set up, click on SAVE, and your profile is completed! You are now ready to use Dolphin{anty} via UI or code.

Use Dolphin{anty} Via The UI

The power of anti-detect browsers rely in allowing you to create different profiles and letting you use the browser with one instance, but different profiles. So, after you created the profiles, click on START to launch the instances:

Dolphin{anty} will launch a new browser instance, allowing you to manage as many profiles as you have created and activated. Below is the expected result:

That’s it for using Dolphin{anty} via UI!

Use Dolphin{anty} Via Code

Before using Dolphin{anty} via code, you have to create an API key. To do so, navigate through the API panel in the web app:

Now you can connect to a profile through a port generated at startup and automate the browser using tools like Puppeteer, Playwright, Selenium, and others.

Basic automation you can do includes the following:

1. Start a profile via API with DevTools Protocol enabled.

2. Connect to the profile’s port using a browser tool.

3. Run your own automation script through the open connection.

Dolphin{anty} allows you maximum flexibility, so you can use your favourite programming language. For example, below is how you can write an authorization script:

import requests
api_url = "<http://localhost:3001/v1.0/auth/login-with-token>"
token = "your-api-key"
request_data = {"token": token}
headers = {"Content-Type": "application/json"}

response = requests.post(api_url, json=request_data, headers=headers)
if response.status_code == 200:
	print("OK", response.json())
else:
	print("Error", response.status_code)

If the response is successful, you will receive a message like the following:

{"success": true}

Discover how to use Doplhin{anti} via API by reading the documentation!

Pros and Cons of Dolphin{anty}

Like any tool, Dolphin{anty} has its strengths and weaknesses. Here is a breakdown of what you need to know before deciding if it fits your stack.

👍 Pros:

• Top-tier fingerprinting: The ability to generate real, unique fingerprints for every profile is its biggest selling point. It goes beyond simple user-agents, making your scrapers look genuinely human.

• Built-in automation tools: The “Scenarios” builder and the Synchronizer are massive time-savers. You can automate routine warm-up tasks or replicate actions across dozens of profiles without writing a single line of code.

• Team-centric design: If you work with a team, the ability to transfer profiles and share them instantly is invaluable. It removes the friction of sharing session data manually via files or text.

👎Cons:

• REST API complexity: This is a significant friction point for developers. Unlike other solutions that offer native SDK wrappers, Dolphin{anty} relies only on REST API calls for automation. This adds “boilerplate” complexity compared to simply importing a library.

• Resource intensive: Running multiple browser profiles with full fingerprinting requires significant system resources. You will need a powerful machine if you plan to run dozens of concurrent sessions locally.

Conclusion

In this article, you discovered Dolphin{anty}, a flexible anti-detect browser that can be used both via UI and via code. As you’ve learned, it comes packed with interesting features that can speed up your processes. In particular, we found that the “Scenarios” feature is the one that actually makes it stand out.

So, let’s discuss in the comments: Were you already using Dolphin{anty} before reading this article? What’s your experience with it?

We are not talking about a cease-and-desist letter or a Terms of Service dispute. Google did not send SerpApi any communication before filing the lawsuit. No cease-and-desist, no attempt to resolve their concerns directly. SerpApi told us this was highly unusual, and that had Google reached out, they might have learned that their claims lack merit.

Google is invoking the Digital Millennium Copyright Act, specifically Section 1201, the anti-circumvention provision. The same statute originally designed to prevent people from cracking DVD encryption is now being pointed at a SERP scraping API.

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

We reached out to both Google and SerpApi for comment on this case. Google did not respond. SerpApi did, and we will include their statements throughout this article where relevant.

Let us break down what happened, why it matters, and what it could mean for anyone who scrapes the web for a living.

The Facts

Google’s complaint tells a straightforward story. SerpApi, founded in 2017 by Julien Khaleghy, operates a paid API that sends automated queries to Google Search and returns the results as structured JSON. Google estimates that SerpApi sends hundreds of millions of artificial search requests per day, and that this volume has increased by as much as 25,000% over the past two years.

In January 2025, Google deployed a technological protection measure called SearchGuard. SearchGuard works by sending JavaScript challenges to incoming search queries. For regular browser users, the challenge is invisible: the browser runs the JavaScript, sends back the expected response, and the search results load normally. For automated systems, the challenge is a wall. Bots that cannot execute JavaScript or that fail behavioral checks get blocked.

According to Google’s complaint, SerpApi’s response to SearchGuard was to build circumvention mechanisms. The complaint alleges that SerpApi creates “fake browsers using a multitude of IP addresses that Google sees as normal users,” misrepresents device and location information when solving challenges, and syndicates authorization tokens from legitimate requests to unauthorized machines around the world. Google also alleges that SerpApi uses automated means to bypass CAPTCHAs that SearchGuard deploys as a secondary verification layer. SerpApi disputes these factual allegations.

The complaint cites SerpApi’s own blog posts, where the company reportedly described SearchGuard as making “web scraping more difficult” but claimed to be “fortunate to be minimally impacted” because its services had “already pre-solved Google’s JavaScript challenge.”

The Legal Theory

This is where it gets interesting for the scraping industry, because Google chose not to sue under the Computer Fraud and Abuse Act (CFAA). That would have been the traditional route. Instead, Google went with the DMCA.

The context matters. The CFAA path has been significantly narrowed by the hiQ Labs v. LinkedIn case. In that landmark decision, the Ninth Circuit held that scraping publicly available data does not violate the CFAA, and warned against allowing companies to create “information monopolies.” The Supreme Court vacated and remanded the case under its Van Buren ruling, but on remand, the Ninth Circuit reaffirmed its original position.

After hiQ, the CFAA is a much weaker weapon against scraping of publicly visible content. Google needed a different legal framework. Section 1201 of the DMCA provides one.

Section 1201 has two relevant provisions. The first, Section 1201(a)(1)(A), prohibits the act of circumventing a technological measure that effectively controls access to a copyrighted work. The second, Section 1201(a)(2), prohibits trafficking in technology designed to circumvent such measures. Google’s complaint invokes both.

The argument chain goes like this: Google’s search results contain copyrighted content, specifically images in Knowledge Panels licensed from third parties, merchant-supplied product images in Google Shopping, and licensed content from Google Maps. SearchGuard is a technological measure that controls access to these search results pages (and therefore to the copyrighted works within them). SerpApi circumvents SearchGuard. Therefore, SerpApi violates Section 1201.

Each act of circumvention carries statutory damages of between $200 and $2,500. Google alleges billions of individual circumventions. Do the math, and the potential damages exceed what SerpApi could ever pay. Google itself notes in the complaint that SerpApi “reportedly earns a few million dollars in annual revenue, but already faces liability that is orders of magnitude higher and growing.”

Check the TWSC YouTube Channel

SerpApi’s Position

When we reached out to SerpApi, they were clear about their stance. On the fundamental legality of what they do, SerpApi told us: “We embrace the term ‘scraping,’ and we practice it legally and transparently. SerpApi accesses publicly visible search results, the same ones available to any browser, and delivers clean, structured JSON back to our customers. We’ve operated this way since 2017, serving developers, researchers, and businesses who need reliable access to public information at scale.”

On the legal boundaries of automated access to search results, their position is equally direct: “The law on this is clear, and we’re prepared to defend that position in court. Scraping is legal, and we stand behind our products and customers. Our API replicates real-time searches with no login, no bypass of any paywall, and no access to anything that isn’t already available to anyone with a browser. U.S. courts have upheld this repeatedly; hiQ Labs v. LinkedIn is a key precedent. The data Google surfaces lives on the open web. Google didn’t create it.”

In February 2026, SerpApi filed a motion to dismiss. Their arguments include the assertion that the DMCA is a copyright protection statute, not a website protection statute, and that Google is improperly trying to use it to control access to public portions of its website. They also argue that mimicking browser behavior to access publicly available pages is not the same as cracking encryption or disabling authentication, and that any ambiguity in the definition of "circumvention" must be given its narrowest reasonable reading, citing the "First Amendment interest in maintaining accessibility of the Internet as an open forum."

SerpApi also pointed out what they see as an absurdity in Google’s theory. If statutory damages were calculated at scale, the total “would exceed U.S. GDP.” Congress, they argue, never intended Section 1201 to be used this way.

On the DMCA claim specifically, SerpApi told us: “The DMCA’s anti-circumvention provision was designed to protect copyrighted works, full stop. Google is not protecting access to copyrighted works. Google is improperly attempting to use the DMCA to limit access to the public portions of its website. We believe that the law is on our side.”

Need help with your scraping project?

The Hypocrisy Argument

SerpApi is not shy about making this point. In a blog post about the lawsuit, they argue that Google’s case threatens access to public data on the open internet and this resonates widely in the scraping community. As they told us: “Google indexed the web without anyone’s permission. That’s how search works. Now it’s trying to pull up the ladder behind it, prohibiting the practices that it used, and still uses today, to build its business empire. That’s why SerpApi is standing up to Google. Not just to protect our business, but to protect legal competition and open access to public information on the internet.”

Google Search operates by crawling, indexing, and presenting content from billions of websites. Many of those website owners never explicitly consented to being indexed. Google’s position has always been that robots.txt provides the mechanism for opting out, and that the default state of the open web is crawlable. Now Google is arguing that its own search results should be exempt from the same logic.

The irony is not lost on legal commentators either. Above the Law( described the case as Google “pulling up the ladder after climbing it.” Eric Goldman’s blog published an extensive guest analysis arguing that Google’s DMCA strategy represents an attempt to relitigate hiQ Labs through a different statutory framework.

Why This Matters Beyond SerpApi

If Google’s legal theory prevails, the implications extend far beyond one API company. The core question is whether deploying an anti-bot system on a publicly accessible website is enough to invoke federal copyright law against anyone who bypasses it.

Think about what that means in practice. Every CAPTCHA, every JavaScript challenge, every behavioral analysis system deployed on a public website could potentially become a “technological protection measure” under Section 1201. Any scraper that solves a CAPTCHA, executes JavaScript to render a page, or rotates IP addresses to avoid detection could be committing a federal offense.

This is not hypothetical. The legal theory applies to any website that hosts copyrighted content (which is almost all of them) and deploys some form of bot detection (which is increasingly all of them).

Eric Goldman’s blog highlighted this exact concern. The guest analysis by Kieran McCarthy warns that accepting Google’s theory would allow any website deploying anti-bot technology to invoke federal law against circumvention, “transforming speed bumps and CAPTCHAs into federally enforceable copyright perimeters.”

The Electronic Frontier Foundation has also weighed in. Staff attorney Tori Noble stated that “the right to scrape publicly available information keeps the Internet free and open,” cautioning that overly broad DMCA interpretations undermine innovation and research.

SerpApi made a similar point when we asked about the impact on consumers: “Scraping-powered services benefit all kinds of consumers who use the web every day. Scraping helps to maintain the free and open flow of information across the internet, ultimately encouraging things like price transparency, competition, and informed decision-making, all to benefit consumers. Expanding the DMCA as Google has suggested would only benefit the largest tech incumbents and hinder transparency and healthy competition.”

The Emerging Legal Pattern

Google’s lawsuit does not exist in isolation. In October 2025, Reddit filed a 41-page complaint against SerpApi, Perplexity AI, Oxylabs, and AWMProxy in the Southern District of New York. The complaint is far more aggressive than Google’s, both in tone and in scope: six legal counts including three separate DMCA claims, unfair competition, unjust enrichment, and civil conspiracy.

Reddit’s framing is vivid. It describes the defendants as “similar to would-be bank robbers, who, knowing they cannot get into the bank vault, break into the armored truck carrying the cash instead.” AWMProxy is characterized as “a former Russian botnet.” Perplexity is compared to “a North Korean hacker.” The language is clearly designed to make scrapers look like criminals.

The underlying theory is similar to Google’s. Reddit has signed licensing deals with both Google and OpenAI to grant them programmatic access to its data. Companies that want Reddit content at scale are expected to pay for it. But when scrapers circumvent SearchGuard to harvest Google’s search results, they also harvest Reddit content without paying a cent. According to data Reddit obtained through a subpoena to Google, the three scraping defendants accessed almost three billion Google SERPs containing Reddit content in just two weeks during July 2025. SerpApi alone accounted for over 1.8 billion of those page accesses. Like Google, Reddit did not send SerpApi any communication before filing suit. SerpApi disputes these figures and the other factual allegations in Reddit’s complaint, and has filed a motion to dismiss in that case as well.

Reddit also produced a piece of evidence that reads like a detective novel. It created a hidden “test post” that could only be crawled by Google’s search engine and was not otherwise accessible anywhere on the internet. Within hours, the contents of that post appeared in Perplexity’s “answer engine.” The only way Perplexity could have obtained that content was through scraping Google’s search results. Reddit calls this technique the equivalent of “marked bills” in a bank robbery investigation.

The Reddit complaint also reveals a detail that connects directly to our industry: after Reddit sent a cease-and-desist letter to Perplexity in May 2024, Perplexity’s citations to Reddit content did not decrease. They increased forty-fold.

And in December 2025, in Ziff Davis v. OpenAI, a federal judge in the Southern District of New York ruled that robots.txt files do not “effectively control access” under Section 1201. Judge Sidney Stein compared robots.txt to a “keep off the grass” sign that “relies on readers to decide to comply rather than enforcing any kind of access control itself.” The ruling is important because it sets a baseline: passive, voluntary measures are not enough to trigger DMCA protection.

But SearchGuard is not robots.txt. It is an active system that executes JavaScript, performs behavioral analysis, deploys CAPTCHAs, and makes real-time decisions about whether to grant access. Whether this kind of system meets the “effectively controls access” standard is the open legal question. The answer will likely set the direction for the entire industry.

Legal commentators have identified what they call the “DMCA 1201 scraping strategy”: platforms deploy technological protection measures specifically to create legal standing under Section 1201, then sue when those measures are circumvented. The sequence is intentional. Deploy, document, sue. Whether courts view this as legitimate copyright protection or as strategic rent-seeking will determine the outcome.

There is also a relevant doctrinal debate. The Lexmark case in the Sixth Circuit introduced the “front door/back door” argument: if a house’s front door is unlocked, putting a lock on the back door does not mean the house is “access-controlled.” Applied here: if anyone with a regular browser can access Google Search results, does deploying SearchGuard against automated systems meaningfully “control access” to the copyrighted works within those results?

The AI Angle

There is one more layer worth noting. As Search Engine Land reported, OpenAI used SerpApi to scrape Google Search results for ChatGPT responses on current events, after Google declined to provide direct access to its search index. SerpApi listed OpenAI as a customer on its website as recently as May 2024 before removing the listing. Other reported customers include Meta, Apple, and Perplexity.

This context matters because Google already has a massive structural advantage in the AI race when it comes to fresh web data. Cloudflare CEO Matthew Prince put numbers on it: “For every one page that OpenAI sees, Google is seeing 3.2 pages.” Against Microsoft, the ratio is 4.8 to 1. The reason is simple. Publishers cannot block Googlebot without disappearing from search results. So Google gets access to the web at a scale that no competitor can match, and it can use that data not just for search but also for training and running its AI products.

In this context, suing companies that make it easier for competitors to scrape Google’s search results is not just about protecting copyrighted images in Knowledge Panels. It is also an act of defense of a competitive advantage. If OpenAI or any other AI company can get structured search data through SerpApi, they partially close the gap that Google’s crawler monopoly creates. Shutting down that channel through litigation serves Google’s position in the AI race, even if the complaint is framed purely in terms of copyright protection.

What Happens Next

The case is still in its early stages. SerpApi filed its motion to dismiss on February 20, 2026. According to the court docket, the initial case management conference before Judge Yvonne Gonzalez Rogers is scheduled for March 30, 2026, and a hearing on the motion to dismiss is set for May 19, 2026.

If the motion to dismiss fails and the case proceeds to discovery and trial, it will force courts to answer questions that have been left open since hiQ. Is a JavaScript challenge a “technological protection measure” under the DMCA? Can anti-bot systems on publicly accessible websites invoke federal anti-circumvention law? Does the DMCA protect the act of accessing a public webpage, or only the copyrighted works behind genuine access controls like encryption and authentication?

For the scraping industry, the stakes are high. A ruling in Google’s favor would give any website with copyrighted content and a bot-detection system a federal cause of action against scrapers. A ruling in SerpApi’s favor would confirm that the DMCA was not designed to protect public webpages from automated access, regardless of the technical measures deployed.

We will follow the case closely. Whatever happens, the days of operating in a legal gray area are coming to an end. The courts will have to draw a line, and that line will define the rules for the next decade of web scraping.

*Disclaimer: We are not lawyers. This article represents our analysis of publicly available court filings and legal commentary. Consult legal counsel for advice specific to your situation.*

The scraping industry is well aware of this problem. A recent analysis by ScrapeOps found that even though proxy prices have dropped by 67% over the past five years, the cost per successful payload has actually increased by 133%, mostly because anti-bot defenses now require heavier infrastructure. When each request costs more, wasting them on unchanged pages hurts even more.

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

Several approaches try to solve this. Tools like changedetection.io monitor pages for visual or structural changes and alert you when something is different. On the more technical side, recently explored using SimHash as a client-side fingerprint to determine whether a document has changed since the last crawl, without downloading the full body. These are valid strategies, but they all share one trait: they require you to build and maintain the change detection logic yourself.

What you might not know is that the HTTP protocol already has a native mechanism for this, and it has been part of the spec since 1999. It is called conditional requests, and it lets the server itself tell your scraper “nothing has changed” by responding with a 304 status and zero bytes of body. No diffing, no hashing, no client-side state management beyond storing a single header value.

We have written about proxy cost optimization before in articles like Optimizing Proxy Usage for Large-Scale Scraping and Analyzing the Cost of a Web Scraping Project, but we have never covered this technique. In this article, we will test it against real e-commerce sites and measure exactly how much bandwidth and money it can save.

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

How HTTP caching works (the short version)

When a web server responds to a request, it can include headers that describe the freshness and identity of the content. Two of these headers are relevant for our purposes.

The first is ETag, short for Entity Tag. It is a string that uniquely identifies a specific version of a resource. Think of it as a fingerprint of the page content. When the content changes, the ETag changes.

The second is Last-Modified, a timestamp indicating when the resource was last updated.

These two headers enable what HTTP calls conditional requests. The idea is simple. After your first request, you store the ETag (or the Last-Modified value) returned by the server. On the next request to the same URL, you send it back using the `If-None-Match` header (for ETags) or `If-Modified-Since` (for timestamps). The server compares your stored value with the current one. If they match, the server responds with status 304 Not Modified and an empty body. If they do not match, you get a regular 200 response with the fresh content.

A 304 response contains zero bytes of body. For a proxy billed per GB, that is a request that costs almost nothing in bandwidth.

The tools we used

The HTTP caching technique itself is protocol-level and works with any HTTP client that allows setting custom headers. You could implement it with Python’s `requests`, httpx, or even raw curl.

For this article, we used curl_cffi, a Python HTTP client built on top of curl-impersonate. Its main strength for our purposes is TLS fingerprinting: it can impersonate the TLS handshake of real browsers (Chrome, Firefox, Safari), which prevents e-commerce sites from blocking the request before we even get to test caching behavior. Without TLS fingerprinting, some of the e-commerce targets we wanted to test would have returned 403 immediately, making it impossible to evaluate their caching support.

Then later in the article, we’ll see if we can use the same approach with Scrapy.

Check the TWSC YouTube Channel

The audit methodology

Before attempting conditional requests, we need to check whether a target supports them. We wrote a simple audit function that makes two requests to any URL.

The first request is a standard GET. We capture the ETag, Last-Modified, and Cache-Control headers from the response, along with the response body size.

If an ETag or Last-Modified header is present, we make a second request with the corresponding conditional header (If-None-Match or If-Modified-Since). If the server responds with 304, the site supports conditional requests and we measure the bandwidth saving.

import time
from curl_cffi import requests


def audit_caching(url: str) -> dict:
    headers = {
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        "Accept-Language": "en-US,en;q=0.9",
        "Accept-Encoding": "gzip, deflate, br",
    }

    resp = requests.get(url, headers=headers, impersonate="chrome", timeout=30)

    resp_headers = {k.lower(): v for k, v in resp.headers.items()}
    etag = resp_headers.get("etag")
    last_modified = resp_headers.get("last-modified")
    cache_control = resp_headers.get("cache-control")
    response_size = len(resp.content)

    result = {
        "url": url,
        "status": resp.status_code,
        "etag": etag,
        "last_modified": last_modified,
        "cache_control": cache_control,
        "response_size_bytes": response_size,
        "supports_304": False,
    }

    if etag or last_modified:
        time.sleep(2)

        cond_headers = dict(headers)
        if etag:
            cond_headers["If-None-Match"] = etag
        if last_modified:
            cond_headers["If-Modified-Since"] = last_modified

        cond_resp = requests.get(
            url, headers=cond_headers, impersonate="chrome", timeout=30
        )

        result["conditional_status"] = cond_resp.status_code
        result["conditional_size_bytes"] = len(cond_resp.content)
        result["supports_304"] = cond_resp.status_code == 304

    return result

You can find the code in our GitHub repository reserved to paying users, inside the folder 99.CONDITIONAL_SCRAPING.

Need help with your scraping project?

Shopify stores: full conditional request support

We focused our testing on Shopify stores because, while working on various scraping projects, we came across several Shopify-hosted sites that had this caching system enabled. Shopify powers hundreds of thousands of online stores and is one of the most common scraping targets in e-commerce, so the finding felt worth investigating systematically. The results were clear: Shopify stores with the native page cache enabled support conditional requests out of the box.

Allbirds, Kylie Cosmetics, and Brooklinen all returned 304 responses consistently. Here is what we measured on Allbirds:

URL: https://www.allbirds.com/products/mens-tree-runners.json
Status: 200
Response size: 7,961 bytes

Caching headers:
  ETag: "page_cache:11044168:ProductDetailsController:de822deb7906aa6f9932541f4fe3dae9"
  Last-Modified: not present
  Cache-Control: not present

Conditional request support:
  304 Not Modified: YES
  Conditional response size: 0 bytes
  Bandwidth saving: 100.0%

The saving is 100% because the 304 response body contains exactly zero bytes. The only cost is the request/response headers, which are a few hundred bytes.

This behavior was consistent across three types of Shopify endpoints. The Product HTML page is the standard storefront URL that a browser would load (e.g. /products/mens-tree-runners), which includes the full rendered page with images, reviews, and theme assets. The Product JSON endpoint is the same URL with .json appended (e.g. /products/mens-tree-runners.json), which returns only the structured product data: variants, prices, inventory, and metadata. The Catalog JSON endpoint (/products.json) returns the first page of the store’s entire product catalog in a single response.

We ran repeated conditional requests on each endpoint and confirmed that all returned 304 consistently. The ETag stayed stable as long as the product data did not change.

Read more

Lots of tools and libraries for the main programming languages have indeed driven web scraping to significant growth. The need companies have for data also makes such growth the actual reason for existing.

In this article, I’ll talk about Kadoa: A tool that lets you create resilient scraping workflows in minutes. I’ll show you its strengths, why you should consider it, and how it works, with a practical guide.

Let’s dive into it!

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

What is Kadoa?

Kadoa is a web scraping tool that automatically and programmatically extracts web data at scale. You can use it either via the UI or via code, as it has SDKs and provides you with REST APIs.

The best part of using it is that you can just paste the target URL and the tool retrieves the data for you. Forget about anti-bot measures, fingerprinting issues, or proxy management: Kadoa does all of that for you very simply. Also, thanks to its AI engine, it can automatically recognize the structure of the data you want to scrape from a target website. So, say goodbye also to CSS selectors and any other strategy you use to go beyond the DOM using LLMs.

Using the right tool is just the first steps for a successful data extraction pipeline. Having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

Why Consider Kadoa for Your Web Scraping Projects?

The top reasons why you should consider Kadoa are the following:

• Scrape via workflows: Kadoa’s UI is built to help you set scraping workflows step-by-step. Insert your target URL(s), define the data schema (or let AI make the work for you), and choose to scrape all the available pages or to remain on page and see the agent work for you.

• Write code only if you need it: Other than the UI, Kadoa provides you with Python and JavaScript SDKs and a wide set of REST APIs you can call. This allows you to create workflows via UI, but to manage and call them via code if you need to.

• Integrated data quality management: Before starting the scraping process of your target data, Kadoa allows you to manage data quality. In practice, it allows you to set data quality rules or to manage the rules it provides you, thanks to its AI agent.

• Easy proxy management: If you’ve been scraping for a while, you know that you have low chances of successfully scraping the majority of the content you need without using proxies. Using proxies is not a very big issue if you are used to it and if you already have a favourite provider. However, Kadoa simplifies proxy management. It already provides you with a list of countries you can choose from and, under the hood, it manages everything that’s needed to integrate proxies in your workflow.

• Scheduling feature: There are cases where you need to scrape the same target data from time to time. Or, eventually, you’d like to be notified when data in a target page has changed. Kadoa provides both these features. You can choose to schedule your workflow to scrape at precise time intervals. You can also choose among different notifications, one of which is getting notified when data is changed.

Kadoa’s Main Features

Below is a list of Kadoa’s top features to help better understand its potential:

• Simple and intuitive UI: Kadoa’s UI is simple and intuitive. It allows you to create workflows in minutes. Every scraping workflow is subdivided into steps, and Kadoa provides you with different screens. In a matter of a few minutes, you can define your preferred setup, insert the target page(s), and leave it scraping for you.

• Chrome extension: Other than the UI, Kadoa provides you with a Chrome extension. If you are a Chrome user, this feature allows you to define everything you need directly on the target page, then trigger the workflow to let Kadoa’s agent start scraping.

• Code integrations: If you are a developer or if you simply need to invoke your workflows via code, Kadoa offers you two possibilities. It provides you with Python and JavaScript SDKs in an open-source repository, so that you can use custom code to invoke your scrapers. Also, if you like to use code but prefer REST APIs, Kadoa provides you with several endpoints.

• Scraping suitable for structured or unstructured data: One of the difficult aspects you may encounter when manually scraping websites is defining how to grab unstructured data. This is one of the typical use cases where you could use AI to detect patterns in data in your scraping projects. The good news is that you don’t need to come up with imaginative solutions. Kadoa automatically retrieves unstructured data for you thanks to its AI engine.

• Data schemas definition: The tool provides you with a feature that allows you to define recurrent data structures. This can be helpful when you retrieve similar data from different websites. If you leave its AI engine to automatically define the data structure, in such cases, you could lose consistency across similar data.

• Proxy and anti-detection features: Forget about anti-bot measures and proxy management. Kadoa manages anti-bot solutions under the hood. It also provides you with a predefined list of locations you can choose from, and it will automatically set coherent proxies.

• Error handling: It provides you with advanced error handling management. Common cases are when the target site goes offline, is under maintenance, or encounters a technical issue. When this happens, Kadoa detects the problem, it notifies you, and automatically retries the data extraction. If recovery still fails, its support team is notified and investigates.

• Integration capabilities: The software allows you to integrate with several third parties. One interesting one is the integration between n8n and Kadoa, which allows you to get your scraping automation workflow a step forward.

• Pricing model and usage graphs: Kadoa offers a free tier option, for which you can use 500 credits. Its pricing model is based on credit consumption, and it provides you with a UI section where you can see a graph of the consumption.

• Extensive docs: Kadoa has extensive documentation that covers both UI and API usage.

Check the TWSC YouTube Channel

Hands-on Kadoa: Step-by-step Scraping Tutorial

In this section, I’ll show you how to use Kadoa on an actual scraping task via the UI. The workflow will retrieve Intel’s historical price from Yahoo Finance:

In this scraping workflow, I will:

• Set the target web page.

• Define the data schema.

• Set scheduling options and notifications.

• Retrieve the actual data.

Before starting the actual workflow, log in to Kadoa. Below is the first access page you will see:

Perfect! You are now ready to create your first scraping workflow with Kadoa.

Step #1: Create a New Workflow

From the main page, click on Add workflow to create a new one and paste the target URL. The Proxy location box allows you to select a country where proxies are localized; leave it to AUTO to let the tool automatically manage it. Click on Continue to proceed with the next step:

Note that inside the Enter one or more URLs box, you have to insert the target page. If the target page is more than one, you can insert all the target pages you are interested in.

Alright, you created a new workflow in Kadoa. Let’s proceed with the next step and customize it!

Step #2: Define the Data Schema

As the next step, define the data schema:

If you want to insert the schema manually, Kadoa already provides you with some predefined schemas. For this tutorial, I’ve chosen to let AI do the job. So I selected AI Suggest Fields.

The system, then, asks you how you want to navigate the data. For the sake of this example, I decided to scrape only the current page from the target one, but you can also choose among three different options:

After clicking on Continue, the agent will start doing its job:

Step #3: Review Extracted Fields and Schedule the Workflow

Because I let AI work, the agent automatically tries to extract the data from the target page. But before proceeding, Kadoa asks for your review:

As you can see from the previous image, the agent has correctly detected the data to extract from the target page. Also, this job is finely improved as the tool provides you with a screenshot of the data it will extract, so that you can visualize it even better.

In the next step, you have to define the scheduling:

For the sake of this example, I decided to run the workflow only once. But, as you can see, you can choose among several scheduling options.

Step #4: Set Notifications and Final Details

As the next step, define the way you want to be notified:

In this case, I decided to be notified via email if the workflow fails. You can add different notification channels by clicking on Add channel:

Next, define the latest details of your scraping workflow:

Before starting with the actual scraping, the system asks you to approve the sample data it proposes to you or to review the data quality rules:

By clicking on Review rules, the tool provides you with automated data quality rules. You can select them if you think this will improve the quality of the scraping result:

When you are done reviewing quality rules, click on Approve. The actual scraping workflow will start and will be queued:

Et voilà! You have launched your first scraping workflow with Kadoa.

Need help with your scraping project?

Download Data, See Logs and Statistics in Kadoa

The workflow section reports all the workflows you created, their status, and the token consumption for each scraper:

By clicking on one workflow, you can see the data it retrieved and can decide the format you want to download it:

The Activity log page reports detailed logs of every action occurred to your workflows:

The Usage page reports graphs of the trend in terms of active workflows and the number of rows extracted for workflow, as well as the remaining total tokens on your plan:

Manage Kadoa’s Workflows via APIs

As introduced before, Kadoa provides you with several endpoints for making calls via REST APIs. The APIs allow you to perform several actions that are not strictly necessary for workflows already created. For example, you can start crawling sessions and create data schemas.

Before using the API, get your API Key under the Settings page:

If you want to manage already existing workflows, either created via the UI or APIs, you have to use the specific workflow’s ID via the UI.

Then you can perform several actions by invoking the REST endpoints. For example, you can schedule a particular workflow for later:

curl --request PUT \\
  --url <https://api.kadoa.com/v4/workflows/{workflowId}/schedule> \\
  --header 'Content-Type: application/json' \\
  --header 'x-api-key: <api-key>' \\
  --data '
{
  "date": "2025-02-07T10:00:00.000Z"
}
'

Where you have to insert the following:

• workflowId : Is the ID of the workflow you want to schedule.

• <api-key>: Is your KadoaAPI key.

• The actual date you want your workflow to start the scraping task. You have to use the ISO format for the date in UTC.

Kadoa: Final Comments

After analyzing and testing the tool, I can say the following are its main advantages and disadvantages:

👍 Pros:

• Ready for AI integration. You can download the scraped data or integrate it into your AI projects directly via API.

• Suits all the user needs, as it provides APIs, SDKs, and the UI.

• Supports structured output formats, including JSON.

• Offers virtually unlimited scalability on the side of infrastructure management and the number of URLS to scrape.

• Focuses on data quality before scraping, not later.

👎 Cons:

• Currently, it supports only 5 proxy locations.

• You can’t scrape all the websites you’d like:

Conclusion

In this article, I’ve presented Kadoa: An AI-powered scraping tool that helps you simplify your scraping projects. As you’ve seen, this is a ready-to-use tool that creates scraping workflows in minutes via UI and also supports code.

Let us know in the comments: Did you know this tool before? Have you already tested it?

In this article, I’ll break down the main reasons so you can understand why LLM-ready scrapers work this way—and how you could even build a simple one yourself!

A Brief Reminder About Web Scraping Tools for AI

With the recent rise of AI, some web scraping solutions have specialized in returning content optimized for LLM ingestion.

That means the content returned by AI-ready web unlockers or open-source scraping libraries isn’t just plain HTML. On the contrary, you often get an optimized Markdown version of the page. (Sometimes it’s even parsed JSON, but that’s a different story I won’t cover here.)

The Markdown content is then ready to be processed by an LLM as part of an AI agent, an AI workflow or pipeline, a multi-agent system, or similar system. In some cases, these web scraping tools are even accessed autonomously by AI agents, which decide when to use them to retrieve web content based on the task at hand.

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

Why AI-Ready Web Scrapers Choose Markdown in the First Place

Let me introduce you to Markdown, the language spoken by LLMs.

Note: I assume you already know what Markdown is, but if not, read its Wikipedia page (as it gives a quick overview with everything you need to know about its syntax).

A Bit of Context About Data Formats in LLMs

Most LLMs can handle pretty much any text-based format you throw at them, whether it’s plain text, HTML, JSON, CSV, XML, or others. Some even have vision capabilities and can understand images or other multimodal content.

Still, under the hood, most LLMs actually “speak” Markdown. That’s how they handle code blocks, tables, and other structured content, if you’ve ever wondered…

I’m sure I’m not the only one who has received a response from ChatGPT or Gemini in pure Markdown, even if I didn’t ask for it. Or sometimes, you can even catch the LLM responding in Markdown, and the page renders it in real time:

Note that the LLM is writing the “```” characters used in Markdown to signify code blocks.

Why Markdown Is a Perfect Data Format for LLMs (and AI-Ready Scrapers Use It Too)

So, cool, LLMs love Markdown and use it behind the scenes. But why? Well, because Markdown is a versatile format that hits all the sweet spots LLMs care about:

• Structured content: Markdown gives you hierarchy and organization out of the box (H1, H2, H3, lists, images, code blocks, etc.), making it easy for LLMs to parse and understand the structure of your content.

• Concise and LLM-friendly: Compared to raw HTML or JSON, Markdown is much more concise. Less unnecessary markup or structure means fewer tokens consumed, which also reduces the risk of hallucinations, truncations, or context overflows.

• De facto standard: While there’s no single formal Markdown standard, GitHub-flavored Markdown has become the widely adopted baseline, so most tools and scrapers default to it.

• Rich content support: Markdown supports images, links, tables, code snippets (and in some cases, such as with MDX/MarkdownX, even raw HTML or embedded React components), making it flexible for a wide range of content types.

• Alignment with training data: LLMs are trained on massive datasets like Common Crawl, where a huge portion of high-quality technical documentation (READMEs, wikis, Stack Overflow posts, etc.) is written in Markdown. This means most AI models don’t just “understand” Markdown. Instead, they learned to reason through its structure during training, giving them a natural intuition for the format.

Long story short, that’s why most web scraping solutions built for AI integrations return content in Markdown (or at least give you the option).

By converting a scraped HTML page directly into Markdown, AI-ready scrapers help the underlying LLM (whether it’s part of a machine learning pipeline, AI agent, RAG workflow, plugin, or other application) process the content efficiently and effectively while also saving on token usage.

Markdown vs HTML

Still not convinced? Take a look at an HTML-to-Markdown conversion of a sports news page from ESPN:

As you can see, the original HTML page contains 125.88 KB of content. After converting to Markdown, it drops to 35.84 KB. That’s a ~28% reduction in size just from a simple data format conversion, without any significant loss of actual content!

If we look at token usage, the difference can appear even more striking. The original HTML page translates to 40,125 tokens:

Meanwhile, the Markdown version corresponds to only 11,175 tokens:

Again, that’s roughly a 3.6× reduction in token usage (which usually translates directly into cost savings, since most AI providers charge based on LLM usage).

For a more direct comparison between data formats, explore the AI data format comparison research piece (which I wrote in collaboration with Bright Data on its Kaggle account).

But Raw Markdown Alone Isn’t Enough…

Now, you might be thinking: “Okay, I’ll just convert HTML pages to Markdown using one of the many HTML-to-Markdown libraries out there, and I’m done.” Well… not quite.

The problem is that a direct HTML-to-Markdown conversion isn’t enough, and below are the main reasons why (and how to address them).

1. Non-Content HTML Tags Get Treated as Content

HTML pages are full of blocks that are required for rendering, but are completely useless for understanding the page itself. Think <script>, <style>, inline JSON configs, and similar tags.

After all, those HTML blocks contain plain text. Thus, conversion libraries (rightfully) treat them like any other text node and include them in the Markdown output:

This clearly pollutes the output with noise the LLM doesn’t need, while also greatly increasing token usage (as <script> and <style> blocks can be surprisingly long!)

🎯 Solution: Use an HTML parser (or, in simpler cases, well-scoped regexes) to remove <script>, <style>, and similar rendering-only HTML tags before converting the page to Markdown.

For instance, simply removing <script> and <style> tags from the input HTML produces an impressive reduction, from 1.93 MB down to 68.37 KB, which also translates into huge token savings!

2. Ads and Promotional Content

Ads, sponsored sections, and “recommended for you” blocks might have nothing to do with the main content of the page. Leaving them in the converted Markdown can confuse the LLM or skew its understanding of what the page is really about.

🎯 Solution: Use proxies that support ad-blocking when retrieving the HTML page, enable OS-based ad blockers on your deployment server, or apply rules to remove ads after fetching the HTML and before converting it to Markdown.

3. Navigation, Headers, and Footers

Menus, breadcrumbs, and footer links are all technically “content,” but could be semantically irrelevant for your use case (particularly if you’re not interested in links for crawling or further exploration).

If those elements aren’t removed or downweighted, they increase token usage. Plus, the LLM may overemphasize them or mistake them for part of the main content.

🎯 Solution: Conveniently remove tags like <header> and <footer>, or design your HTML-to-Markdown system to accept only specific CSS selectors for the blocks you want to include in the conversion process (just like Crawl4AI does).

4. Repeated and Boilerplate Text and Content

Things like “Sign up,” “Log in,” newsletter popups, cookie banners, or legal disclaimers (like GDPR notices) appear on almost every page of a site. Including them wastes tokens and adds repetition, which can degrade reasoning quality and increase the risk of hallucinations.

🎯 Solution: This is a tricky problem, as there’s no easy way to identify and remove all of these elements automatically. I know for a fact that some industry leaders have trained small LLMs specifically for this task, letting them process the remaining HTML (after earlier cleaning steps) to filter out all irrelevant content.

How to Convert a Web Page from HTML to LLM-Optimized Markdown

I was recently asked by a client to analyze specific web pages from competitors’ websites. These included structured pages with hidden elements that required basic interactions (like dropdowns). Plus, some information was spread across badge images, links, etc.

Now, if you’re trying to get high-level insights from web pages, your first idea might be to just copy all the text on a page (CTRL+A + CTRL+C) and paste it into ChatGPT (or a similar AI solution), analyzing it with the right prompt. That’s far from optimal, because you lose structure, links, image URLs, and other important context.

Instead, I wrote a simple Python script that:

1. Reads HTML from an index.html file.

2. Keeps only the <body> tag with Beautiful Soup for restricting the content to what you’re typically interested in.

3. Remove <script>, <style>, <header> and <footer> nodes.

4. Converts it to Markdown using markdownify.

5. Writes the output to an output.md file.

Let me show you this script!

HTML to LLM-Optimized Markdown Script

Here’s the simple script for converting HTML files to LLM-ready Markdown outputs:

# pip install beautifulsoup4 lxml markdownify

from bs4 import BeautifulSoup
from markdownify import markdownify as md

def html_to_markdown(html_input_path: str, output_markdown_path: str):
    # Load the input HTML from a file
    with open(html_input_path, "r", encoding="utf-8") as f:
        html = f.read()

    # Parse the HTML (using lxml for high performance)
    soup = BeautifulSoup(html, "lxml")

    # Remove the undesired tags
    for tag in soup(["script", "style", "header", "footer"]):
        tag.decompose()

    # Keep only the <body> content (if present)
    body_html = soup.body.decode_contents() if soup.body else str(soup)

    # Convert the HTML to Markdown
    markdown = md(
        body_html,
        bs4_options="lxml" # Set the underlying HTML parser
    )

    # Write the Markdown output to disk
    with open(output_markdown_path, "w", encoding="utf-8") as f:
        f.write(markdown)


if __name__ == "__main__":
    html_to_markdown("index.html", "output.md")

Check the TWSC YouTube Channel

How to Use the Script

The script above still involves a few manual steps, but it greatly improves the transformation of a web page into content that’s ready to be sent to any LLM.

First, load the target page in your browser (ideally with an ad blocker enabled):

Once the page has fully rendered, right-click and select the “Inspect” entry. Locate the <html> tag, then use “Copy > Copy outerHTML” option to get the complete HTML of the rendered page:

Note: Copying the rendered HTML is better than copying the HTML from the “View page source” option. The latter misses all dynamic content (basically, anything that requires JavaScript execution and rendering in the browser won’t appear in the raw page source).

Next, in your project folder, paste the HTML into a file named index.html:

Run the Python script, and it’ll generate an output.md file:

You can now pass the resulting Markdown to an LLM for processing.

Compared to a traditional HTML-to-Markdown approach, the tweaks in this process save tons of tokens. In particular, the output produced by this method is just 41.2 KB (compared to 611.68 KB for the original HTML), which corresponds to 11,006 tokens.

If you applied a basic HTML-to-Markdown conversion, you’d end up with a 430.21 KB Markdown file, resulting in 154,191 tokens:

In other words, these basic tricks lead to over 14× token savings. Not bad!

Et voilà! Simple, manual, but highly effective.

Need help with your scraping project?

Next Step

The script I just shared achieves its goal, but it’s super straightforward. I presented it simply to prove how a basic HTML-to-Markdown conversion is essentially suboptimal.

For a more sophisticated result, you could integrate a similar process into your LLM-ready scraper, including CLI options for more control over which tags to remove, which nodes to select, and other conversion settings.

As a project idea, you could even turn this approach into a browser extension that converts rendered web pages in a user’s browser into LLM-ready Markdown output files. Clearly, if you go this route, make sure to follow ethical web scraping practices.

Is Markdown Always the Right Choice?

This is the final question to ask after all this discussion. Now, you might be thinking: “Okay, there are no good reasons to stick with plain HTML when passing web pages to an LLM.”

That’s not true since there are situations where having access to the raw HTML can make a difference. Think of when the HTML contains semantic attributes or metadata. This information would be lost during HTML-to-Markdown conversion.

In detail, these are some scenarios where sticking to HTML for LLM ingestion is beneficial:

• data-* attributes storing product IDs, prices, etc.

• ARIA attributes that convey accessibility or structural information.

• class or other HTML attributes that reveal context beyond the visible content on a node.

• HTML comments contain useful information about the page.

For example, consider this HTML node on an Amazon page:

This is just an empty <span>, but its data-state attribute contains information-rich JSON data that you would lose during the Markdown conversion (as the node doesn’t contain text).

Another common example is visual elements, which often carry semantic information not captured by visible text. For instance, based on the image below, you might think the rating is 5/5, but the aria-label attribute reveals it’s actually 4.3/5:

In short—let’s be honest, as it always happens in IT—converting to Markdown isn’t a one-size-fits-all solution. Therefore, it’s no surprise that most web scraping solutions built for direct AI integrations also offer the option to return raw HTML.

That said, based on my experience in the field and everything highlighted here, I highly recommend sticking to Markdown when feeding web pages to LLMs for processing or data parsing, as the benefits far outweigh the downsides in the vast majority of use cases.

Conclusion

In this post, I’ve outlined why Markdown is the language of LLMs and, consequently, the preferred output format for all web scraping tools that integrate directly into AI systems like workflows, pipelines, agents, and so on.

The reasons are intuitive: Markdown is concise and strips unnecessary markup (reducing token usage) while preserving structure, images, links, lists, tables, and more.

As highlighted, plain HTML-to-Markdown conversion isn’t always optimal, and you need to apply some extra tricks to get the best results.

If you have any questions or comments, drop them below. Until next time!

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

This makes SERP scraping fundamentally different from conventional web scraping. The data you collect is only as reliable as your control over these variables. Scrape from a datacenter IP in Virginia with a desktop Chrome fingerprint while logged out, and you will get one set of results. Scrape the same query from a mobile device in Milan while logged into a Google account, and you will get something entirely different. Both are “correct” Google results. Neither tells the full story.

In this article of The Lab, we wanted to understand how much these variables actually change the output, and more importantly, how to control them reliably.

Google does not want you scraping its results

Before we get into the technical setup, we need to acknowledge something that changed the landscape significantly in early 2025.

Starting in January 2025, Google began releasing SearchGuard, a technical protection measure designed to make scraping search results harder.

SearchGuard works by sending JavaScript challenges to search queries originating from unrecognized sources, as we covered on these pages when it started. When a query arrives, Google’s system transmits JavaScript code that requires the browser to compute and return a “solve”, a set of specific information about the browser environment and the user generating the request. For human users, the solution happens transparently in the browser. For automated systems, it is a wall.

This change in strategy put pressure on all “SEO tools” and the operators that needed to scrape Google search results, suddenly increasing their day-to-day operational costs.

Need public web data, not scraper headaches?

SerpApi turns search results into predictable JSON with built-in scale, location options, and speed. All with no maintenance.

Try for free

The change in this strategy, and especially its timing, prompted professionals to raise questions that will likely never receive an answer. Does this have something to do with the AI race? Is this a way to make it harder for other AI companies rely on Google searches for their answers?
We’ll probably never know the answers, but they’re legitimate questions: SERP scraping is as old as Google search, so why bothering stop bots in 2025 and not some years ago?
However, this is today’s reality, and we need to adapt to it. Let’s examine the specifics of SERP scraping on Google (as always, we’re showing this for educational purposes; be aware of current copyright and scraping laws).

What shapes a Google SERP response

To scrape Google Search reliably, we need to model the system we are interacting with. Google personalizes search results along several axes, each of which produces measurably different output.

Geographic location is one of the most impactful variables. Google determines your location through your IP address and, when available, browser geolocation permissions. A query for “pizza restaurant” from a New York IP returns local results for Manhattan. The same query from a Milan IP returns pizzerias in Milan. This extends beyond local searches: news results, shopping results, and even organic ranking order shift based on geography.

We’ll see in the test part of this article that changing location and mimicking another one is less trivial than expected, since not every proxy type works as expected.

Device type determines the structure and content of the SERP page itself. Mobile and desktop results are not just different layouts of the same data. Google serves genuinely different content. Mobile SERPs prioritize featured snippets, location-based answers, and nearby points of interest. Desktop SERPs give more space to organic links and Knowledge Panels. Some results appear exclusively on mobile or exclusively on desktop. For anyone collecting SERP data for analysis, this distinction is not cosmetic. It is structural.

Login state introduces personalization based on your Google account history. When you are logged in, Google uses your search history, location history, and account preferences to tailor results. When logged out, you get a more “generic” version of the results for your location and device. The difference can be subtle for generic queries and dramatic for anything Google considers personal.

Keywords, of course, are the main driver of change. But in addition to returning different results for different keywords, the answer layout also varies accordingly. If you look for “trousers”, you’ll see more shopping results and product data, while if you’re looking for “aspirin”, you’ll see a more traditional layout.

These four variables interact. A logged-in mobile user in Tokyo sees a fundamentally different page than a logged-out desktop user in London, even for the same query. Controlling all four simultaneously is what makes SERP scraping an infrastructure problem, not just a coding problem.

Check the TWSC YouTube Channel

Tools: An Anti-Detect browser and Selenium

Given the variables we need to control (device type and login state, specifically), and the fact that we are not building a massive scraping operation here, the best setup we can use is Playwright paired with an anti-detect browser.

We need a real browser, not just an HTTP request library like requests or httpx, because Google’s SearchGuard validates the browser environment through JavaScript challenges. A raw HTTP client has no JavaScript engine, no DOM, no window object. It cannot compute the “solve” that SearchGuard requires. The request simply fails or returns a challenge page. To pass these checks, we need something that renders JavaScript and exposes a complete browser environment.

But a standard browser is not enough either. Regular Chrome or Firefox, even when automated with Playwright or Selenium, carries detectable signals: the navigator.webdriver flag, predictable fingerprint values, and missing or inconsistent browser properties. Google’s systems can identify these inconsistencies and treat the session as automated.

That’s why we’re pairing Selenium with an anti-detect browser, which is a modified browser engine that spoofs the properties websites use for fingerprinting. Navigator properties, screen resolution, WebGL parameters, canvas behavior, AudioContext values, font lists, language headers, and device type. Instead of presenting the same default fingerprint every time, an anti-detect browser generates a consistent, realistic identity that looks like a genuine user on a specific device and operating system.

The critical feature for our use case is persistent profiles. An anti-detect browser manages browser profiles that survive across sessions. Each profile stores its fingerprint configuration, cookies, local storage, proxy, and device settings. When we start a profile, it resumes exactly where it left off. This means we can log into a Google account through one profile, close the browser, and reopen it days later with the session still active. Without persistent profiles, we would need to authenticate on every run, which is both impractical and a red flag for Google’s security systems.

For this article, we use Kameleo as our anti-detect browser. It runs as a local service (Kameleo.CLI) exposing a REST API on port 5050, controllable via a Python client. It supports Chromium-based profiles (Chroma) for Chrome and mobile device emulation, and Firefox-based profiles (Junglefox). Each profile is an isolated browser session with its own fingerprint, proxy, and cookies.

Need help with your scraping project?

Setting up the infrastructure: deploying Kameleo on AWS

Our Kameleo instance runs on a Windows EC2 in the US. This means that without a proxy, all traffic exits via a US-based AWS IP address. We will use this setup to demonstrate the difference between the instance’s own IP and a proxy claiming to be somewhere else. I’m sure you’ll be surprised by what we’ll find later.

Installing Kameleo on AWS

We installed Kameleo on a Windows EC2 instance using the standard graphical installer, no rocket science here. Once Kameleo is running on the AWS machine, it exposes its API on port 5050. Our Python scripts run locally and connect to the remote Kameleo instance over the network.

The architecture is straightforward: Kameleo manages browser profiles and runs the actual browsers on the AWS instance. Our local machine sends API commands (create profile, start browser, stop browser) and connects to the browser via WebSocket for Playwright automation. The AWS instance needs port 5050 open in its security group for this to work.

Every script in this article follows the same initialization pattern. We read the remote IP from an environment variable:

from kameleo.local_api_client import KameleoLocalApiClient
import os
kameleo_ip = os.getenv(’KAMELEO_IP’)

kameleo_port = os.getenv(’KAMELEO_PORT’, ‘5050’)

client = KameleoLocalApiClient(endpoint=f’http://{kameleo_ip}:{kameleo_port}’)

Test 1: setting the right location

As we said, one of the keys to extracting SERP data is setting the location we’d like to know more about. Our Kameleo installation is on an AWS US machine, so we expect to get SERP data from there. But if we want to change location?

We run the same query, “weather”, three times from the same AWS instance in the US. First, without any proxy, the traffic exits from the instance’s own IP. Then, through a residential proxy geolocated in Italy. Finally, through a datacenter proxy also claiming to be in Italy. For each run, we first visit whatismyipaddress.com to verify the exit IP, then navigate to Google, type the query in the search bar with randomized keystroke delays, and capture the results.
You can find the code in our GitHub repository reserved to paying users, inside the folder 98.SERP-DATA. If you’re one of them but cannot access the repository, please fill out this form.

In the file test_location_comparison.py, we’ll see how Google responds to us when we’re using different types of proxies.

Read more

In this article, we’ll discuss what copyright in the context of web scraping is, when it occurs, and how to avoid it.

Let’s dive into it!

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

What is Copyright Violation in the Context of Scraping?

Generally speaking, a copyright violation occurs when you reproduce, display, distribute, or create derivative works from someone else’s creative work without their permission (or without a valid legal exception). In the context of web scraping, the “creative work” involves (but is not limited to) the following:

• Articles.

• Images.

• Audio and video.

• Code (under particular conditions).

In other words, if you scrape and reproduce an article (even a small part of it) on your website without the author’s permission, you can be infringing copyright. Whether it is actually infringement depends on context (how much content you copied, how you used it, and which jurisdiction applies), but “a small part of a whole article” is not a safe harbor.

So here’s the thing to bear in mind: Just because some content is accessible on the Internet, it doesn’t mean you can take it. Even though some content is publicly accessible, ownership and reproducibility are not. This is why minding the data you scrape is one of the best practices for ethical scraping.

For your ethical scraping activity, you need IPs with good reputation. For this reason, we’re using a proxy provider like our partner Ping Proxies, that’s sharing with TWSC readers this offer.

💰 - Use TWSC for 15% OFF | $1.75/GB Residential Bandwidth | ISP Proxies in 15+ Countries

How Can Copyright Violations Occur While Scraping?

To avoid copyright infringements, you should know the common cases to take care of. Below is a list of common situations where copyright can be violated while scraping data from websites:

• Copying content: Technically speaking, scraping is copying. When you download a webpage’s HTML to your disk, you’ve made a copy. If that HTML contains creative expression, you have created a copy of copyrighted material. That does not automatically mean you are infringing, but this is the exact action copyright law regulates. And if you store, reuse, or republish that expression without permission (or a solid exception), you’re in infringement territory. Note that courts don’t need the copied content to be 1:1 identical. For them, “substantial similarity” can be enough.

• Copying images and media: Images are typically strongly protected. Scraping image URLs and hotlinking can still be risky, even if you report the source URLs while republishing the images. And, of course, downloading and rehosting is even more direct copying.

• Copying “creative fields” that look like “data”: Product descriptions, editorial blurbs, “about” sections, hotel/restaurant descriptions, FAQs, and similar content is often copyrighted text. While editorial blurbs and similar text are obviously copyrighted content, the others are not so obvious. The point to always take care of is in relation to “creative work”. A product description can be creative work when it contains original language, structure, or marketing copy. But not every description is protected. For example, a purely functional description text may have weak or no copyright protection, depending on the jurisdiction and the originality of the content itself.

• Scraping for training LLMs: Scraping web pages to get data for training LLMs is surely part of the evolving career of web scraping professionals. However, scraping data to train Large Language Models can trigger reproduction/derivative-work arguments in courts. This is still an evolving legal area, so you should not assume “transformative” automatically saves you from legal troubles, especially at scale. The issue between studio Ghibli and OpenAI on copyright violations due to LLMs’ training is one among the open ones, but keep in mind: allegations, investigations, and lawsuits are not the same thing as a final court ruling.

How to Avoid Copyright Violations While Scraping

Having legal issues is probably the worst nightmare for professional scrapers. So, how can you be sure you are not violating copyright while scraping? Below is a list of guidelines to take into consideration:

• Scrape facts, not expression: Copyright protects expression, not facts. Scraping the price of a stock, the temperature in London, or a flight arrival time doesn’t infringe any copyright because these are facts. No one owns the fact that today it is 20 degrees in London. On the other hand, scraping a journalist’s analysis about why the price of a stock moved in a certain direction, or a photographer’s image of London, is a creative expression.

• Transform, don’t replicate: When repurposing content (on your website or anywhere else), transform it. This is a general rule of thumb, but if you are in the US, one of your best defenses is “Fair Use”. But to claim this, your use must be transformative. For example, scraping Amazon reviews and posting them on your own e-commerce site is replicating, not transforming. Even summarizing reviews cannot be considered transformative in some cases, and even when it is, it’s not a guaranteed shield.

• Don’t store raw pages by default: As said before, storing the HTML of entire pages means creating copies. To solve this, you can follow two paths:

  • Parse in-memory.

  • Extract only the necessary content, not whole pages.

• Treat images as a separate “danger zone”: Images are a type of content that, during the whole Internet era, had the majority of copyright issues so far. The safest options are:

  • Using the website’s official APIs when scraping images, if available.

  • Scraping images under a Creative Commons license with compliance.

  • Asking and getting direct licensing from the owner.

Check the TWSC YouTube Channel

Standardized Processes to Stay Safe

So far so good, but let’s be honest: When you are taken by your daily job tasks, it’s easy to lose your compass. To avoid it, the best thing to do is to create standardized (and documented) processes and procedures so that you always operate under a guardrail. This section provides you with a couple of ideas you can implement as standardized processes to be sure you don’t violate any copyright while scraping.

Procedure #1 to Avoid Copyright Violations While Scraping: Develop a Copyright Risk Check

Most copyright problems in scraping are self-inflicted. This happens because developers often scrape “everything on the page,” save it “for later,” and only then do they ask: “Wait, can we ship this?”.

Before you add a field (or a selector) to your scraper, ask yourself the following questions:

• “Is this a fact, or is this someone’s writing?”: Prices, dates, SKUs, addresses, and opening hours are facts. A paragraph of an article is someone’s writing. Remember to treat those differently.

• “If I publish this, would it compete with the source?”: If your application lets users consume the content without clicking the original, you’re not “aggregating.” You’re substituting.

• “Am I copying just what I need, or am I copying the entire page?”: If the answer to this question is: “We only store it for debugging”, then you are building a copy.

• “How much am I taking?”: A single excerpt is one thing. Thousands of excerpts across a site start looking like a dataset designed to recreate the whole content.

• “What am I going to do with it later?”: Internal analysis is one risk profile. A public API that returns the scraped text is a completely different risk profile.

• “Is my plan defensible if someone sends a legal notice?”: If your only defense is “but the content is publicly available”, you don’t have a defense. As said before, public availability is different than ownership-

If answers to these questions feel shaky, the fix is usually boring: don’t collect it, collect less, or get permission.

Procedure #2 to Avoid Copyright Violations While Scraping: Build Your Scraper So It’s Hard to Do Something Dumb

If you want to stay out of trouble, don’t rely on “policy.” Rely on defaults and standards.

Here’s what I mean: The safest scraper is the one that can’t casually vacuum up article bodies, image files, and review text unless you deliberately build it that way.

Below is a process that works safely:

1. Fetch the page.

2. Extract only what you came for.

3. Store facts + metadata (source URL, timestamp).

4. Throw the rest away.

When you really do need to keep anything close to “content” (ie, media), treat it as a special case: short retention, locked-down access, and a reason written down somewhere if needed. Not “maybe we’ll need it later”: You must have a valid reason.

If you want a mental model, you can think of it like so: You’re not building a web scraper. You’re building a pipeline. And pipelines need guardrails.

Need help with your scraping project?

Examples: What “Safe-ish” Looks Like vs What Can Surely Get You in Trouble

Let’s be practical now and see some examples of what is generally safe and what is not. Of course: The following examples are not court outcomes. They’re the kind of setups that tend to be boring (safe-ish) or spicy (trouble-ish):

• Price tracker (safe-ish): You scrape SKU + price + availability + timestamp and show a price history chart. You don’t copy product descriptions or images. This is the classic “facts + original output” use case.

• Product catalog clone (risky): You scrape titles, descriptions, bullet points, images, and reviews, then you show them on your site. That’s not “data.” That’s content. You’re rebuilding their user experience.

• News aggregation (high risk): If you store headlines + links and add your own tags/filters, you’re closer to indexing. If you store full articles and users can read all the content as is without leaving your site, then you’re highly risking getting a trip to the nearest court.

• Review analytics (mixed): Using reviews internally to compute “top complaints this month” is one thing. Republishing reviews precisely as they are is another.

• Business directory (often safer, until you start copying the fluff): Name, address, phone, opening hours: These are usually factual. “About us” sections and photos, on the other hand, are where you cross over into copyrighted expression.

So notice the pattern: The moment your product starts looking like a substitute for the source, your legal risk goes up fast.

The Traps That Have Nothing to Do with Copyright (But Still Hurt)

Copyright is only one way scraping can go wrong. Plenty of scraping disputes are won on issues that are simpler to prove than infringement. Below are the big ones you should treat as “no trespassing” signs:

• Circumvention (DMCA Section 1201): If the site uses a login wall, CAPTCHA, paywall, anti-bot challenges, or IP blocking to stop you, and you write code to bypass those measures, you are potentially violating anti-circumvention laws. This is not “copyright infringement” in the traditional sense, but the practical takeaway is simple: If you have to defeat a technical barrier to get the data, you’re walking into a high-risk territory fast.

• Disregarding robots.txt: The robots.txt isn’t the law, but ignoring it has its implications. In disputes, it can be used as evidence that you knew you were unwelcome and kept going anyway. It can also be relevant to arguments about authorization and “bad faith,” even if it doesn’t create copyright liability by itself.

• Terms of service (contract risk): If the ToS explicitly forbids scraping (and most do), and you scrape anyway, you may be liable for breach of contract. This is often easier for the content owner to win than a copyright claim because the argument is straightforward: You agreed (explicitly or implicitly) to a contract, then you violated the agreement.

• Do not scrape behind a login: Once you log in, you have affirmatively agreed to a contract. Breaking that contract to scrape is a fast track to a lawsuit. If your plan requires authenticated access, treat it as a licensing/permission problem, not an engineering challenge.

Conclusion

In this article, we’ve discussed how copyright infringements can occur while scraping and how to avoid them. As said, it’s not always easy to understand when you are actually infringing copyright, as it depends on the governing laws which, often, are local ones. Still, the main ideas proposed can help you be conservative and stay pretty safe while scraping web pages.

So, let us know: Did you find those practices useful? Do you apply other frameworks to be sure you’re not violating copyrighted content? Let us know in the comments!tat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.”

Think about it: how would you convince several million people to share their spare internet connection with companies that you don’t know how they will use it?
Not an easy task, and some companies took shortcuts, as the IPIDEA case shows.
Let’s see it in detail: more than the takedown itself, we’ll use it as an opportunity to look under the hood: how do residential proxy networks acquire millions of IP addresses, what keeps them running, and why are they so difficult to shut down permanently? The IPIDEA case provides unusually detailed answers to all these questions.

What happens when Big Tech goes after the infrastructure that powers both scrapers and threat actors

On January 28, 2026, Google Threat Intelligence Group (GTIG) announced what they called the disruption of “one of the largest residential proxy networks in the world.” The target was IPIDEA, a name that most people outside the proxy industry had never heard. Yet according to Google’s analysis, IPIDEA’s infrastructure was being used by over 550 distinct threat groups in a single week, including state-sponsored actors from China, North Korea, Iran, and Russia.

This is not just a story about a takedown. It is a detailed look at how residential proxy networks actually work, how they acquire millions of IP addresses, and why disrupting them is harder than it sounds.

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

What Google Actually Did

The disruption involved three coordinated actions:

First, Google took legal action to seize the domains used to control enrolled devices and route traffic through them. Without these command-and-control (C2) domains, the SDK code running on millions of devices loses the ability to receive instructions and proxy traffic.

Second, GTIG shared technical intelligence about IPIDEA’s SDKs with platform providers, law enforcement, and research firms. The goal was to trigger ecosystem-wide enforcement, getting these SDKs flagged and removed across multiple app stores and platforms.

Third, Google updated Play Protect to automatically warn users and remove applications known to contain IPIDEA SDKs. This blocks the network’s ability to recruit new devices on certified Android devices.

Google claims these actions reduced IPIDEA’s available device pool by millions. Whether that number holds up over time is a different question, and we will get to that.

Not all residential proxy networks operate in gray zones. Decodo built theirs on user consent, ISO 27001 certification, and co-founded the Ethical Web Data Collection Initiative to prove the model works.

Try Decodo Now

The Anatomy of a Residential Proxy Network

To understand why this matters, we need to understand what a residential proxy network actually is and how it differs from datacenter proxies.

A datacenter proxy routes your traffic through IP addresses belonging to cloud providers or hosting companies. These IPs are easy to identify and block because they belong to known ASNs (Autonomous System Numbers) associated with commercial hosting.

A residential proxy routes traffic through IP addresses assigned by consumer ISPs to regular households. When you connect through a residential proxy, your request appears to come from someone’s home internet connection in Omaha, Tokyo, or Milan. This makes detection and blocking significantly harder because the traffic looks indistinguishable from a regular consumer browsing the web.

The challenge for proxy providers is obvious: they need access to millions of consumer devices to build a usable network. These devices need to be online, geographically distributed, and willing (or unwilling) to forward traffic.

There is an important nuance here. While Google’s report focuses on residential proxies, the same SDK installation on a mobile phone yields two distinct proxy classes. When the phone is connected to home WiFi, traffic exits through a residential IP assigned by the home ISP. When that same phone disconnects from WiFi and switches to 5G or LTE, traffic now exits through a mobile carrier IP. The device has not changed. The SDK has not changed. But the proxy class has shifted from residential to mobile.

This matters because mobile proxies are typically sold at a premium, sometimes 2-3x the price of residential proxies. Mobile carrier IPs are considered even harder to block than residential IPs because they are shared across thousands of legitimate mobile users through carrier-grade NAT. A single SDK deployment on mobile devices effectively generates inventory for two separate product lines.

Check the TWSC YouTube Channel

How Residential Proxy Providers Acquire IP Addresses

The IPIDEA takedown revealed the specific mechanisms used to build and maintain a large-scale residential proxy network. These methods fall into several categories, ranging from semi-legitimate to clearly deceptive.

SDK Integration

The primary method is embedding Software Development Kits into legitimate applications. IPIDEA operated multiple SDK brands: PacketSDK, HexSDK, CastarSDK, and EarnSDK. These SDKs are marketed to app developers as monetization tools. The pitch is simple: integrate our SDK, and we will pay you based on downloads or active users.

Once embedded, the SDK turns the device into an exit node for the proxy network. The device will accept incoming connections from the proxy infrastructure and forward requests to target websites. The app continues to function normally. The user has no obvious indication that their device is being used as a proxy.

Google’s analysis found that many applications containing these SDKs did not disclose this functionality to users. The SDK was hidden, not mentioned in the terms of service, and ran silently in the background.

Trojanized Applications

Beyond SDK integration, IPIDEA directly operated or controlled VPN applications that served as trojan horses. Galleon VPN, Radish VPN, and Aman VPN all provided genuine VPN functionality while simultaneously enrolling devices into the proxy network.

The logic is effective: users who install VPN applications expect their traffic to be routed through external servers. They are primed to accept unusual network behavior. The proxy functionality hides inside this expected behavior.

Google identified over 600 Android applications across multiple download sources with code connecting to IPIDEA’s C2 infrastructure. On Windows, they found 3,075 unique executables making DNS requests to IPIDEA’s Tier One domains, including applications masquerading as OneDriveSync and Windows Update.

Pre-Infected Devices

Researchers have documented cases of uncertified Android devices shipping with residential proxy payloads already installed. Set-top boxes, TV boxes, and other IoT devices from off-brand manufacturers have been found with hidden proxy software baked into the firmware.

This method bypasses the need for user installation entirely. The device arrives compromised.

The Technical Architecture

Google’s reverse engineering of the SDK code revealed a two-tier command-and-control system.

When an infected device starts up, it contacts a Tier One server. The device sends diagnostic information including OS version, device identifier, and a key parameter that appears to be used for affiliate tracking (determining which app developer gets paid for the enrollment). The Tier One server responds with timing configuration and a list of Tier Two server IP addresses.

The device then periodically polls a Tier Two server, checking for proxy tasks. When a task arrives, it contains a target FQDN (like www.google.com:443) and a connection ID. The device establishes a connection to the target, receives data payloads from the Tier Two server, and forwards them unmodified to the destination.

Google found approximately 7,400 Tier Two servers at the time of their analysis, hosted globally including in the United States. The number fluctuated daily, suggesting a demand-based scaling system.

The infrastructure analysis revealed something important: despite different brand names (PacketSDK, HexSDK, CastarSDK, EarnSDK), all the SDKs connected to the same pool of Tier Two servers. The brands were marketing fronts for a single unified network.

Need help with your scraping project?

The Brand Proliferation Problem

This brings us to one of the most interesting findings. IPIDEA did not operate under a single name. Google identified at least 13 ostensibly independent proxy and VPN brands controlled by the same actors:

• 360 Proxy

• 922 Proxy

• ABC Proxy

• Cherry Proxy

• Door VPN

• Galleon VPN

• IP 2 World

• Ipidea

• Luna Proxy

• PIA S5 Proxy

• PY Proxy

• Radish VPN

• Tab Proxy

These brands operated separate websites, separate marketing, and separate pricing. A customer comparing “922 Proxy” to “Luna Proxy” would have no obvious indication that they were buying access to the same underlying network.

This is not unique to IPIDEA. Industry analysis suggests there are only about 7 truly unique residential proxy networks globally, despite hundreds of brands competing in the market. The rest are resellers, white-labels, or, like IPIDEA, multiple storefronts for the same infrastructure.

Why Residential Proxies Are Hard to Block

The fundamental problem for defenders is that residential proxy traffic looks legitimate. When a request arrives from a Comcast IP in Chicago, there is no technical marker indicating whether it comes from an actual Comcast customer browsing normally or from a proxy network routing traffic through that customer’s compromised device.

The proxy networks exploit this by design. The value proposition they sell is precisely this difficulty of detection.

Security researcher Antoine Vastel published concrete data that illustrates the scale of this problem. By actively testing proxy endpoints, he verified more than 16 million unique IP addresses that were functional and associated with the IPIDEA network during the 30 days preceding the takedown. The breakdown by brand shows the relative sizes within the IPIDEA ecosystem: PY Proxy (PyProxy) accounted for 13.4 million IPs, PIA S5 Proxy for 2.2 million, and Luna Proxy for 549,000.

These are not theoretical numbers from marketing materials. These are IP addresses through which Vastel routed traffic and confirmed as working proxy endpoints. And here is the critical insight from his analysis: even with 16 million identified proxy IPs, defenders cannot simply block them.

The reason is that residential exit nodes mix traffic from automated tools and legitimate human users on the same IP. The device owner browses the web normally, while the SDK, in the background, forwards proxy traffic over the same connection. Blocking these IPs based on proxy activity would inevitably block real users who happen to share an IP or whose IP was previously used as an exit node.

Vastel’s recommendation is telling: use these IoCs for risk scoring, behavioral enrichment, and incident investigation, but not for direct blocking. The data is context, not a verdict. This fundamental asymmetry makes residential proxies valuable to attackers and frustrating for defenders.

His research also confirmed another pattern: IP addresses frequently appear across multiple proxy ecosystems simultaneously. IPIDEA did not rely exclusively on its own residential pool. Requests were routed through or resold from other networks. The same IP might be accessible through IPIDEA, a competitor, and a reseller all at once. This interconnection means that even identifying an IP as “IPIDEA-linked” does not tell you the full story of how it is being used.

Traditional IP reputation systems struggle with this complexity. Blocking known bad IPs works for datacenters where the IP assignments are stable, and the ASNs are identifiable. Residential IP addresses rotate frequently as ISPs reassign addresses, and blocking residential IP ranges blocks legitimate users.

Google’s Approach: Attacking the Infrastructure

Rather than trying to block individual IP addresses, Google attacked the control infrastructure. By taking down the Tier One C2 domains, they severed the connection between infected devices and the proxy operators. Without C2 connectivity, the SDK code on millions of devices becomes inert.

This approach has precedent. It is the same strategy used against botnets: identify the command-and-control infrastructure and take it down. The infected devices remain infected, but they can no longer receive instructions.

Google also partnered with Cloudflare to disrupt IPIDEA’s domain resolution, adding another layer of infrastructure disruption beyond the legal domain seizures.

Will It Work? The Persistence Problem

Here is where we need to be realistic about the limitations of this approach.

The takedown disrupted IPIDEA’s current infrastructure. The domains are gone. The C2 servers are unreachable. Millions of devices are no longer participating in the proxy network. But “no longer participating” is not the same as “cleaned up.”

The fundamental problem is that infected devices remain infected. The SDK code is still installed on millions of phones, tablets, TV boxes, and computers worldwide. Google can take down domains. Google can update Play Protect to block new installations. What Google cannot do is reach into millions of devices and uninstall the malicious code that is already there.

For the SDK to be removed, one of these things needs to happen: the user manually uninstalls the app containing it, the device gets factory reset, or the device gets replaced. None of these happens at scale. Most users are unaware that the SDK exists. The apps that contain it often provide real functionality, games, utilities, and VPNs that users want to keep. There is no mechanism to notify millions of people across dozens of countries that their flashlight app is secretly a proxy node.

This creates an asymmetry that favors the attackers. Google invested significant legal, technical, and coordination resources to take down IPIDEA’s infrastructure. The IPIDEA operators (or anyone who acquires their codebase) can spin up new C2 domains, update their DNS configuration, and potentially reactivate a substantial portion of the dormant network. The SDK code often includes fallback mechanisms and update capabilities precisely for this scenario.

The brand proliferation we discussed earlier is part of this resilience. IPIDEA operated 13+ brands. If some domains get seized, others may survive. If the entire IPIDEA operation is compromised, the operators can rebrand entirely and inherit a pre-installed base of millions of devices waiting for new instructions.

Google acknowledged this reality in its announcement, noting that “this industry appears to be rapidly expanding” and that “there are significant overlaps across providers.” The reseller and partnership agreements that connect different proxy brands mean that disruption propagates unpredictably through the ecosystem, but so does recovery.

This is not a battle that can be won definitively. It is a cost-imposition strategy. The goal is to make operating these networks so expensive and risky that some operators exit the market or shift to more legitimate practices. But as long as the infected device base exists, the infrastructure can be rebuilt. The realistic outcome is degradation, not eradication.

The Economics Behind Residential Proxy Networks

Understanding why these networks exist requires understanding the economics.

Residential proxy bandwidth sells for $4-8 per gigabyte or more to end customers. The cost to acquire that bandwidth through SDK partnerships is measured in cents per gigabyte. The gross margins appear enormous.

But running a proxy operation is not just bandwidth arbitrage. The real costs include:

Engineering and Infrastructure: Maintaining thousands of C2 servers globally, building rotation logic, handling the unreliable nature of consumer devices going online and offline unpredictably.

SDK Distribution: Paying app developers for integration, maintaining relationships with publishers, and navigating app store policies that increasingly scrutinize monetization SDKs. The silver lining is that mobile app SDKs generate dual inventory: residential IPs when devices are on WiFi, mobile IPs when they switch to cellular. This allows providers to sell the same underlying device pool across two product categories at different price points.

Customer Acquisition: Finding buyers for proxy services is expensive. The market is niche, competition is intense, and customers are price-sensitive.

Legal and Compliance: Or in IPIDEA’s case, the lack thereof. Operating in legal gray zones creates ongoing risk. The Google takedown demonstrates what happens when that risk materializes.

Industry estimates suggest customer acquisition costs consume 40-60% of revenue even at scale. The apparent margin compression means that most residential proxy providers operate on thin actual profits despite the high sticker prices.

This economic pressure explains the proliferation of brands. Running multiple storefronts for the same underlying network lets operators segment the market, test different price points, and spread legal risk across multiple corporate entities.

The Bigger Picture

The IPIDEA takedown is part of a larger pattern. Google previously took action against the BadBox2.0 botnet, which shared infrastructure with IPIDEA. Law enforcement agencies worldwide are paying greater attention to residential proxy networks, recognizing the role this infrastructure plays in facilitating activities ranging from credential stuffing to espionage.

The residential proxy industry has partially operated in a legal gray zone for years. The Google action, particularly the legal component, establishes a clearer precedent that enrolling devices without consent and facilitating malicious activity creates meaningful legal exposure.

This does not mean residential proxies are going away. The demand exists, and where demand exists, supply follows. However, the industry may be compelled toward more transparent practices: clearer consent mechanisms, improved disclosure in applications that embed SDKs, and more careful vetting of customers who purchase proxy access.

For those of us who work with web scraping, the takedown is a reminder that the infrastructure we rely on has a supply chain. Understanding the supply chain, including its technical architecture, its business model, and its vulnerabilities, helps us make better decisions about which providers to trust and how to build resilient scraping systems.

The IPIDEA network may be rebuilt, rebranded, or replaced by competitors. However, the detailed technical analysis Google published provides the entire industry with greater visibility into how these networks operate. That visibility, more than the takedown itself, may be the most lasting impact.

Of course, I could not miss running it, and in the past ten days, I’ve played with OpenClaw. What follows is what I learned, what surprised me, and why I think it can be interesting also for someone doing web scraping, even if it’s not its core use.

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

What OpenClaw actually is

OpenClaw is an AI assistant that runs locally on your machine. It can control your computer: read and write files, execute shell commands, manage your calendar, send emails, and browse the web. You interact with it through messaging apps you already use. Telegram, WhatsApp, Discord, Signal, Slack, iMessage. You message it like you would message a coworker, and it does things on your behalf.

The key difference from cloud-based assistants is that everything runs on your infrastructure. The agent lives on your computer, sees what you allow it to see, and acts within the boundaries you define. It supports multiple AI backends: Anthropic Claude, OpenAI models, or local models if you prefer to keep everything offline.

The tagline on the repository says it well: “A smart model with eyes and hands at a desk with keyboard and mouse.”

What makes OpenClaw more than a simple automation wrapper is the breadth of its integration layer. Out of the box, it connects to over 50 services and tools. Gmail for email, Obsidian for notes, GitHub for repositories, Spotify for music, Philips Hue for smart home control. Each integration is a capability the agent can invoke when relevant. You do not need to specify which tool to use. You describe what you want, and the agent figures out which integration applies.

The architecture includes a skill system that deserves attention. Skills are modular capabilities that the agent can learn, create, and modify. Users have reported that OpenClaw has written its own extensions and updated its own prompts autonomously. This is not science fiction. The agent has file system access and can modify its own configuration. Whether this is exciting or terrifying depends on your perspective.

There is also a memory layer. OpenClaw remembers context across sessions. It learns your preferences, your common requests, and your workflow patterns. Over time, it becomes more useful because it accumulates knowledge about how you work. This persistence is stored locally, which matters for privacy, but it also means the agent builds an increasingly detailed model of your behavior.

Background execution is another differentiator. OpenClaw can run cron jobs, scheduled reminders, and background tasks. You can tell it to check something every hour or to notify you when a condition is met. It is not just reactive. It can be proactive, monitoring and acting without constant prompting.

Finally, it works in group chats. You can add your OpenClaw bot to a Telegram group, and it will participate in conversations, responding when mentioned or when configured to do so. This opens possibilities for shared assistants across teams, though it also multiplies the security considerations.

If you don’t use LLMs for scraping, you need IPs with good reputation. For this reason, we’re using a proxy provider like our partner Ping Proxies, that’s sharing with TWSC readers this offer.

💰 - Use TWSC for 15% OFF | $1.75/GB Residential Bandwidth | ISP Proxies in 15+ Countries

Setting it up

Installation is straightforward. A one-liner gets you started:

curl -fsSL https://openclaw.ai/install.sh | bash

Or via npm:

npm i -g openclaw
openclaw onboard

I went with the npm route. The onboarding process walks you through connecting your AI provider and setting up your first communication channel.

For Telegram, you create a bot through BotFather, grab the token, and configure it in OpenClaw. The CLI guides you through the pairing process. When someone messages your bot for the first time, you approve them with a simple command:

openclaw pairing approve telegram <CODE>

After that, you can message your bot, and it responds as your personal assistant.

The browser extension required a few extra steps. The official documentation says it works with Chrome only, but since Brave is Chromium-based, I decided to try it anyway. It worked. You install the extension with:

openclaw browser extension install

Then load it as an unpacked extension in Brave by enabling Developer Mode and pointing to the path returned by `openclaw browser extension path`. The extension lets you attach specific tabs to OpenClaw’s control. Only attached tabs can be controlled, which is a reasonable security measure.

Check the TWSC YouTube Channel

Living with it

I configured OpenClaw to use Claude as its backend. The experience has been surprisingly natural. I message it on Telegram with requests, and it executes them. Check my calendar for conflicts. Find a file I worked on last week. Send a reminder at 3 pm.

What makes it different from simply asking ChatGPT is that it actually does things. It does not tell me how to check my calendar. It checks my calendar and tells me what it found.

I also connected it to my TWSC accounting system, which has some APIs. In a few steps, OpenClaw built a small app that lets me use Telegram to check invoice status, revenues, expenses, and more.

The browser extension adds another dimension. I can ask it to navigate to a website, read specific content, fill out forms, or extract information. It operates within a real browser session, with real cookies, logged into my real accounts if I choose to attach those tabs.
This is what I used to automate posting a note on Substack, which doesn’t have an API. I just prompted the desired message, and the LLM (Claude 4.5) understood where on the Substack website notes are posted and how to create one.

The elephant in the room

I need to address something that should be obvious by now. Using OpenClaw means giving an AI agent extensive access to your computer. It can see your browser sessions, your files, your credentials. It can execute commands.

Consider what this means in practice. When you attach a browser tab, the agent can read every cookie, every session token, every piece of data on that page. If you attach a tab where you are logged into your email, the agent can read your email. If you attach a tab with your banking session, the agent theoretically has access to that session. The same applies to files. If you give OpenClaw access to your file system, it can read your SSH keys, your environment files with API credentials, your password manager exports if you have any lying around.

The agent can also execute shell commands. This means it can install software, modify system configuration, create network connections, and run arbitrary code. OpenClaw does have a sandbox mode that restricts some of these capabilities, but the default configuration is permissive because that is what makes it useful.

With my current setup using Claude as the backend, every interaction passes through Anthropic’s servers. The model sees what I ask, sees the context from my computer, and processes it on their infrastructure. When I ask OpenClaw to read a balance sheet, the request goes to Anthropic’s API, is processed, and returns. I trust Anthropic’s privacy practices, but this is still a significant amount of sensitive data leaving my machine.

There is also the question of prompt injection and model behavior. What happens if you navigate to a malicious page that contains instructions designed to manipulate the agent? Modern LLMs are susceptible to prompt injection attacks where content on a webpage could potentially influence the agent’s behavior. This is not theoretical. It is an active area of security research, and there are no perfect defenses yet.

The risk profile changes depending on your configuration. Using a cloud model like Claude or GPT-4 means your data flows through external servers, but those models are also more capable and more likely to handle edge cases correctly. Using a local model keeps everything on your machine, but local models may make mistakes that a more capable model would avoid.

This is why my next step is to migrate to a local model. OpenClaw supports running with local LLMs, which means the entire pipeline can stay on my hardware. The tradeoff is capability. Local models are not yet at the level of Claude or GPT-4 for complex reasoning tasks. But for an assistant that executes relatively simple commands, they might be good enough. And critically, my cookies, my session tokens, my file contents never leave my network.

Practical recommendations if you decide to use OpenClaw: use a dedicated browser profile for attached tabs, separate from your personal browsing. Do not attach tabs with active banking or financial sessions. Be cautious about which directories you give the agent access to. Consider running in sandbox mode until you understand the tool’s behavior. And seriously consider the local model option if you plan to use this for anything sensitive.

If you are going to run an AI agent with this level of access, running it locally is the only configuration that makes full sense from a security perspective. The convenience of cloud models is real, but so is the exposure.

Need help with your scraping project?

Why this matters for web scraping

Read more

It’s not magic, but the result of a few very specific mechanisms—namely, browser automation protocols. The most important ones are:

• WebDriver

• Chrome DevTools Protocol (CDP)

• WebDriver BiDi

In this post, I’ll break down each of them to cover what they are, how they work at a low level, and how they enable programmatic browser control. You’ll also discover where browser automation is headed next!

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

---

Everything You Need to Know About WebDriver

Let me start this WebDriver vs Chrome DevTools Protocol (CDP) vs WebDriver BiDi piece by focusing first on the protocol at the top of the list: WebDriver!

What It Is

WebDriver is a W3C Recommendation that standardizes a remote control interface for automating and inspecting “user agents” (in other words, web browsers). In practical terms, it lets external programs interact with a browser through a language- and platform-agnostic protocol.

In detail, WebDriver exposes a concise, object-oriented API for cross-browser control. This makes it a reliable and realistic foundation for end-to-end testing and automation, whether the browser runs locally or on a remote machine.

📖 Further reading:

• WebDriver W3C Recommendation

• WebDriver Standard GitHub repository

• WebDriver documentation on MDN

How It Works

The W3C WebDriver protocol isn’t tied to any specific programming language or framework. Because of that, browser automation client libraries built on top of it—regardless of the language they’re written in—are essentially thin wrappers. Their main job is to translate application-level API calls into WebDriver-compliant commands and then deal with the results produced by the browser.

More in depth, there are 3 main components you need to keep in mind:

• The WebDriver-based browser automation client library (e.g., Selenium) that exposes a developer-friendly, application-level API.

• A browser-specific driver server (e.g., ChromeDriver, geckodriver). That’s usually a standalone executable that runs on your machine or in a remote environment. It understands the WebDriver protocol and maps incoming commands to the browser’s native automation interfaces. Depending on the automation tool you’re using, this driver may be downloaded and managed automatically, or you may have to install and version-match it yourself.

• The browser application itself (Chrome, Firefox, Edge, Safari, etc.), which ultimately performs the actions.

At runtime, every WebDriver interaction follows a strict client–server communication model defined by the W3C specification. Thus, when a script built with a browser automation client issues a command (e.g., element.click()), the following happens under the hood:

1. The client serializes the command into a standardized request (targeting a specific endpoint and including a well-defined JSON payload) as defined by the W3C WebDriver specification.

2. That request is sent over HTTP to a browser-specific WebDriver server endpoint exposed by the driver.

3. The driver server receives the request, interprets the WebDriver protocol command, and maps it to the appropriate native browser automation call.

4. The browser executes the action as a real user would.

5. The browser returns the execution result (e.g., status, errors, or requested data) back to the driver using browser-internal communication mechanisms.

6. The driver wraps that result into a WebDriver-compliant response, including a standardized status code and JSON payload.

7. The response is sent back over HTTP to the client, which deserializes it into application-level objects.

One of the biggest advantages of modern WebDriver servers speaking the W3C WebDriver protocol directly is predictability. Because the communication language is standardized, different browser drivers implement the same semantics, resulting in more consistent behavior across browsers and environments.

This architecture also explains why WebDriver requires a browser-specific driver server in the first place. The protocol itself is browser-agnostic, while the driver is responsible for bridging that standardized protocol to each browser’s internal automation APIs.

Scraping and Automation Libraries Built on Top of It

Out of the libraries built on top of the WebDriver mechanism, Selenium is by far the most popular and widely adopted. Other interesting libraries include WebDriverIO and Nightwatch.js.

The main challenge with this type of library is configuring the correct browser driver server. Luckily, as of Selenium 4.10 (release in June 2023), you no longer need to manually download the correct browser driver for your specific browser type and version. Selenium now automatically detects, downloads, and configures the appropriate driver, ensuring that your tests or scraping scripts run smoothly out of the box.

Extra: What About Selenium 3’s JSON Wire Protocol? A Bit of History…

Before the W3C standardized the WebDriver protocol in 2018, older versions of Selenium (especially before 3.8) relied on a non-standard JSON Wire Protocol.

In this architecture, the client library serialized commands into JSON, but there was no unified specification. Thus, each browser driver (developed by different teams) had to implement its own logic to map those instructions to the browser’s native automation APIs.

That created a kind of “dialect problem,” where the same command could behave slightly differently or have different timing across Chrome, Firefox, Internet Explorer, and Safari. These inconsistencies were a major source of latency and flaky behavior.

Selenium 4 resolved that by adopting the W3C WebDriver protocol as its standard, eliminating the intermediary translation layer and ensuring consistent, predictable automation across browsers.

Chrome DevTools Protocol (CDP) Explained

Time to dive into the second protocol under analysis: the Chrome DevTools Protocol (also known simply as CDP).

What It Is

The Chrome DevTools Protocol (CDP) is a low-level, JSON-based protocol that lets you inspect, debug, and instrument web pages in Chromium-based browsers, such as Chrome, Edge, Brave, and Opera.

It provides programmatic access to browser internals, enabling control over the DOM, network requests, and performance metrics. The protocol is designed by Google and is commonly used for automation, testing, and scraping.

📖 Further reading:

• Chrome DevTools Protocol Docs

• Chrome DevTools Protocol GitHub repository

How It Works

CDP is organized into domains. Domains represent functional areas in the browser and handle specific tasks, such as DOM manipulation, network interception, console logging, performance profiling, and device or network emulation.

Each domain exposes commands and events:

• Commands are JSON requests sent to the browser.

• Events are JSON messages sent by the browser back to the client.

Both commands and events can be transmitted over HTTP or WebSocket, with WebSocket being the preferred approach due to its support for quick, bidirectional communication.

When it comes to browser control via CDP, there are only 2 elements at play:

1. The CDP-based browser automation client: A library (e.g., Playwright) that communicates with the browser using JSON commands over the Chrome DevTools Protocol.

2. A Chromium-based browser: Expose a CDP endpoint (by default on ws://localhost:9222 for local instances). For remote browsers, libraries like Playwright can control them using a CDP URL, which typically starts with wss:// (WebSocket over TLS).

Now, here’s what happens under the hood when you call a high-level API like page.screenshot() in a CDP-based library:

1. The client library establishes a WebSocket session with the browser’s CDP endpoint. This creates a bidirectional communication channel between the automation library and the browser.

2. The client sends a JSON command targeting a specific domain and method, with the required parameters (e.g., {”cmd”:”Page.captureScreenshot”,”args”:{”format”:”jpeg”}}).

3. The browser receives the JSON request and maps it to the corresponding native browser operation, such as capturing a screenshot.

4. The browser executes the action as if a real user or system process had triggered it.

5. The browser sends a JSON response back to the client over the WebSocket, containing execution results, metrics, or errors.

6. The browser automation client framework parses the response and converts it into usable objects or data structures for scripts or tests.

Note: Asynchronous events, like network requests or DOM mutations, are sent by the browser over the same WebSocket channel and can be subscribed to.

Unlike the W3C WebDriver protocol, which standardizes client-server browser automation across all major browsers, the Chrome DevTools Protocol (as the name suggests) is Chromium-specific.

Scraping and Automation Libraries Built on Top of It

Many libraries and frameworks leverage the Chrome DevTools Protocol to provide higher-level browser automation capabilities for testing, monitoring, or scraping, such as:

Note: Until full support for the WebDriver BiDi protocol is fully implemented, Selenium 4 plans to also give access to CDP features where applicable, in addition to the standard W3C WebDriver protocol (of course).

Learn more about Pydoll in my previous article for The Web Scraping Club!

Extra: What About the Firefox Remote Debug Protocol?

CDP works specifically with Chromium-based browsers, but how can Playwright (and other automation libraries) control Firefox? The answer is the Firefox Remote Debug Protocol.

Similar to CDP, the Mozilla protocol allows a debugger or automation client to connect to Gecko-based browsers. In particular, it provides a unified view of the DOM, CSS rules, and other client-side web technologies.

At one point, Firefox even offered limited CDP support, but this was deprecated in Firefox Nightly 141 to jump on the WebDriver BiDi train.

(And for Safari? The CDP-equivalent protocol is the WebKit Debug Protocol!)

Understanding WebDriver BiDi: The What, Why, How, and When

WebDriver relies on a strict request-response model over HTTP. Commands are synchronous and unidirectional: the client sends a request to the browser, waits for a response, and then sends the next command.

That approach works well for standard UI testing, ensuring actions like clicks, typing, and navigation occur in the correct order. Still, it limits asynchronous interactions (such as monitoring network requests, console logs, or DOM changes) because the client must continuously poll the browser for updates.

To overcome those limitations, the Selenium team, together with major browser vendors, is developing the WebDriver BiDi (Bidirectional) Protocol. Currently (as of this writing) a W3C Working Draft (WDC), WebDriver BiDi is designed to provide real-time, cross-browser automation.

WebDriver BiDi introduces bidirectional communication via WebSockets, allowing the browser to push events directly to the client as they occur. This enables streaming of logs, network activity, JavaScript exceptions, and other runtime events without the overhead of repeated HTTP requests, resulting in faster, more responsive, and richer automation.

Basically, BiDi combines the strengths of traditional WebDriver and the Chrome DevTools Protocol (CDP). While CDP offers low-level control over Chromium-based browsers, it isn’t standardized across browsers. BiDi fills that gap by providing true cross-browser support, while also opening the door to features that were previously limited to Chromium—such as network interception, performance monitoring, and console logging—in a consistent way across Firefox and Safari.

📖 Further reading:

• WebDriver BiDi W3C Working Draft

• WebDriver BiDi GitHub page

• Selenium’s BiDirectional functionality documentation page

Check the TWSC YouTube Channel

WebDriver vs CDP vs BiDi: Final Comparison

Compare the most important protocols for browser control in the summary table below:

WebDriver (Classic)

👍 Pros:

• W3C standard

• Stable and predictable model

• Cross-browser consistency

👎 Cons:

• Unidirectional and synchronous, with no real-time event streaming

• Limited low-level browser access

• Polling-based architecture, which can be slower for advanced use cases

• Requires external browser-specific driver binaries (e.g., ChromeDriver, GeckoDriver) that must be downloaded, managed, and kept in sync with browser versions

CDP / Firefox Remote Debug Protocol / WebKit Debug Protocol

👍 Pros:

• Fully bidirectional, with real-time event streaming via WebSockets

• Deep, low-level browser control (network interception, performance, DOM internals, etc.)

• Excellent for debugging, scraping, and advanced automation

👎 Cons:

• Not standardized, relying on engine-specific APIs and semantics

• Not cross-browser, as each protocol targets a single engine (Chromium-only, Gecko-only, or WebKit-only)

• API instability, where protocol changes may require frequent updates to client libraries

WebDriver BiDi

👍 Pros:

• W3C-backed, cross-browser standard for modern browser automation

• Bidirectional communication with real-time events (network activity, logs, JavaScript errors, etc.)

• Combines WebDriver’s stability with CDP-like advanced capabilities

👎 Cons:

• Still an evolving draft

• Ecosystem adoption is ongoing

• Requires browser driver server management

Need help with your scraping project?

Final Comment: Is WebDriver BiDi the Future?

Yes, it appears so! But some explanation is needed…

Note: Since the WebDriver BiDi is the next-generation iteration of the WebDriver protocol, the request-response model used by Selenium is now referred to as “WebDriver Classic.”

The Selenium team is actively transitioning from WebDriver Classic to WebDriver BiDi, while also gradually replacing CDP support for cross-browser automation, all while maintaining backward compatibility with existing tests. Similarly, Cypress has already adopted BiDi for Firefox automation. In Puppeteer, when launching Firefox, WebDriver BiDi is enabled by default. Other major browser automation tools like Playwright are also exploring BiDi support.

So, if WebDriver Classic will eventually be replaced by BiDi, what about CDP?

WebDriver BiDi doesn’t aim to replace CDP!

Chrome DevTools Protocol remains optimized for low-level, Chromium-specific debugging and browser control. In contrast, BiDi is a modern, cross-browser standard focused on test automation. That’s why Puppeteer still uses CDP when launching Chrome (as CDP features aren’t yet fully supported by BiDi).

BiDi’s goal is to standardize automation across browsers, not to replace engine-specific debugging tools like CDP, Firefox Remote Debug Protocol, or WebKit Web Inspector.

For the foreseeable future, WebDriver Classic will gradually be phased out. Chromium browsers will continue to support CDP for low-level debugging, while BiDi complements it by providing standardized, real-time, cross-browser automation and testing capabilities.

Conclusion

In this post, I’ve outlined the three main protocols used by browser automation libraries for testing and web scraping to programmatically control browser instances.

As you’ve seen, Playwright and Selenium follow different approaches, relying on distinct sets of protocols. However, WebDriver and CDP are ultimately complementary, each serving its own purpose in the automation ecosystem.

Feel free to share your thoughts or questions in the comments. Until next time!

For this reason, we tested five open-source scraping tools on 1000 Nike product URLs to measure success rate, speed, and reliability. The results challenge a common assumption in the scraping community: that modern e-commerce sites require browser automation to reliably extract data.

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

Nike.com system model

Before testing, we need to understand what Nike.com actually serves and where potential blocking might occur.

Nike.com is protected by both Akamai Bot Manager and Kasada, but the two systems guard different parts of the site. Akamai handles the public-facing catalog, including product pages and search results. Kasada protects authenticated flows, such as login and checkout. This layered approach makes sense from Nike’s perspective: catalog data is semi-public anyway (they want customers to browse), while account actions carry real business risk.

We focus exclusively on public catalog data in this article. Scraping behind authentication raises legal and ethical concerns we prefer not to encourage. For the product catalog, we only need to bypass Akamai. We found no trace of Kasada challenges or fingerprinting scripts on product pages during our tests.

Nike product pages are server-side rendered. When you request a product URL, the server returns complete HTML with product data embedded in the DOM. This means that, unlike many modern e-commerce sites, which use client-side rendering, where the initial HTML is a shell and product data loads via JavaScript API calls. This is relevant for us, since there’s a practical consequence for scrapers: JavaScript execution is not required to extract product information because the data is already in the first response.

So we can use simple HTTP requests, but with some tweaks: modern WAFs inspect TLS handshake characteristics (cipher suites, extensions, ordering) to identify non-browser clients. This is where tools like Python’s requests library fail immediately: its TLS fingerprint looks nothing like Chrome or Firefox.

HTTP/2 fingerprinting adds another layer: header order, pseudo-header placement, and SETTINGS frames can reveal automation tools. Even if your TLS handshake passes, sending headers in the wrong order or with unusual HTTP/2 settings can trigger detection.

First of all, you need IPs with good reputations for scraping. For this reason, we’re using a proxy provider like our partner Ping Proxies, that’s sharing with TWSC readers this offer.

💰 - Use TWSC for 15% OFF | $1.75/GB Residential Bandwidth | ISP Proxies in 15+ Countries

On top of that, IP reputation plays a role. Datacenter IPs receive more scrutiny than residential ones because most legitimate users browse from home or mobile networks.

For this test, we focused on TLS and HTTP/2 fingerprinting. We scraped from a residential IP, which neutralized IP reputation as a variable. We did not interact with the page, so behavioral signals were not applicable for HTTP clients. We observed no JavaScript challenges on Nike product pages during testing. This last point is crucial: Nike could deploy Akamai’s JavaScript challenge on product pages, but they have chosen not to. Whether this is a deliberate trade-off (challenges slow down real users) or an oversight, we cannot say. But it opens the door to HTTP-based scraping.

Check the TWSC YouTube Channel

The Tool Landscape

The five tools we tested fall into two categories: browser automation and HTTP clients with fingerprint emulation.

On the browser side, Pydoll is an async Python library built on Chrome DevTools Protocol. It controls Chromium without WebDriver, avoiding the navigator.webdriver flag.

Camoufox takes a different approach: it is a custom Firefox build that spoofs fingerprints (WebGL, canvas, audio, navigator) and patches headless detection vectors.

Scrapling sits somewhere in between, offering multiple fetcher types from simple HTTP to full browser automation via Playwright. Its StealthyFetcher wraps Chromium with anti-detection features while the basic Fetcher uses just requests with TLS impersonation.

On the HTTP client side, Rnet is a Rust-based Python client that emulates browser TLS and HTTP/2 fingerprints (JA3, JA4, Akamai). It supports Chrome, Firefox, Safari, Edge, and OkHttp profiles.

Undetected-httpx is built on `curl_cffi` and provides browser-identical TLS fingerprints without running a browser.

The fundamental difference: browser tools execute JavaScript and render pages, while HTTP clients make requests with browser-like signatures but cannot handle JS-dependent content. This distinction matters because it determines both what you can scrape and how fast you can do it. A browser spins up an entire rendering engine, consumes hundreds of megabytes of RAM per instance, and waits for network events, DOM parsing, and JavaScript execution. An HTTP client sends a request and receives bytes. The performance gap is enormous when the extra capability is not needed.

In case you’re still struggling with browser automation you can try out rayobrowse - a self-hosted Chromium stealth browser from Rayobyte. Have a look at it here.

💰 - Rayobyte is offering an exclusive 55% discount with the code WSC55 in all of their static datacenter & ISP proxies, only to web scraping club visitors.
You can also claim a 30% discount on residential proxies by emailing sales@rayobyte.com.

Test Setup

We extracted 1000 product URLs from Nike’s sitemap (Austria EN locale). The sitemap is publicly accessible and provides a clean list of product URLs without requiring crawling. Each tool scraped the same URL set sequentially, with no delays between requests. This aggressive pacing represents a worst-case scenario for detection: real scrapers would typically add delays to reduce load and avoid rate limits.

Extraction logic: All tools used identical HTML parsing. We targeted stable data-testid attributes, which Nike uses for internal testing. These selectors are more reliable than class names, which often change with CSS updates:

# Title
soup.select_one('h1[data-testid="product_title"]')

# Price
soup.select_one('[data-testid="currentPrice-container"]')

# Color
soup.select_one('[data-testid="product-description-color-description"]')

# Style code (SKU)
soup.select_one('[data-testid="product-description-style-color"]')

This approach keeps the comparison fair: differences in results reflect fetching capability, not parsing logic.

Need help with your scraping project?

Tool configurations:

- Pydoll: Headless Chromium, 3-second wait after page load, new browser instance per URL

- Camoufox: Headless Firefox, networkidle wait, single browser session with new pages

- Scrapling: Fetcher.get() with impersonate=’chrome’ and stealthy_headers=True

- Rnet: BlockingClient with Impersonate.Chrome137, 30-second timeout

- Undetected-httpx: httpx.Client with browser-like headers, 30-second timeout

The browser tools opened new page contexts for each URL. The HTTP clients reused the same session.

As always, the full code can be found on The Lab GitHub private repository, inside the folder 96.NIKE, available only for paid subscriber of TWSC.

Read more

The document, titled “The age of fast-forward web data”, identifies six trends reshaping the industry in 2026. We went through the report and pulled out what matters for anyone working in data extraction.

The Market Has Exploded

The report opens with a number: the web scraping market reached $1.03 billion in 2025, with projections pointing to $2 billion by 2030 (some estimates double that figure). The majority of mid-to-large enterprises now use web scraping for competitive intelligence, and most e-commerce companies monitor competitor prices using scraped data.

Web scraping, in other words, is no longer a fringe practice. It has become critical economic infrastructure.

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

Trend 1: Full-Stack APIs Replace Separate Components

The first trend concerns the end of standalone proxies. Zyte reports that the market now counts over 250 proxy vendors, with price wars that have eroded margins and turned proxies into commodities. The problem is that websites have evolved their defenses well beyond simple IP blocking: TLS fingerprinting, behavioral analysis, canvas fingerprinting, and JavaScript traps. Some systems claim 99.9% accuracy in distinguishing humans from bots through behavioral biometrics alone.

The market response, according to Zyte, is migration toward APIs that handle the entire stack transparently: proxy rotation, browser automation, unblocking, parsing, and retry logic. The cited figure: request volume through the Zyte API grew 130% year-over-year in 2025.

The trend is real, though it needs context. For those operating at a large scale with specific control requirements, direct component management remains relevant. What we find more interesting is the underlying shift: defense complexity has crossed the threshold of manual manageability for most use cases. Now more than before, it’s a buy vs make choice: you can always try (and probably should, at least to improve your skills in web scraping) to create your in-house solutions but the game is becoming so hard that the market is looking for all-in-one APIs.

Check the TWSC YouTube Channel

Trend 2: AI Enters the Web Scraping Toolchain

The second trend describes AI integration across every link in the chain. The report cites a Technavio projection: the AI-based web scraping market will reach $3.16 billion by 2029, growing at 39.4% annually.

The concrete applications listed in the report cover the entire cycle: auto-classification of content for schema-specific extraction, LLM-powered extraction for unstructured data, automatic identification of selectors and field mappings, change detection, crawler code generation, browser interaction via natural language, data cleaning, anomaly detection, and real-time unblocking strategies.

The key distinction Zyte proposes: LLM extraction for low-volume projects with volatile sites (higher cost per request, but flexibility compensates), code generation for high-volume mission-critical projects (generated code can be tested, versioned, and costs less at scale).

The report also mentions computer-use models for multi-step navigation (forms, filters, gated screens). We think this is a rapidly evolving area worth watching closely, and we have talked about this trend in several other articles on these pages. LLMs are not a silver bullet for HTML parsing, but their use certainly improves the productivity of data acquisition teams, both when they need to write scrapers and when they need to check data quality.

Need help with your scraping project?

Trend 3: The Era of Autonomous Pipelines

The third trend is the most ambitious: end-to-end automation through agents. Zyte cites a Deloitte study showing 30% of organizations exploring agentic approaches, 38% piloting them, but only 11% with production deployments. The gap, according to Zyte, will narrow in 2026.

The proposed vision: a team specifies an outcome (dataset with schema, coverage targets, freshness, failure tolerance), an agent explores the site, discovers necessary actions, chooses the most efficient method, and when the site changes, the agent diagnoses the breakage, regenerates code, re-validates outputs, and escalates only when confidence drops below a threshold.

In practice, the report describes a multi-agent system: API discovery agents, schema-first extraction agents, self-healing testing agents, vision-based computer-use agents, DOM-native browser agents, and coding agents. Each agent handles one specific job, and an orchestrator supervises.

The vision is compelling on paper, but the report itself admits that production adoption is still limited. Zyte’s practical advice is telling: apply agents selectively. For stable, straightforward sources, a conventional setup remains more cost-effective and we could not agree more.

Trend 4: The Arms Race Accelerates

The fourth trend is perhaps the most concrete. Anti-bot systems now reconfigure continuously, driven by ML models that adapt in minutes. The report cites Proxyway: “Two days of unblocking efforts used to give two weeks of access... now it’s the other way around.”

Zyte reports observing a major bot management vendor deploy over 25 version changes in 10 months, often releasing updates multiple times per week. Cloudflare, according to the report, has a near-real-time system that adapts its detection strategy every few minutes.

The factors amplifying the mismatch: ML-driven detection with polymorphic JavaScript, WASM obfuscation, RASP, passive fingerprinting at scale; detection mechanisms monitoring timing patterns, network-level anomalies, device fingerprint consistency, pointer curves, scroll variance; growing AI bot traffic volume pushing sites to respond with automatic tuning.

Zyte’s conclusion: manual access strategies are no longer sustainable at scale. Only automated, self-adjusting pipelines survive. We can also say that web scraping is becoming more expensive, and there should be a smarter way to do it. Our idea at Databoutique.com is to share scraping costs across multiple data buyers, and this can be a way to do so.

Trend 5: The Web Fragments Into Access Lanes

The fifth trend describes a trifurcation of the web from a bot access perspective.

The hostile web: sites deploying aggressive honeypot traps, AI-targeted challenge flows, sophisticated fingerprinting. Cloudflare has deployed AI Labyrinth, traps specifically designed for AI crawlers, claiming to have blocked 416 billion AI bot requests in six months.

The negotiated web: publishers adopting licensing, attestation, pay-per-crawl, paywalls. Standards like ai.txt, llms.txt, Really Simple Licensing (RSL) attempt to make permissions machine-readable. Adweek reports that 2026 will see LLM deals shift from one-time payments to usage-based revenue shares.

The invited web: sites exposing machine-first interfaces for approved actors. Shopify, Google, Visa, Stripe, OpenAI either support Model Context Protocol (MCP) or have launched proprietary protocols like Agentic Commerce Protocol (ACP), Universal Commerce Protocol, Trusted Agent Protocols.

The key point: identity becomes a first-class citizen. Initiatives like “Know Your Agent” are gaining traction. Verified or attested bots receive preferential routing, unverifiable bots face increasing friction.

This chapter describes attempts to make the web sustainable for content publishers via protocols that could reward them, but it’s still a long way to go

Trend 6: Regulatory Compliance Arrives

The sixth trend concerns the evolving legal landscape. Two relevant dates: California AB 2013 took effect January 1, 2026; the EU AI Act takes effect August 2, 2026.

California AB 2013 requires developers of publicly available generative AI systems to publish detailed documentation: data sources, dataset size, data types, whether copyrighted material is included, whether datasets were purchased or licensed, whether personal information is included, and data processing methods used.

The EU AI Act imposes transparency and other obligations based on risk to users. General-purpose AI providers must publish training dataset summaries and respect copyright holder opt-outs. Penalties: up to 35 million euros or 7% of global annual turnover.

On the litigation front, the report cites Bartz v. Anthropic (2025): training on legally obtained works is defensible, training on pirated content is not. In Kadrey v. Meta, market harm was a decisive factor.

Compliance, Zyte concludes, becomes an operational requirement. Enterprises will not adopt AI systems without evidence of lawful data sourcing. Provenance tracking becomes necessary for audits, investors, and enterprise customers.

Conclusions

We’re living in an unprecedented era: we have tools in our hands that are improving the efficiency of our work, and we’re still scratching the surface of what they can do. But the LLM training and Agents brought scraping operations to a new level, making anti-bot softwares are more important than ever and raising the bar for the scraping industry itself, which is responding with more advanced tools and APIs. For sure, we’ll have interesting times ahead.

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

---

Tool Landscape

We evaluated three browser automation tools that claim to bypass Cloudflare in 2026.

Camoufox is a custom Firefox build with fingerprint rotation and stealth patches. It uses Playwright’s Juggler protocol and focuses on avoiding detection through realistic Firefox fingerprints and non-default configurations.

Pydoll uses Chrome DevTools Protocol (CDP) for async Chromium automation. It avoids WebDriver entirely, emphasizing human-like interactions and behavioral anti-detection.

undetected-chromedriver provides a patched Selenium ChromeDriver. It modifies startup behavior and WebDriver fingerprints, serving as a drop-in replacement for standard Selenium workflows.
The code used for this test can be found on The Lab GitHub repository, inside the folder 95.CLOUDFLARE-2026, available only for paid subscriber of TWSC.

System Model: Cloudflare’s Detection Layers

Cloudflare operates as a multi-layered defense system. Understanding which layer blocks your requests determines which tool characteristics matter.

Layer 1: involves TLS and network fingerprinting. Cloudflare analyzes TLS handshakes and HTTP/2 frame ordering, inspecting cipher suites, negotiation order, and HTTP/2 header sequences to identify non-browser clients. Browser automation tools using real Chrome or Firefox inherit legitimate TLS fingerprints, typically passing this layer.

Layer 2: runs JavaScript Detections (JSD). Cloudflare’s JavaScript Detections engine executes lightweight JavaScript on HTML page requests to identify headless browsers and automated clients. The detection runs via an invisible code snippet that analyzes browser environment properties without visible challenges. When verification succeeds, Cloudflare issues a cf_clearance cookie with cf.bot_management.js.detection.passed = true. Failed verifications block the request or trigger additional challenges.

Layer 3: presents the JavaScript Challenge (IUAM), the visible “Checking your browser” interstitial. This executes complex JavaScript to validate browser capabilities and measure execution timing. The challenge tests Canvas and WebGL rendering, navigator property consistency, and execution patterns.

Layer 4: implements behavioral analysis. Cloudflare monitors mouse movements, scrolling patterns, click timing, and request sequencing to identify bot-like behavior. These signals feed into the bot scoring system.

Layer 5: applies machine learning bot scoring. Cloudflare’s ML engine assigns Bot Scores (1-99) using supervised learning to distinguish human and bot traffic. This accounts for the majority of detections and incorporates signals from all other layers plus IP reputation.

For this test, we focused on whether tools can pass the JavaScript detection and challenge layers (Layers 2-3). We assumed clean residential IPs and did not evaluate IP reputation impact.

Target Selection and URL Collection

We needed real production targets with active Cloudflare protection, and we chose two very well-known websites.

Harrods.com operates as a high-value e-commerce site where Cloudflare protects product pages and navigation. The sitemap is accessible at https://harrods.com/sitemap.xml.

Indeed.com is a job board with aggressive protection. When we attempted sitemap access, we received 403 Forbidden, confirming active Cloudflare filtering even for automated sitemap requests.

URL Extraction Process

For Harrods.com, standard sitemap parsing works:

def extract_urls_from_domain(domain: str, limit: int = 1000) -> List[str]:
    sitemap_url = f'https://{domain}/sitemap.xml'
    xml_content = fetch_sitemap(sitemap_url)

    if '<sitemapindex' in xml_content:
        sitemap_urls = parse_sitemap_index(xml_content)
        for sm_url in sitemap_urls:
            sm_content = fetch_sitemap(sm_url)
            urls = parse_sitemap(sm_content)
            all_urls.update(urls)
    else:
        urls = parse_sitemap(xml_content)
        all_urls.update(urls)

    return list(all_urls)[:limit]

For Indeed.com, sitemap access returned 403, forcing us to generate URLs based on known patterns:

def generate_indeed_urls(limit: int = 1000) -> List[str]:
    urls = []
    job_queries = ['software-engineer', 'data-scientist', 'product-manager', ...]
    locations = ['New-York-NY', 'Los-Angeles-CA', 'Chicago-IL', ...]

    for query in job_queries:
        for location in locations:
            urls.append(f'https://www.indeed.com/jobs?q={query}&l={location}')
            for start in [10, 20, 30]:
                urls.append(f'https://www.indeed.com/jobs?q={query}&l={location}&start={start}')

    return urls[:limit]

The 403 on sitemap access itself demonstrates Cloudflare’s filtering. Even basic reconnaissance triggers blocks.

Test Framework Design

We designed our test framework to validate successful page loads by checking for site-specific content elements rather than generic Cloudflare blocking indicators.

Content-Based Validation

We discovered early that generic Cloudflare detection (searching for “checking your browser” or “just a moment” strings) produces false positives. Tools can retrieve partial pages or incomplete JavaScript renders that contain neither Cloudflare challenges nor the actual target content.

Our validation approach checks for critical page elements that must be present in a successfully loaded page:

Indeed.com validation:

has_search_button = (
    'yosegi-InlineWhatWhere-primaryButton' in html and
    '<button' in html and
    '>Search</span>' in html
)

result = {
    'success': has_search_button,
    'status_code': 200 if has_search_button else (403 if html else 0),
    'content_length': len(html),
}

The search button element appears only when the full job listing page renders. Its absence indicates either Cloudflare blocking or incomplete JavaScript execution.

Harrods.com validation:

has_product_name = (
    'data-test-id="pdp-product-name"' in html and
    '<h1' in html
)

result = {
    'success': has_product_name and status_code == 200,
    'status_code': status_code,
    'content_length': len(html),
}

Product pages must contain the product name h1 element. Pages without this element are either editorial content, error pages, or Cloudflare blocks.

Metrics Collected

For each request, we recorded a success flag (based on content validation), HTTP status code (or inferred status), content length in bytes, final URL after redirects, and error messages when exceptions occurred.

Check the TWSC YouTube Channel

Tool Implementation

Camoufox Setup

Camoufox requires downloading the custom Firefox binary:

pip install -U "camoufox[geoip]"
python -m camoufox fetch

We configured it for headless operation with fingerprint rotation:

from camoufox.sync_api import Camoufox

def scrape_with_camoufox(url: str) -> Dict:
    with Camoufox(
        headless=True,
        humanize=True,
        os=['macos', 'windows'],
        geoip=False,
    ) as browser:
        page = browser.new_page()
        response = page.goto(url, timeout=30000, wait_until='domcontentloaded')
        page.wait_for_timeout(2000)
        html = page.content()

        return {
            'html': html,
            'status_code': response.status if response else 0,
            'url': page.url,
        }

We set headless=True for server execution. The humanize=True option enables human-like cursor movement to reduce behavioral signals. The os=[’macos’, ‘windows’] parameter rotates OS fingerprints between requests. We disabled geoip due to a lack of proxy infrastructure for this test.

The wait_for_timeout(2000) accounts for JavaScript challenges that execute after page load.

Undetected-chromedriver Setup

Standard pip installation works:

pip install undetected-chromedriver

Configuration uses the new headless mode to reduce detection surface:

import undetected_chromedriver as uc

def scrape_with_undetected_chromedriver(url: str) -> Dict:
    options = uc.ChromeOptions()
    options.add_argument('--headless=new')
    options.add_argument('--disable-blink-features=AutomationControlled')
    options.add_argument('--disable-dev-shm-usage')
    options.add_argument('--no-sandbox')

    driver = uc.Chrome(options=options, version_main=None)
    driver.get(url)

    time.sleep(2)
    html = driver.page_source
    final_url = driver.current_url

    driver.quit()

    return {
        'html': html,
        'status_code': 200,  # Selenium doesn't expose status
        'url': final_url,
    }

The --headless=new flag uses Chrome’s updated headless mode with reduced fingerprint differences. The --disable-blink-features=AutomationControlled option removes the navigator.webdriver flag. The version_main=None parameter auto-detects the installed Chrome version.

The 2-second sleep allows JavaScript challenges to complete.

Pydoll: Chromium CDP-Based Automation

Pydoll uses Chrome DevTools Protocol (CDP) for browser control. We found that the API differs from the documentation, requiring experimentation to identify working methods.

Working configuration:

from pydoll.browser.chromium import Chrome
from pydoll.browser.options import ChromiumOptions

options = ChromiumOptions()
options.add_argument('--headless=new')
options.add_argument('--disable-blink-features=AutomationControlled')
options.add_argument('--disable-dev-shm-usage')
options.add_argument('--no-sandbox')

async with Chrome(options=options) as browser:
    tab = await browser.start()
    await tab.go_to(url)
    html = await tab.page_source  # Use page_source, not tab.html()
    current_url = await tab.current_url  ```

We encountered API inconsistencies during testing. Documentation suggests from pydoll import Browser, but the Browser class doesn’t exist. The correct method is tab.page_source, not tab.html(). URL retrieval uses tab.current_url as an attribute, not a method call.

These inconsistencies required us to test against the installed version (2.15.1) to identify working patterns.

Need help with your scraping project?

Test Results: Indeed.com with Camoufox

When we tested Camoufox against Indeed.com, we observed behavioral patterns specific to how Cloudflare’s Turnstile operates on this target.

Read more