In this article, you’ll find out this and more. I’ll walk you through a complete guided example of how to use it, and break down its (Spoiler: undoubtedly serious) limitations.

An Introduction to the Cloudflare Crawl Endpoint

Before exploring the technical aspects behind the Cloudflare /crawl endpoint and seeing it in action, let me first give you some context!

What Is the Cloudflare /crawl Endpoint?

The /crawl endpoint is a new addition to Cloudflare’s REST APIs. Its goal is to crawl an entire website (or just a portion of it) starting from a single URL.

Note: The Crawl endpoint is currently in beta and was introduced on March 10, 2026, as highlighted in the Cloudflare changelog.

Under the hood, it automatically discovers and visits new pages, rendering them in a headless browser. It returns the discovered content as HTML, Markdown, or structured JSON, making it ideal for RAG pipelines, monitoring, or dataset creation.

As I’ll dive into later, it respects robots.txt and doesn’t bypass bot protection or captchas. Thus, it’s designed as a compliant approach to web crawling!

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

How It Works at a High Level

At a high level, the /crawl endpoint involves a two-step flow:

1. You kick off an asynchronous crawl job, passing a starting URL. Cloudflare returns a job ID.

2. You use that job ID to periodically check the job’s status or fetch results as they become available, following typical polling behavior.

Important: A crawl job can run for up to seven days! Results remain available for 14 days after completion, after which the job data is deleted.

Behind the scenes, the crawler expands outward from the starting URL. By default, the API follows a clear order:

1. The initial page.

2. Sitemap URLs.

3. Links discovered within pages.

Still, you can tweak that depending on whether you want to prioritize sitemaps, page links, or both.

Supported Use Cases

The officially promoted use cases for the Cloudflare /crawl API are just two:

• Creating knowledge bases or training AI systems (like RAG applications) using up-to-date web content.

• Collecting and analyzing content across multiple pages for research, summarization, or monitoring purposes.

Pricing

Compared to most other web crawling or discovery APIs on the market, Cloudflare’s /crawl API doesn’t charge by the number of pages. Instead, costs are based on resource usage, which depends on whether you enable the headless browser rendering feature.

If headless rendering is active, pricing follows the Browser Rendering model:

If rendering isn’t active, pricing follows the Workers model:

Yeah, I know… It’s honestly a bit confusing, and it’s almost impossible to predict the exact cost of a crawling task. The good news? You can test it for free!

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

Cloudflare Crawl Endpoints: Technical Analysis

Now that you know what Cloudflare is and what it brings to the table, it’s time to better understand its functioning, strengths, and limitations.

Endpoint Presentation

The Cloudflare Crawl API is built around two main endpoints. Both share the same base URL:

https://api.cloudflare.com/client/v4/accounts/<ACCOUNT_ID>/browser-rendering/crawl

Where <ACCOUNT_ID> is your Cloudflare account ID.

1. Initiate the Crawl Job (POST)

To start a new crawl, you need to send a POST request with the target URL (and optional parameters like depth, rendering mode, etc.) as below:

curl -X POST 'https://api.cloudflare.com/client/v4/accounts/<ACCOUNT_ID>/browser-rendering/crawl' \
  -H 'Authorization: Bearer <CLOUDFLARE_API_TOKEN>' \
  -H 'Content-Type: application/json' \
  -d '{ "url": "https://example.com" }'

Keep in mind that the endpoint supports several parameters, allowing you to greatly customize the crawling behavior, output format (JSON, HTML, or Markdown), rendering options, caching, and more. Check out the full list of supported body parameters for all available options.

Cloudflare immediately returns a job ID that you’ll use to track or retrieve results. A possible response looks like this:

{
  "success": true,
  "result": "9f1c2d3a-4b5e-6f7a-8c9d-0e1f2a3b4c5d"
}

The UUID in the result field is the Crawl job ID you’ll use to poll for updates.

2. Request Crawl Results (GET)

Once the crawl is running, make a GET request with the job ID to check the status or fetch results:

curl -X GET 'https://api.cloudflare.com/client/v4/accounts/<ACCOUNT_ID>/browser-rendering/crawl/<JOB_ID>' \
  -H 'Authorization: Bearer <CLOUDFLARE_API_TOKEN>'

Here, the <JOB_ID> placeholder is the UUID retrieved before from the result field.

The response either includes a status field like this:

{
  "result": {
    "id": "3e7a1c92-b4d8-4f6e-9a21-6c0d5b8e2f14",
    "status": "running"
    // ...
  }
}

The possible status values are: running, completed, errored, or one of several cancellation states (cancelled_due_to_timeout, cancelled_due_to_limits, cancelled_by_user).

Or, once the job is completed, calling the API returns the full results in the records field:

{
  "result": {
    "id": "3e7a1c92-b4d8-4f6e-9a21-6c0d5b8e2f14",
    "status": "completed",
    "browserSecondsUsed": 98.3,
    "total": 12,
    "finished": 12,
    "records": [
      {
        "url": "https://example.com/",
        "status": "completed",
        "markdown": "# Example Domain\nThis domain is for use in illustrative examples...",
        "metadata": {
          "status": 200,
          "title": "Example Domain",
          "url": "https://example.com/"
        }
      },
      {
        "url": "https://example.com/about",
        "status": "completed",
        "markdown": "## About\nLearn more about this example site...",
        "metadata": {
          "status": 200,
          "title": "About - Example Domain",
          "url": "https://example.com/about"
        }
      }
      // additional entries omitted for brevity...
    ],
    "cursor": 10
  },
  "success": true
}

Note that the response will vary based on the specified query parameters. For example, you can filter by specific statuses, limit the number of results, and navigate through them using a pagination system.

Features

Below is a list of the main, most relevant capabilities provided by the Cloudflare Crawl API:

• Asynchronous crawl jobs: Trigger crawling jobs and poll results when they are ready, enabling non-blocking, large-scale crawling workflows.

• Automatic URL discovery: Finds pages from the starting URL, sitemaps, and in-page links, with configurable source control.

• Flexible output formats: Returns HTML, Markdown, or structured JSON. JSON leverages Workers AI for schema-driven data extraction.

• Headless browser rendering: Control JavaScript execution with render: true or perform fast static HTML fetches with render: false.

• Fine-grained crawl control: Configure limit, depth, and URL inclusion/exclusion with the includePatterns/excludePatterns fields.

• Incremental and cache-aware crawling: Use modifiedSince and maxAge parameters to avoid re-fetching unchanged content, optimizing performance and cost.

• Advanced filtering and pagination: Retrieve results using limit, cursor, and status filters to handle large datasets efficiently.

• Authentication and custom headers: Supports HTTP auth, cookies, and custom headers for crawling protected or API-driven content.

• Dynamic content handling: Wait for JS-rendered content using gotoOptions and waitForSelector, ideal for SPAs and interactive pages.

• Resource skipping for performance: Optionally block images, media, fonts, or stylesheets to speed up crawling and reduce unnecessary bandwidth usage.

Your scraping workflows deserve a proxy infrastructure that just works. With Swiftproxy on your side, consistency is built-in.

Try Swiftproxy today

Limitations

Cloudflare’s /crawl API also comes with several important limitations, such as:

• Respects bot protection: The crawler can’t bypass CAPTCHAs (including Turnstile challenges) or Cloudflare bot protections. As a rule of thumb, sites protected via Cloudflare Bot Management or other WAFs tend to block crawling tasks entirely, limiting automated access and leading to incomplete datasets.

• Fixed User-Agent: The /crawl endpoint sets a non-customizable User-Agent value (CloudflareBrowserRenderingCrawler/1.0). You can’t change it, which may cause sites to block requests or serve different content based on the User-Agent.

• Content Signals enforcement: If a site disallows AI usage via Cloudflare Content Signals, crawl requests for those purposes are rejected with a 400 Bad Request error. Even if the site allows other uses, attempts to crawl disallowed categories will fail, limiting AI-specific data collection.

• Rate limiting and crawl pacing: Sites with strict rate limits can slow down crawling. The crawler respects the robots.txt Crawl-delay directive and implements backoff. Large crawls may need to be split into smaller jobs to avoid throttling or skipped URLs.

• Browser usage limits and job cancellation: Accounts on Workers free plans are capped at 10 minutes of browser time per day. Exceeding this limit results in a cancelled_due_to_limits status. To avoid that, you can upgrade to a paid plan.

How to Use the Cloudflare Crawl Endpoint: Step-by-Step Tutorial

In this guided section, I’ll show you how to use the Cloudflare Crawl Endpoint to crawl a website in Python. The target site will be the “Quotes to Scrape” sandbox. The goal here is to demonstrate how to use the API, rather than actually collecting relevant data.

Follow the instructions below!

Prerequisites

To follow this tutorial section, make sure you have:

• Your Cloudflare account ID at hand.

• A Cloudflare API token with the “Browser Rendering - Edit” permission.

For the sake of simplicity and to keep this tutorial concise, I’ll assume you already have a Python project set up with requests installed. That said, you can use any programming language and any HTTP client, because the high-level logic remains the same.

Step #1: Set Up the Configurations

Start by importing the required libraries and reading the necessary secrets (your Cloudflare API token and account ID). Use these secrets to prepare the Cloudflare Crawl base URL and headers. Also, specify the starting target URL as a constant.

import requests
import time
import json

# The Cloudflare secrets required for authenticating Crawl API calls
CLOUDFLARE_API_TOKEN = "<YOUR_CLOUDFLARE_API_TOKEN>"
CLOUDFLARE_ACCOUNT_ID = "<YOUR_CLOUDFLARE_ACCOUNT_ID>"

# The base URL used for all Crawl API calls
CLOUDFLARE_CRAWL_BASE_URL = f"https://api.cloudflare.com/client/v4/accounts/{CLOUDFLARE_ACCOUNT_ID}/browser-rendering/crawl"

# Common headers shared by all API calls
HEADERS = {
    "Authorization": f"Bearer {CLOUDFLARE_API_TOKEN}",
    "Content-Type": "application/json"
}

# The URL to crawl
START_URL = "https://www.ssense.com/en-us/men/product/acne-studios/silver-folded-leather-wallet/18169981"

Tip: In a production script, read the Cloudflare API token and account ID from environment variables rather than hardcoding them.

Step #2: Trigger the Crawling Job

Define a start_crawl() function to send a POST request to Cloudflare’s Crawl API:

def start_crawl(start_url):
    # Customize according to your needs
    payload = {
        "url": start_url,
        "limit": 20,
        "depth": 2,
        "formats": ["markdown"],
        "render": True
    }

    response = requests.post(CLOUDFLARE_CRAWL_BASE_URL, headers=HEADERS, json=payload)
    response.raise_for_status()

    job_id = response.json()["result"]
    return job_id

This creates a new crawling job for the target URL. Then, it returns a job_id that identifies this specific crawl.

Tip: In a production-level script, make the payload object configurable via function input arguments for greater flexibility and reusability.

Step #3: Poll Over the Job

Next, add a wait_for_completion() function to repeatedly check the job status every few seconds until the crawl finishes or times out:

def wait_for_completion(job_id, max_attempts=60, delay=5):
    status_url = f"{CLOUDFLARE_CRAWL_BASE_URL}/{job_id}?limit=1"

    for attempt in range(max_attempts):
        res = requests.get(status_url, headers=HEADERS)
        res.raise_for_status()

        status = res.json()["result"]["status"]
        print(f"Attempt {attempt+1}: {status}")

        if status != "running":
            return status

        time.sleep(delay)

    raise Exception("Timeout waiting for crawl job")

This makes GET calls to the Cloudflare /crawl endpoint. It ensures you’re waiting for the task to complete processing before fetching the crawled records.

Tip: The limit=1 query parameter is recommended to restrict the number of retrieved records, keeping the response lightweight. After all, at this stage, you’re only interested in checking the job status, not in retrieving the actual output data.

Check the TWSC YouTube Channel

Step #4: Get the Crawled Content Pages

Build a fetch_records() function to collect all crawled pages:

def fetch_records(job_id):
    print("Fetching results...")

    all_records = []
    cursor = None

    while True:
        url = f"{CLOUDFLARE_CRAWL_BASE_URL}/{job_id}"
        # Getting 10 records at a time
        params = {
            "limit": 10
        }

        if cursor:
            params["cursor"] = cursor

        response = requests.get(url, headers=HEADERS, params=params)
        response.raise_for_status()

        result = response.json()["result"]
        records = result.get("records", [])

        all_records.extend(records)
        print(f"+{len(records)} records (Total: {len(all_records)})")

        cursor = result.get("cursor")
        if not cursor:
            break

    return all_records

This handles pagination using a cursor, accessing records in batches (10 per request) until all results are returned.

Step #5: Put It All Together

Finally, in the main() function, orchestrate the workflow:

1. Start the crawl

2. Wait for completion

3. Fetch all results

Then, you can export the crawled records to a local JSON file for further use, store the retrieved data in a database, process it there, etc.

def main():
    job_id = start_crawl(START_URL)
    status = wait_for_completion(job_id)

    if status != "completed":
        raise Exception(f"Crawl failed: {status}")

    print("Crawl completed!")

    records = fetch_records(job_id)

    # Export the crawled pages to an output JSON file
    with open("records.json", "w", encoding="utf-8") as f:
        json.dump(records, f, indent=2, ensure_ascii=False)

if __name__ == "__main__":
    main()

Step #6: Complete Code

This is what your Python script for interacting with the Cloudflare Crawl API will look like:

# pip install requests

import requests
import time
import json

# The Cloudflare secrets required for authenticating Crawl API calls
CLOUDFLARE_API_TOKEN = "<YOUR_CLOUDFLARE_API_TOKEN>"
CLOUDFLARE_ACCOUNT_ID = "<YOUR_CLOUDFLARE_ACCOUNT_ID>"

# The base URL used for all Crawl API calls
CLOUDFLARE_CRAWL_BASE_URL = f"https://api.cloudflare.com/client/v4/accounts/{CLOUDFLARE_ACCOUNT_ID}/browser-rendering/crawl"

# Common headers shared by all API calls
HEADERS = {
    "Authorization": f"Bearer {CLOUDFLARE_API_TOKEN}",
    "Content-Type": "application/json"
}

# The URL to crawl
START_URL = "http://quotes.toscrape.com/"

def start_crawl(start_url):
    """
    Triggers the Cloudflare Crawl API job
    """

    # Customize according to your needs
    payload = {
        "url": start_url,
        "limit": 20,
        "depth": 2,
        "formats": ["markdown"],
        "render": True
    }

    response = requests.post(CLOUDFLARE_CRAWL_BASE_URL, headers=HEADERS, json=payload)
    response.raise_for_status()

    job_id = response.json()["result"]
    return job_id

def wait_for_completion(job_id, max_attempts=60, delay=5):
    """
    Waits for the crawling task to complete
    """

    status_url = f"{CLOUDFLARE_CRAWL_BASE_URL}/{job_id}?limit=1"

    for attempt in range(max_attempts):
        res = requests.get(status_url, headers=HEADERS)
        res.raise_for_status()

        status = res.json()["result"]["status"]
        print(f"Attempt {attempt+1}: {status}")

        if status != "running":
            return status

        time.sleep(delay)

    raise Exception("Timeout waiting for crawl job")

def fetch_records(job_id):
    """
    Collects all records from the paginated results
    """

    print("Fetching results...")

    all_records = []
    cursor = None

    while True:
        url = f"{CLOUDFLARE_CRAWL_BASE_URL}/{job_id}"
        # Getting 10 records at a time
        params = {
            "limit": 10
        }

        if cursor:
            params["cursor"] = cursor

        response = requests.get(url, headers=HEADERS, params=params)
        response.raise_for_status()

        result = response.json()["result"]
        records = result.get("records", [])

        all_records.extend(records)
        print(f"+{len(records)} records (Total: {len(all_records)})")

        cursor = result.get("cursor")
        if not cursor:
            break

    return all_records

def main():
    job_id = start_crawl(START_URL)
    status = wait_for_completion(job_id)

    if status != "completed":
        raise Exception(f"Crawl failed: {status}")

    print("Crawl completed!")

    records = fetch_records(job_id)

    # Export the crawled pages to an output JSON file
    with open("records.json", "w", encoding="utf-8") as f:
        json.dump(records, f, indent=2, ensure_ascii=False)

if __name__ == "__main__":
    main()

Step #7: Test the Script

Launch the script, and it’ll produce an output like this:

The polling mechanism required 5 attempts (~25 seconds), and the API discovered and retrieved 22 pages.

A records.json file will appear in your project directory. Open it, and you’ll see:

Notice how the “Quotes to Scrape” entries contain a markdown field with the Markdown version of the page. Instead, external links like Zyte’s homepage and Goodreads.com are skipped, since includeExternalLinks is set to false by default. In other words, the Cloudflare Crawl API doesn’t automatically attempt to fetch data from different domains than the target source URL.

Et voilà! Implementation complete.

Need help with your scraping project?

Benchmark Against Protected Websites

Cool! The Cloudflare Crawl endpoint works like a charm and is easy to use. However, I was particularly concerned about its documented limitations and wanted to verify whether they actually hold up in practice…

So, I ran tests against several well-known sites protected by common WAF and anti-bot solutions (from different providers). Here’s a summary of the results:

As you can tell, the limitations are very real. The results are quite discouraging: the Cloudflare Crawl API failed against all anti-bot–protected websites I tested.

So, is this solution reliable for web scraping? When (and how) should you actually use it? Let me break that down in a final comment!

Final Comment

In this article, I introduced you to one of the newest tools in Cloudflare’s growing ecosystem: the Crawl API! This endpoint is designed to help you crawl entire websites using distributed crawling tasks running on Cloudflare’s infrastructure.

Sure, the crawling mechanism works and is easy to launch, control, and implement. With just a few lines of code, you can get started. Still, several concerns should be raised:

1. Opaque pricing: Costs are tied to resource usage rather than the number of pages crawled, making them harder to predict.

2. Fixed User-Agent: The API doesn’t allow User-Agent customization, meaning even basic server-side checks can block it.

3. Limited effectiveness on protected sites: The API has an intended very low success rate against anti-bot–protected websites (unless you specify in Cloudflare Bot Protection settings that you allow it against your site).

4. Rate limiting constraints: It strictly respects robots.txt directives and crawl delays, which can significantly slow or limit large crawls.

In simple terms, if you want to use it for general-purpose, large-scale web crawling, I wouldn’t recommend it. The market offers more effective solutions that can actually bypass anti-bot limitations. Plus, remember that around 35% of the entire Internet is estimated to be protected against bots (i.e., you won’t be able to crawl it with this API).

Yet, if you know the target site is not protected, budget isn’t a concern, and you want to remain (overly?) ethical and compliant, the Cloudflare Crawl API can be an option.

I hope this breakdown helps you better understand this new solution and make an informed decision on whether to adopt it. Lastly, remember that the Cloudflare Crawl API is still in beta, so things may change soon. Just keep an eye on the docs for updates. Until next time!

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

The short answer is that a couple of tools finished the task, just one with any consistency. The story behind why is worth telling, because it explains what is happening at the intersection of AI agents and web data right now. We ran a similar exercise against Cloudflare earlier this year, and the conclusion is broadly the same: each anti-bot needs its own answer, and the answer changes every quarter.

From workflows to agents, and why that changes the data problem

Most code shipped under the AI banner is not really agentic. It is workflow code with an LLM dropped into a slot: generate a summary here, classify a record there, draft an email at the end. The control flow is hard-coded, and the model is one component among many.

The definition of an agent is quite different. The model decides the next action, observes the outcome, and decides again. The control flow lives inside the loop, not outside it. The agent has goals rather than scripts, and it picks tools and steps based on what it sees. That is what makes the engineering interesting, that is what makes it hard, and that is what sometimes makes it unreliable.

It also forces a different relationship with data. An agent that only sees its training corpus is stuck in the past. To make decisions worth anything, it has to read prices that change daily, stocks that move minute by minute, listings that did not exist last week. Some of that data sits behind APIs. Most of it does not. The web is still the largest and most current dataset in the world, and most of it is reachable only through a browser. So if we want our agents to act on real information, we have to give them a way to browse: opening a page, reading it, clicking a link, typing into a search bar, following a result, filling a form, all on sites that were never built for machines.

Start your scraping journey with Byteful: 10GB New Customer Trial | Use TWSC for 15% OFF | $1.75/GB Residential Data | ISP Proxies in 15+ Countries

Claim your 10GB here

This is the constraint that produced the wave of “agentic browser” launches we have seen over the last twelve months. Y Combinator alone has backed a long string of them. Hyperbrowser (S21) was an early entry: scalable cloud browser infrastructure with built-in CAPTCHA solving, proxy management, and now a multi-agent playground. The newer cohort followed the agent wave more directly: BrowserOS (S24) is an open-source agentic browser that runs the agent locally on the user’s machine; Browser Use (W25) offers an open-source agent loop on top of Playwright, plus a cloud version. Skyvern is a self-hostable browser agent that uses an LLM and computer vision instead of fixed selectors. Outside the YC pipeline, Lightpanda is doing something different again, a headless browser engine written from scratch in Zig and aimed squarely at agents and crawlers (claiming roughly 9x faster execution and 16x lower memory than Chrome). It fits the “browser built for machines” line of thought we covered in Rethinking the web browser earlier this year. Browserbase ships a managed browser plus Stagehand for natural-language automation. And the big AI labs are now in the same space: OpenAI shipped Operator and the ChatGPT Atlas browser, Anthropic shipped Computer Use, Perplexity launched Comet. Each project attacks the same problem from a slightly different angle, but the goal is identical: a browser an agent can drive without immediately tripping every detection mechanism on the other side.

Your scraping workflows deserve a proxy infrastructure that just works. With Swiftproxy on your side, consistency is built-in.

Try Swiftproxy today

The same problem scrapers have been chasing for a decade

For anyone who has worked in web data, none of this is new. The fight over whether a request looks human or automated has been going on as long as commercial scraping has existed. The product names have changed but the purpose not.

What has changed is who is selling the bypass. The companies that have spent years selling residential proxies and unblockers noticed quickly that the agentic boom is good for their business. They already have the IP networks, the fingerprint research, the bypass code, the cat-and-mouse experience. They know what TLS handshake Chrome sends in October 2025 and what it sent in October 2024. Pivoting all of that into a managed browser is a smaller leap than building one from scratch. Bright Data, Oxylabs, Rayobyte, ZenRows have all added a managed browser product alongside the proxy.

The other side of the line is moving in the opposite direction. Bot traffic has grown faster than human traffic for years, and the operators of large public sites care more about it than ever. DataDome, Cloudflare Bot Management, Akamai Bot Manager, HUMAN, Kasada: every one of them ships updates that target the exact tools we just listed. Fingerprint checks get stricter. Behavioral models get more sensitive. The JavaScript challenge changes shape every few weeks. There is no silver bullet, and there is no tool, browser, proxy, or service that bypasses every anti-bot on every site at all times. Anyone who claims otherwise is selling something that worked last quarter and might still work this week. The useful question is what works on a given target, today, at what cost.

Check the TWSC YouTube Channel

Picking a hard target

To answer that question concretely, we needed a target where the anti-bot was good and the signal was clean. We picked leroymerlin.fr, the French DIY retailer. Leroy Merlin runs DataDome standalone, with no other anti-bot layer on top, so attribution is straightforward. It also runs one of the more verbose DataDome configurations we have come across: response headers expose x-datadome-riskscore, x-datadome-protection, x-datadome-cid, and x-datadome-endpointid. Most DataDome-protected sites only show us the outcome. Here we see the score the engine assigns at every request, which is rare and very useful when comparing tools side by side.

The task we picked is small but realistic. From the homepage, the agent has to type “ampoule B22 led blanc” into the search bar, click the first product result, and add the product to the cart. Four steps. We dropped the login step on purpose: leroymerlin.fr requires an OTP to sign in, and we did not want OTP friction to confound an anti-bot test.

A run is a pass if the agent reaches the cart confirmation. Otherwise we record where it stopped and what DataDome said about it. Each tool runs ten times back to back, and we aggregate the results. Tools that support an external proxy use the same residential pool: Bright Data residential FR for the Bright Data runs, Geonode residential FR for the Geonode runs. Tools that ship their own proxy use it. The reason behind two different providers was because we wanted to diversify the IP addresses, to be sure that blocks were not a matter of IP reputation.

Need help with your scraping project?

The contestants

As you’ve seen before, the browser landscape is quite crowded and we could not cover all the tools. We picked four open-source projects and seven commercial products. Let’s start with the open source.

Camoufox is the stealth Firefox fork most people in the scraping world have already met (we introduced it on TWSC back in September 2024). It rotates real-world fingerprints, patches the obvious automation tells, and ships a Playwright-compatible API. We pair it with both Bright Data and Geonode residential proxies in France.

Pydoll takes a different route: it drives Chromium directly over CDP without WebDriver, with built-in humanized cursor movement and typing. Importantly, Pydoll implements an explicit Fetch.authRequired handler, which lets it authenticate proxies that require Basic auth.

Scrapling is a higher-level Python library. We use it in two modes. DynamicFetcher launches vanilla Playwright Chromium driven by Scrapling’s session manager. StealthyFetcher does the same, but under the hood uses patchright, a stealth-patched Playwright fork. Each gets its own row in the comparison.

RayoBrowse is the self-hosted stealth Chromium fork from Rayobyte, distributed as a Docker container that exposes a CDP endpoint on port 9222. Here we hit a wall worth flagging: for some reason RayoBrowse could not use the Bright Data residential proxy in our setup. Every navigation through that proxy failed instantly, even though the same credentials worked fine through curl from inside the same container. The same RayoBrowse setup worked fine with Geonode. We did not isolate the root cause, so we report RayoBrowse on Geonode only.

The commercial side is more crowded.

Browser Use exists in two flavors, and we tested both. The cloud version is the managed Browser Use, with its own residential proxy, its own stealth fingerprinting, and a fixed set of supported models; we drove it once in raw CDP mode (we steer it ourselves with Playwright) and once in agent mode (we hand the LLM the task in natural language and let it plan the steps).

Browserbase is a managed Chromium with optional residential proxies, Cloudflare Web Bot Auth verification, and the Stagehand agent SDK. We discovered during the test that the free tier excludes proxies entirely; without one, the session egresses from a US datacenter. We left this configuration in the test because it is what a free user would experience.

Browserless is a managed browser-as-a-service whose anti-bot story is a stealth path (/chromium/stealth) plus optional residential proxies for paid plans. The free plan caps sessions at 60 seconds, which is tight for a four-step flow. We tested it with the built-in residential proxy targeting France, and tried to test it with our external proxies via the externalProxyServer parameter; the external mode failed at connection time on every run, in the same Chromium-side authentication way that broke RayoBrowse, so we drop those configurations from the comparison.

ZenRows Scraping Browser is a managed Chromium with a built-in residential proxy network and built-in CAPTCHA solving; we connect via the WSS endpoint with proxy_country=fr to get a French exit point.

Bright Data Browser API sits at the other end of the same product category: a managed Chromium with built-in residential rotation and CAPTCHA solving, on a dedicated Browser API zone we configured on their dashboard.

As always, the code can be found in our GitHub repository reserved to paying users, inside the folder 103.BROWSERS.

What we had to fix before the numbers made sense

Read more

This means that if you route all your traffic through a single egress IP, you will be rate-limited in seconds and blacklisted in minutes. To survive at scale, you need to distribute your requests across a massive pool of IP addresses.

Traditionally, the web scraping industry has solved this issue thanks to commercial proxy providers. However, this is not the only approach. This article responds to the following question: “Is there a way to scrape at scale without burning budget on proxies?”

The answer is yes. But let’s be clear from the beginning: This approach is not a universal silver bullet. Let’s see how it works, how to build it, and what its limitations are.

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

The Typical Solution for Scraping at Scale: Proxy Provider Services

Let’s start this discussion with the typical choice for scraping at scale. IP bans and rate limits are the #1 operational problem in scraping, especially at scale. The typical solution every web scraping engineer integrates is using proxy servers, for a simple reason: proxies act as intermediaries between your scrapers and the Internet, avoiding your scrapers from getting banned. To do so, companies buy proxy IPs from proxy providers. The most common categories, both with their flaws, are the following:

• Datacenter proxies: These are cheap and fast, but their ASNs(Autonomous System Numbers) are heavily scrutinized. WAFs maintain databases of known datacenter CIDR (Classless Inter-Domain Routing) blocks, so hitting a target with a static list of 100 datacenter proxies usually results in those IPs being flagged and blocked within hours.

• Residential proxies: These route traffic through actual consumer devices. They have highly trusted IP reputations, making them excellent for bypassing anti-bot systems. However, they are priced by bandwidth, so they are very expensive, especially when scraping at scale.

The main limitation of this approach is that it is highly expensive. So, what if you need to scrape at scale but don’t have enough budget for doing so?

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

An Alternative Approach: Scraping at Scale With Dedicated Infrastructure

To escape metered billing, you can move egress back to dedicated infrastructure. But before presenting the solution, let’s first point out shortly what happens when you buy and use proxies, at the infrastructure level.

Buying Proxies Means Delegating Your Infrastructure

When you buy proxies from providers, you are delegating 100% of your infrastructure. When your scrapers make the requests, under the hood, the proxy provider connects to a gateway, which is a massive load balancer controlled entirely by the provider itself.

Let’s consider the case of residential proxies, for simplicity. Behind the gateway is a peer-to-peer (P2P) network of millions of consumer devices that the provider has acquired bandwidth from. When your request hits the gateway, their proprietary routing algorithm decides which consumer device in which country will act as your final exit node.

The second you route traffic through their gateway is the exact moment where you delegate the 100% of your scraping infrastructure.

Your scraping workflows deserve a proxy infrastructure that just works. With Swiftproxy on your side, consistency is built-in.

Try Swiftproxy today

NyxProxy: The Infrastructural Solution

NyxProxy is a self-hosted HTTP/SOCKS5 proxy server that exploits a well-known IPv6 networking trick: When a cloud provider gives you a /64 subnet, you legally own 18.4 quintillion IPv6 addresses.

Let’s explain the number and the trick around IPv6s. An IPv6 address looks like this:

 2a05:f480:1800:25db:0000:0000:0000:0001

They are 128 bits long. That gives 2^128 possible addresses. The number is so large that the designers said: “We can afford to give every organization a massive block and never worry about running out”.

Now, here is the trick. An IPv6 address is split into two halves, 64 bits each:

2a05:f480:1800:25db : 0000:0000:0000:0001
|___________________|   |_________________|
   Network prefix            Host part
   (your subnet)          (you control this)

The /64 notation means: the first 64 bits identify the network, the last 64 bits are yours to assign however you want. The last 64 bits can be any value from 0000:0000:0000:0000 to ffff:ffff:ffff:ffff: That’s 2^64 = 18.4 quintillion combinations. All valid addresses, all routable to your server.

Thanks to this trick, NyxProxy can assign a pool of those addresses to your network interface at startup, then rotate your outgoing traffic across them. This means having a fresh IP per request. The tool handles pool management, background rotation, NDP proxying via ndppd, and exposes a monitoring endpoint.

The best part is, indeed, in the NDP proxying. When your server uses a random address like 2a05:f480:1800:25db:a3f1:9922:beef:1234 as a source IP, your router upstream needs to know your server is responsible for that address. Otherwise, the response packets have nowhere to go.

IPv6 uses NDP (Neighbor Discovery Protocol) for this. The router sends an NDP query: “who has 2a05:f480:1800:25db:a3f1:9922:beef:1234?” and your server must answer.

ndppd (NDP Proxy Daemon) runs on your server and answers those queries automatically for your entire /64 subnet, essentially saying “yes, all of those addresses are mine”. Without it, your packets go out, but responses never come back.

Below is a summary schema of how this whole process works:

Provider gives you:  2a05:f480:1800:25db::/64
                     ↓
Your server can use: 2a05:f480:1800:25db:[anything]
                     ↓
NyxProxy assigns 200 random IPs to your interface
                     ↓
Each outgoing request binds to a different one
                     ↓
Target sees 200 different source IPs
                     ↓
ndppd makes sure responses route back correctly

Check the TWSC YouTube Channel

How To Use NyxProxy

Let’s now see how to use NyxProxy with a practical implementation.

Environment Setup & Prerequisites

To replicate this tutorial for deploying NyxProxy and utilizing it in your scraping scripts, you must have the following system and hardware requirements:

• Hardware: A Virtual Private Server (VPS) or bare-metal server with at least 512 MB of RAM and 100 MB of disk space. Supported architectures are amd64 or arm64.

• Subnet: A cloud provider that natively delegates a full IPv6 /64 subnet to your network interface. Note that not all the VPS providers are supported: Check out the NyxProxy documentation to learn more about supported VPSs.

• Operating system: A modern Linux distribution, specifically Ubuntu or Debian, to ensure compatibility with the automated setup scripts and sysctl kernel modifications.

• Python: Python 3.7 or higher installed on your local machine to run the scraping scripts.

To get your server ready to run the proxy daemon, you need to verify your IPv6 setup and gain root access. Ensure you are logged into your VPS via SSH as the root user, or have sudo privileges.

First, verify that your server has a globally routable IPv6 /64 subnet assigned to it. You can check this by running the following command in your server’s terminal:

ip -6 addr show | grep "scope global"

If done correctly, you should see an output similar to the following:

inet6 2a05:f480:1800:25db::1/64 scope global

If you do not see a /64 subnet, you will not be able to rotate IPs, and you must review your cloud provider’s network settings.

Next, prepare your local development environment. Suppose you call the main folder of your Python project nyxproxy_scraper/. At the end of this step, the folder will have the following structure:

nyxproxy_scraper/
    ├── main.py
    └── venv/

Where:

• main.py is the Python file that will store your proxy request logic.

• venv/ contains the standard Python virtual environment.

You can create the venv/ virtual environment directory like so:

python -m venv venv

To activate it, on Windows, run:

venv\Scripts\activate

Equivalently, on macOS and Linux, execute:

source venv/bin/activate

As a final prerequisite, install the Requests library in your activated virtual environment so your Python script can make HTTP calls:

pip install requests

Well done! You are now ready to test and use Nyxproxy.

Need help with your scraping project?

Installing and Configuring NyxProxy

NyxProxy provides a quick setup script that handles the infrastructural heavy lifting. It auto-detects your network interface, installs ndppd, tweaks the Linux kernel parameters via sysctl to allow non-local binding, and downloads the compiled Go binary.

You can launch it with the following single command:

wget <https://raw.githubusercontent.com/jannik-schroeder/nyxproxy-oss/main/scripts/quick-setup.sh> && chmod +x quick-setup.sh && sudo ./quick-setup.sh

During the setup, you will be prompted to configure your proxy credentials and set your rotation rules. Behind the scenes, the script generates a config.yaml file. Let’s look at the crucial subset of that configuration:

network:
  rotate_ipv6: true
  ipv6_subnet: "2a05:f480:1800:25db::/64"

  # The rotation mechanics:
  ipv6_pool_size: 200
  ipv6_max_usage: 100
  ipv6_max_age: 30

Below is an explanation of what these three parameters mean for your scraping pipeline:

• ipv6_pool_size: NyxProxy keeps 200 mathematically unique IPs “hot” and bound to your network interface at any given time. This keeps proxy startup times under 100ms while maintaining IP diversity.

• ipv6_max_usage: After a specific IP has been utilized for 100 requests, it is considered “burned.” NyxProxy destroys the route and spins up a fresh address to dynamically replace it.

• ipv6_max_age: If an IP hasn’t hit 100 requests but has been alive for 30 minutes, it gets forcefully rotated out. This prevents time-based algorithmic tracking by the target WAF.

Once the daemon is running as a systemd service, your VPS is officially acting as a rotating proxy gateway. When NyxProxy receives a scraper request, the underlying Go binary takes over. It looks at its internal memory, picks one of the 200 rotating IPv6 addresses in its pool, and binds to that specific address to establish the outbound connection.

The expected output is as follows:

IPv6 rotation mode: IP Pool with dynamic rotation
  Interface: enp1s0
  Subnet: 2a05:f480:1800:25db::/64
  Pool size: 200 IPs
  Rotation: Every 100 uses or 30m0s
  Initializing IP pool...
  Progress: 50/200 IPs added
  Progress: 100/200 IPs added
  Progress: 150/200 IPs added
  Progress: 200/200 IPs added
  IP pool ready with 200 addresses
  Background IP rotation started

Starting https proxy on 0.0.0.0:8080 (Protocol: IPv6)

Testing the Proxy Logic

At this point, NyxProxy has done its job. To verify it works correctly, you can use the following Python script that hits api6.ipify.org, which is an API that simply bounces back the IP address it sees:

import requests

# Point this to your VPS IP and the credentials you set during setup
proxies = {
    'http': '<http://admin:password@your-vps-ip:8080>',
    'https': '<http://admin:password@your-vps-ip:8080>'
}

# Test 5 consecutive scraping requests
for i in range(5):
    response = requests.get('<https://api6.ipify.org>', proxies=proxies)
    print(f"Request {i+1}: Target sees IP -> {response.text}")

(NOTE: If you are already familiar with ipify.org, note that the “api6” prefix can be used for IPv6 requests only.)

The result should be similar to the following:

Request 1: Target sees IP -> 2a05:f480:1800:25db:1a2b:3c4d:5e6f:7890
Request 2: Target sees IP -> 2a05:f480:1800:25db:9988:7766:5544:3322
Request 3: Target sees IP -> 2a05:f480:1800:25db:aaaa:bbbb:cccc:dddd
Request 4: Target sees IP -> 2a05:f480:1800:25db:1122:3344:5566:7788
Request 5: Target sees IP -> 2a05:f480:1800:25db:dead:beef:cafe:babe

This shows that every single HTTP request utilizes a completely different, globally routable IPv6 address generated from your subnet block. To the target server, these look like entirely distinct users connecting from across the internet.

Perfect! You have successfully built a self-healing, infinitely rotating proxy pool without handing over your budget for metered residential bandwidth.

The Illusion of Infinity: Critical Limitations of IPv6 Subnet Rotation

At this point, you may think you have found a solution to all of your budgeting problems for scraping at scale. But before you tear down your commercial proxy infrastructure, you must understand that a $5/Mo VPS and an open-source rotation daemon are not a universal silver bullet. If it were that simple, the commercial proxy industry would not exist.

This architecture has the following main limitation:

• The IPv4 compatibility wall: This entire architecture is built on one absolute prerequisite: Your target endpoint must support IPv6. If you are scraping legacy enterprise systems or platforms that haven’t migrated to dual-stack networking, this setup is useless. You cannot route an IPv6 packet to an IPv4-only server.

• Subnet-level bans (/64 prefix blocking): Enterprise WAFs are fully aware of IPv6 prefix delegation standards. They know that hosting providers allocate a /64 subnet to a single client. If their heuristics detect highly concurrent behavioral patterns (like missing browser fingerprints or anomalous TLS handshakes) originating from 2a05:f480...:1a2b, they will ban the entire /64 CIDR block. Once your /64 prefix is banned, all 18 quintillion of your “infinite” IPs are simultaneously dead. To recover, you must physically destroy the VPS and provision a new one in a different IP range.

• ASN reputation: No matter how many IPs you rotate, your traffic still originates from a Datacenter Autonomous System Number (ASN). Target firewalls assign a baseline trust score to every ASN. Traffic originating from a Datacenter ASN always starts with a highly degraded trust score compared to a Residential ASN. For highly restrictive targets, any request from a datacenter IP is instantly met with an unpassable CAPTCHA or a hard 403 Forbidden, regardless of whether it’s IPv4 or IPv6.

• nf_conntrack and hardware exhaustion: You cannot push enterprise-grade throughput on a $5, 1-vCPU server without consequence. Rotating thousands of IPv6 addresses requires the Linux kernel to aggressively maintain the nf_conntrack table and the NDP proxy table. At high concurrencies, the overhead of establishing, tracking, and tearing down thousands of TCP sockets across rotating interfaces will exhaust the memory or CPU of a low-tier VPS. The kernel will begin dropping packets natively, your latency will spike to useless levels, and your scrapers will be greeted with errors.

Conclusion

In this article, you learned how to leverage your hosting provider’s IPv6 /64 subnets to build an infinitely rotating proxy pool with NyxProxy, escaping the metered billing of residential proxy networks.

The competitive advantage of engineering your own proxy infrastructure is in your unit economics and architectural control. However, you also learned that this solution is not a universal silver bullet for every scraping scenario: It comes with trade-offs and constraints.

So, let us know: Have you already experimented with bare-metal IPv6 rotation for your scraping pipelines? What targets did it work best for? Let’s discuss in the comments!

Behind the scenes, it relies on hydration to make server-rendered pages interactive. And here’s the interesting part: the same mechanism that makes Next.js fast and popular also exposes a significant amount of structured data in the HTML sent by the server. From a scraping perspective, that’s a huge opportunity!

In this post, I’ll show you a simple trick to scrape data from virtually any Next.js website. Follow along as I break down how it works and how you can apply it yourself.

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

Next.js in Numbers

Next.js needs no introduction, but it’s worth giving some context to truly understand how popular it is (and therefore how useful the trick I’m about to present for Next.js web scraping can be):

• According to the 2025 Stack Overflow Developer Survey, 20.8% of respondents used Next.js extensively over the past year.

• Next.js is the 14th largest repository on GitHub, with over 138k stars (and still growing!).

• According to W3Techs, Next.js has a 2.9% market share among JavaScript libraries.

• Major brands such as Nike, Stripe, and Notion have chosen this full-stack framework to build their official websites.

Before Getting Started: A Bit of Context on Hydration

I know you probably just want the trick… Still, let me take a minute to explain why it works in the first place, why it’s even possible, and what kind of data you’ll actually retrieve with it!

Start your scraping journey with Byteful: 10GB New Customer Trial | Use TWSC for 15% OFF | $1.75/GB Residential Data | ISP Proxies in 15+ Countries

Claim your 10GB here

What Is Hydration?

Hydration is the process that makes a server-rendered page interactive in the browser.

Frameworks like Next.js, Remix, Nuxt, and SvelteKit employ this mechanism to combine the performance benefits of server-side rendering (SSR) with the interactivity of client-side applications.

The idea is that the server first sends fully rendered static HTML to the browser. Then, hydration happens next.

The browser downloads the JavaScript bundle, and the frontend framework reconstructs the component tree in memory, attaches event listeners, and links that virtual tree to the existing DOM instead of re-rendering it from scratch. The result is a fully interactive application built on top of server-rendered HTML.

How Does the Hydration Mechanism Work?

It’s now clear that in Next.js and similar frameworks, hydration is the process where a static, server-rendered HTML page “comes to life” and becomes fully interactive in the browser. But what’s actually happening under the hood?

At a high level, hydration is a 3-step process:

1. The server generates and sends a fully rendered HTML snapshot. The user immediately sees the content (great for First Contentful Paint). At this point, though, the page is just static HTML. Buttons, forms, and other interactive elements are visible, but they don’t work yet because no JavaScript is attached.

2. The client’s browser downloads the JavaScript bundle (which includes React and your frontend application code) and executes it.

3. React rebuilds the component tree in memory and attaches event listeners to the existing DOM nodes. Instead of discarding the HTML and re-rendering everything from scratch, React “hydrates” the existing markup, meaning it reuses it and wires it up with state and interactivity.

Once hydration completes, the page behaves like a normal single-page application: it responds to clicks, manages state, and updates dynamically.

And here’s an important detail: if the browser doesn’t support JavaScript (or it fails to load), the user still sees the server-rendered HTML. It won’t be interactive, but the core content is there. That’s great for SEO and perceived performance!

Why It Matters for Scraping Next.js (and Other Full-Stack Frameworks…)

The key insight you need to understand is simple: hydration requires data, and that data must be embedded somewhere in the HTML sent by the server!

In Next.js, when the server renders a page, it doesn’t only send markup. It also serializes the data required to rebuild the React component tree on the client. That serialized payload is embedded directly into the page’s HTML.

That’s exactly why hydration matters for scraping. Instead of parsing the DOM or simulating user interactions through browser automation, you can extract the structured data that React itself uses to hydrate the page.

In many cases, hydration data is cleaner and easier to parse than the rendered HTML. It can also contain more information than what’s visibly displayed on the page, including hidden and interesting metadata.

Keep in mind that this principle applies not only to Next.js! All other full-stack frameworks that rely on hydration, such as Remix, Nuxt, Angular Universal, and SvelteKit, tend to dehydrate state on the server and rehydrate it on the client.

So remember this simple rule. If a framework hydrates, it must serialize data. And if it serializes data into the HTML, you can scrape it.

How to Scrape Next.js Websites: 2 Approaches

The approach to scraping Next.js by targeting hydration data depends on how that data is embedded in the HTML generated on the server side.

I won’t go too deep into framework internals here (if you’re a Next.js dev, you already know things shift depending on whether you’re using the App Router or the Pages Router), but there are essentially two scenarios you’ll run into.

In this section, I’ll walk through both of them and show you exactly how I retrieve data from each!

Check the TWSC YouTube Channel

Approach #1: Target the __NEXT_DATA__ Script

As a target site, I’ll use a Nike product page as a reference:

That’s actually a great example because Nike.com is even showcased on the Next.js homepage as a real-world site built with the framework.

Now, right-click on the page and select the “Inspect” option in your browser to open the DevTools. Scroll through the DOM and get familiar with the page structure. If the Next.js site is using the Pages Router, you’ll notice a <script> tag with the id __NEXT_DATA__ containing a large JSON blob:

That JSON data is precisely the hydration data I was referring to earlier.

When a site uses the Pages Router approach in Next.js, the server embeds all the page data directly into that <script> tag. From a scraping perspective, that’s gold, as the data is already structured and ready to be captured.

Below’s a simple JavaScript snippet to extract it:

const hydartionScript = document.querySelector("#__NEXT_DATA__")
const hydrationData = JSON.parse(hydartionScript.innerHTML)
console.log(hydrationData)

What’s happening here is straightforward. The JS script:

• Selects the <script> element with id __NEXT_DATA__.

• Reads its inner HTML (which is a JSON string).

• Parses it into a JavaScript object.

• Logs it to the console.

Run this directly in the DevTools Console, and you’ll immediately see the result:

What’s interesting is how much structured data you get right away. This includes product details, images, metadata, and more. All is neatly organized, and it only took three lines of code!

If you want to store the JSON hydration object, just right-click the object in the Console and select the “Copy object” option:

From there, you can paste it wherever you need (e.g., into a local .json file, a MongoDB collection, etc.).

Need help with your scraping project?

Approach #2: Target the self.__next_f.push() Elements

Another, more complex approach to scraping Next.js involves pages built with the App Router.

Even if the App Router has been the recommended direction for a while, in my experience, it’s still not as widely adopted as the Pages Router. And honestly, that’s a bit of a gift for us (as scraping hydration data in App Router sites is definitely more complex!)

As a reference, let’s look at the “Business Overview” page on the OpenAI website, which is built with Next.js App Router:

Just like before, open DevTools and inspect the page. This time, focus on the <script> tags inside the <body>:

You’ll notice several scripts containing content like:

self.__next_f.push(<some_data>)

That “<some_data>” is serialized using the React Flight protocol for React Server Components (RSC). I won’t go too deep into the internals here (it’s a dense topic!), but what matters is that deserializing that data is not straightforward!

React Flight isn’t plain JSON. It mixes control records (HL, I, J, etc.), module references, streaming boundaries, and serialized model fragments into a transport format that React incrementally resolves at runtime.

You might think: “Why not just reuse the frontend deserialization library?” In practice, that doesn’t work well because:

• The client decoder (react-server-dom-webpack) expects a full React runtime.

• It relies on module maps and webpack IDs generated at build time.

• It resolves component references against the exact bundle that produced the stream.

• It assumes streaming semantics and internal React wiring.

Basically, outside that exact environment, you don’t have the module graph, build manifest, or hydration context. So even if you import the decoder, you can’t reconstruct the component tree the way the browser does.

There have been recent security issues in the React Flight payload deserialization system, highlighting just how sensitive and complex this layer is. For more details, refer to:

• Security Advisory: CVE-2025-66478

• Critical Security Vulnerability in React Server Components

Thus, instead of fighting the protocol, I’d simplify and accept that in this case, it’s better to extract the unparsed React Flight string data. Achieve that with the JS script below:

const nextFlightScripts = [...document.querySelectorAll("script")]
  .filter(script => script.textContent.includes("self.__next_f"))
  .map(script => script.textContent.trim())
console.log(nextFlightScripts)

This selects all <script> elements containing “self.__next_f” and builds an array of their raw contents.

Run it in the Console, and you’ll get an array of React Flight chunks:

From there, the simplest way to extract structured data is often to copy the array, feed it to an AI, and ask it to reconstruct a parsed JSON representation of the meaningful payload sections:

Is this more complicated than the __NEXT_DATA__ trick? Absolutely! Yet, it’s still a powerful way to access a large amount of page data with just a few lines of code.

Final Script to Quickly Access Data From Next.js Sites

If you combine the two approaches, you can build a production-ready script for brute-force hydration data scraping in Next.js:

// Pages Router approach (__NEXT_DATA__)
const hydrationScript = document.querySelector("#__NEXT_DATA__")
let nextData = null
if (hydrationScript) {
  try {
    nextData = JSON.parse(hydrationScript.textContent)
    console.log("__NEXT_DATA__ found:")
    console.log(nextData)
  } catch (err) {
    console.warn("Failed to parse __NEXT_DATA__:", err)
  }
} else {
  console.log("No __NEXT_DATA__ script found.")
}

// App Router approach (self.__next_f)
const nextFlightScripts = [...document.querySelectorAll("script")]
  .map(script => script.textContent.trim())
  .filter(content => content.includes("self.__next_f.push"))

if (nextFlightScripts.length > 0) {
  console.log("React Flight scripts found:")
  console.log(nextFlightScripts)
} else {
  console.log("No React Flight scripts found.")
}

To test it, just open the Console in DevTools, paste the script, and run it.

Important: The <script> components containing hydration data aren’t loaded dynamically via client-side rendering. They’re embedded directly in the HTML generated by the server.

That means you can:

1. Fetch the target Next.js-powered page with an HTTP client.

2. Parse the HTML using an HTML parsing library like Beautiful Soup or Cheerio.

3. Apply a similar version of the JavaScript script above, but adapt it to the API provided by your HTML parser.

In other words, this trick for scraping Next.js doesn’t only work in the browser DevTools. It also works perfectly in regular scraping scripts!

Pros and Cons of This Approach to Next.js Scraping

👍 Pros:

• Simple and effective, requiring only a few lines of code.

• Works on all Next.js websites (and, more generally, on most sites that rely on hydration).

• Can let you access more data than what is actually displayed on the page.

• No need for browser automation, waiting for client-side rendering, or simulating user interactions.

👎 Cons:

• You may only get partial data, meaning you might still need to complement it with a more traditional scraping approach.

• React Flight data is difficult to parse and may require custom logic or even AI-assisted parsing.

Conclusion

Here, I’ve shared a trick I personally documented years ago, and that still works to this day. It allows you to quickly scrape data from virtually any Next.js site by targeting the hydration data embedded in the HTML document generated by the server and sent to the client for rendering.

As you’ve seen, with just a few lines of JavaScript, you can extract hydration data from any Next.js-powered page. What you get back is clean, or at least almost clean, data that you can process directly in your data pipelines.

Instead of fighting the frontend, this Next.js web scraping approach helps you leverage the data the framework itself needs to function!

I hope you found this useful and insightful. If you have questions or thoughts, feel free to share them in the comments below!

But what is Polymarket, and how does it work?

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

What Polymarket is, and why it keeps making headlines

Polymarket is a prediction market, a platform where you buy and sell shares tied to the outcome of real-world events. If the event happens, your share pays $1. If it doesn’t, it pays $0. The trading price at any moment reflects what the market collectively believes the probability of that outcome is. You can bet on elections, geopolitics, sports, crypto prices, and increasingly anything else with a verifiable resolution.

It is the largest prediction market by volume, built on the Polygon blockchain. Its main competitor, Kalshi, operates as a CFTC-regulated exchange in the US. Both are attracting billions in volume, and Wall Street firms are now building dedicated trading desks around them.

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

The platform handled at least $7.5 billion in volume during 2025 (a conservative figure, since Polymarket volume is commonly double-counted due to how OrderFilled events are summed, and set a single-day record of $425 million in February 2026 when Iran-related markets resolved simultaneously. Those are not toy numbers. And with that kind of money flowing through, the headlines have followed.

In January 2026, a newly created Polymarket account invested $30,000 and walked away with $436,759 after correctly betting on Maduro’s removal from power. The account was created less than a week before the U.S. military operation, and the bulk of bids were placed hours before Trump’s announcement. In a separate case, an Israeli Air Force reservist was indicted for leaking classified details about a strike on Iran to guide Polymarket bets, netting roughly $244,000. A different trader has made nearly $1 million since 2024 from dozens of well-timed bets correctly predicting U.S. and Israeli military actions against Iran, winning 93% of five-figure wagers. These incidents triggered at least eight prediction market bills in Congress since January 2026, and federal prosecutors in Manhattan are actively exploring whether certain prediction market bets violate insider trading laws.

But insider trading is not the only way people make money on Polymarket. There is a quieter, more interesting story happening in parallel.

Start your scraping journey with Byteful: 10GB New Customer Trial | Use TWSC for 15% OFF | $1.75/GB Residential Data | ISP Proxies in 15+ Countries

Claim your 10GB here

The efficiency gap

The Efficient Market Hypothesis, formalized by Eugene Fama in the 1960s, states that asset prices reflect all available information, making it impossible to consistently beat the market. In traditional equity markets, this largely holds because massive institutional capital from hedge funds, pension funds, and proprietary trading firms constantly hunts for and eliminates mispricings. The S&P 500 trades roughly $500 billion daily. Any pricing error gets corrected in milliseconds by algorithms running in colocated data centers.

Polymarket’s individual markets often have only tens of thousands of dollars in liquidity. The ratio of “smart money” to “total market cap” is fundamentally different from equity markets, and that is why edges persist longer than they would on Wall Street. A 2025 study by IMDEA Networks Institute documented $40 million in arbitrage profits extracted from Polymarket alone between April 2024 and April 2025, analyzing 86 million bets. Arbitrage opportunities on the platform last an average of 4 seconds, with 73% of profits captured by bots, executing in under 100 milliseconds.

The institutional side is catching up. DRW is hiring dedicated prediction market traders at a $200,000 base salary. Susquehanna International Group became the first official market maker on Kalshi (a competing platform). Jump Trading is building specialized desks. But the market is not there yet. Liquidity is too thin for these firms to deploy serious capital without moving prices, leaving room for smaller, faster actors.

Check the TWSC YouTube Channel

Speed as edge: what people are building

I’ve been studying Polymarket for some months now, and I’ve probably ended up on a bubble on Instagram and other social media. I’m seeing a growing number of traders share systems designed to exploit exactly this kind of market inefficiency. The approaches vary, but the pattern is the same: faster information, faster execution, profit.

One notable case involves a trader who claims to use computer vision models processing live football match video feeds. His system watches the match in real time, detects events (goals, red cards, penalties) through frame analysis, and places bets on prediction markets seconds before the event registers on official data feeds and bookmaker odds adjust. He claims an 8-second advantage over other traders (unfortunately, I cannot find the post on Instagram about it anymore). Whether that specific claim holds up or not, this is nothing new: courtsiders have been doing this in tennis for years, attending live matches and transmitting scores faster than official data feeds reach bookmakers. In 2016, tennis umpires from Kazakhstan, Turkey, and Ukraine were banned for deliberately delaying score updates for courtside accomplices.

The same principle applies at a larger scale. Bill Benter built a multinomial logit model with over 120 variables per horse for Hong Kong racing and extracted over $1 billion between 1987 and 2001. Tony Bloom’s Starlizard syndicate employs 160 people to model Asian handicap football markets and reportedly generates 600 million GBP per year. On Polymarket itself, 14 of the top 20 most profitable wallets are bots. One bot turned $313 into $414,000 in a single month, exploiting temporal arbitrage in 15-minute crypto markets.

All of these systems share two requirements: data and speed. They need real-time access to market prices, order books, and event outcomes, and they need to act on that data faster than everyone else. All of this is possible because Polymarket provides a full set of APIs that can be used to operate programmatically on the platform.

Need help with your scraping project?

Polymarket’s API architecture

Polymarket exposes three distinct APIs, each serving a different purpose. Understanding which one to use and when is the first step toward building anything that trades or monitors this market.

Gamma API: market discovery

The Gamma API is the browsing layer. It returns human-readable market data: questions, descriptions, outcome prices, volume, liquidity, event metadata. No authentication required.

Base URL: https://gamma-api.polymarket.com

Key endpoints:

- GET /markets returns a paginated list of markets with filtering options (limit, offset, closed, tag_id)

- GET /markets/{id} returns a single market by ID or slug

- GET /events and GET /events/{id} return event-level data (events group related markets)

- GET /search?query=... performs keyword search across markets and events

A single call to /markets?limit=1&closed=false returns something like this:

{
  "id": "540816",
  "question": "Russia-Ukraine Ceasefire before GTA VI?",
  "conditionId": "0x9c1a953fe92c8357f1b646ba25d983aa83e90c525992db14fb726fa895cb5763",
  "outcomes": "[\"Yes\", \"No\"]",
  "outcomePrices": "[\"0.545\", \"0.455\"]",
  "volume": "1516211.89",
  "liquidity": "62104.61",
  "clobTokenIds": "[\"850149715...\", \"252731249...\"]"
}

The `clobTokenIds` field is the bridge to the trading layer. Each outcome (Yes/No) gets its own token ID, which is what you pass to the CLOB API to get real-time prices and order book data.

The Gamma API is rate-limited at roughly 60 requests per minute. It is useful for discovery and metadata, not for real-time price monitoring.

CLOB API: the order book

The CLOB (Central Limit Order Book) API is where trading happens. It has both public and authenticated endpoints.

Base URL: https://clob.polymarket.com

Public endpoints (no authentication):

- GET /price?token_id=X&side=BUY|SELL returns the current best price

- GET /midpoint?token_id=X returns the midpoint between best bid and ask

- GET /spread?token_id=X returns the current spread

- GET /book?token_id=X returns the full order book with all bids and asks

- GET /last-trade-price?token_id=X returns the last executed trade price

- GET /tick-size?token_id=X returns the minimum price increment

A call to /midpoint returns a minimal payload:

{"mid": "0.545"}

The /book endpoint returns the full depth:

{
  "market": "0x9c1a95...",
  "asset_id": "850149715...",
  "bids": [
    {"price": "0.54", "size": "15234.50"},
    {"price": "0.53", "size": "8920.00"}
  ],
  "asks": [
    {"price": "0.55", "size": "12100.00"},
    {"price": "0.56", "size": "6500.00"}
  ]
}

These public endpoints are what matters for price monitoring. They are lightweight, return small payloads, and have no authentication overhead.

Authenticated endpoints require a two-level authentication system:

Level 1 (L1) uses EIP-712 wallet signatures. You sign a structured message proving you control a specific Ethereum wallet address. This is a one-time operation that generates API credentials:

POST /auth/api-key
Headers: POLY_ADDRESS, POLY_SIGNATURE, POLY_TIMESTAMP, POLY_NONCE
Returns: { apiKey, secret, passphrase }

Level 2 (L2) uses HMAC-SHA256 signing on every request. Every authenticated call requires five headers: POLY_ADDRESS, POLY_SIGNATURE (computed HMAC of the request), POLY_TIMESTAMP, POLY_API_KEY, and POLY_PASSPHRASE.

Even with L2 auth, placing an order requires the user to sign the order payload locally with their private key. Three cryptographic operations total: key derivation (once), request signing (per call), order signing (per order).

The authenticated endpoints are:

- POST /order places a single order

- POST /orders places a batch of orders

- DELETE /order cancels an order

WebSocket feeds provide real-time streaming at wss://ws-subscriptions-clob.polymarket.com/ws/ for order book updates, price changes, and user-specific events.

Data API: analytics

The Data API at https://data-api.polymarket.com provides analytics-oriented data: user positions, trade history, leaderboards, and holder information. It is less documented and less stable than the other two. Some endpoints returned 404 or empty responses during our testing. Useful for research, not reliable for production.

The speed game: calling the APIs as fast as possible

If arbitrage opportunities on Polymarket last 4 seconds on average, and 73% of profits go to bots executing in under 100 milliseconds, then the speed at which you can read prices and place orders is a direct competitive advantage. We set up a benchmark to answer two questions: where should you run your code, and which language and HTTP strategy gets you there fastest?

We did our tests and chose the /midpoint endpoint for the benchmark because it requires no authentication, returns the smallest possible payload, and isolates HTTP client performance from payload parsing. Each benchmark runs 1,000 requests in two modes: sequential (one request at a time, measuring per-request latency) and concurrent (50 simultaneous workers, measuring throughput).

As always, the code can be found in our GitHub repository reserved to paying users, inside the folder 102.POLYMARKET.

Read more

In particular, your scrapers leak information at four distinct layer levels. Modern anti-bot systems, in fact, fingerprint your browser, analyze your TLS handshake, trace your network infrastructure, and track your behavioral patterns. And a single inconsistency across these layers triggers permanent blocking.

This means your scrapers aren’t competing only against rate limits anymore. Today, they are competing against machine learning models trained on billions of legitimate requests, and any deviation from the expected pattern is a signal. So, if you want to scrape at scale, your infrastructure must be indistinguishable from a real user’s browser, network stack, and behavior.

This article guides you through a systematic approach: First, understanding where leaks occur, then learning how anti-bot systems detect them, and finally building a layered defense that makes your scraper invisible.

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

Identifying the Leaks: Where Your Scraper Exposes Itself

Before fixing anything, you need to understand the complete attack surface. Modern anti-bot systems analyze your scraper at four distinct layers, and a leak at any layer can expose you.

Layer 1: The Browser Level

Headless browsers are loud by default. Launch a Puppeteer instance and check the navigator.webdriver flag. It surely returns true, and that’s a signal every major anti-bot system checks in the first 100ms of page load.

But this obvious flag is just the beginning. Anti-bot systems probe deeper:

• Error messages and stack traces: They differ between headless and headed modes. The execution context leaves fingerprints in error objects.

• Window dimensions: Properties like window.outerWidth and window.outerHeight reveal a headless operation because headless mode doesn’t render a visible window frame.

• Canvas rendering: They can produce pixel-level differences. Software rendering (headless) creates different anti-aliasing and color values than GPU-accelerated rendering (headed). Color channels can differ by 1-2 units per pixel.

• WebGL shader timing: This can vary a lot, depending on the underlying technology. GPU-accelerated browsers complete WebGL operations in microseconds. Software-rendered headless browsers take milliseconds.

• Font rendering: Headless environments often lack the full system font stack. This creates detectable layout differences when JavaScript measures text dimensions.

• Performance benchmarks: When run, they can reveal software rendering. For example, there are websites that run JavaScript stress tests, creating thousands of DOM elements, calculating layouts, and triggering reflows. In such scenarios, real browsers with GPU acceleration show consistent performance. Headless browsers, instead, show different timing patterns.

• The window.chrome object behaves differently: Real Chrome populates this object with specific properties for extension management and runtime APIs. Headless Chrome, instead, either lacks this object or provides an incomplete implementation.

Layer 2: The Network Level

Your SSL/TLS handshake identifies you before you send any application data. When your scraper connects over HTTPS, it sends a TLS Client Hello message containing supported encryption methods, protocol versions, and extensions. All in a specific order.

Here’s what makes this dangerous:

• Every browser and HTTP library has a unique TLS pattern: Real browsers send their TLS parameters in a specific sequence that matches their version and underlying platform. Python’s standard HTTP libraries send a completely different pattern. So do Node.js, Go, and any other programming language you use for coding your scrapers.

• Anti-bot systems fingerprint your TLS handshake: They capture these patterns and convert them into a fingerprint, commonly called a JA3 hash. They maintain databases of known fingerprints for every major browser and HTTP library.

• Mismatches between User-Agent and TLS fingerprint are instant red flags: When you claim to be Chrome in your User-Agent header but your TLS handshake matches Python’s urllib library, that inconsistency triggers blocking.

• Detection happens before you send any application data: The first TCP connection already identifies you as automated traffic.

• HTTP/2 fingerprinting adds another layer: Beyond TLS, the order and priority of HTTP/2 frames, settings, and window updates create additional fingerprints. Your HTTP library’s frame ordering must match your claimed browser identity.

For your scraping needs, having a reliable proxy provider like Decodo with high reputatation IPs, on your side improves the chances of success.

Try Decodo Now

Layer 3: The Infrastructure Level

Your proxy configuration can expose your real infrastructure through network-level leaks via the following main mechanisms:

• DNS leaks: They happen when your browser resolves domain names using your local DNS server instead of routing through the proxy. Your scraper might send requests through a Miami residential proxy, but if DNS queries go through your AWS datacenter in Virginia, the target site knows your real location.

• WebRTC leaks: WebRTC is a browser API designed for peer-to-peer communication. Even with a proxy configured, WebRTC will attempt to discover your real local IP and public IP through STUN servers, completely bypassing your proxy.

• IP reputation: Not all IPs are created equal. Cloudflare and similar services maintain databases of every AWS, Google Cloud, and Azure IP range. Requests from known cloud providers receive instant higher suspicion scores before any other analysis happens.

Layer 4: The Behavioral Level

Even if your browser, network, and infrastructure are perfectly disguised, your behavior patterns can still expose you:

• Timing patterns: Requesting data at fixed and precise intervals creates a perfect periodicity. No human browses with mathematical precision.

• Mouse and scroll behavior: Real humans accelerate and decelerate smoothly. Instant jumps from point A to point B are mechanically impossible.

• Session state: Stateless scrapers that never accumulate cookies or maintain persistent sessions across days look like fresh bots on every run.

• Interaction sequences: The time between page load and first click, between mouse-over and click, or the pattern of how you scroll through content. They all follow detectable human patterns.

Understanding the Detection: How Anti-Bot Systems Catch You

Now that you know where leaks occur, let’s understand how anti-bot systems actually detect them.

Fingerprint Consistency Checks

Anti-bot systems cross-reference your claimed identity with actual behavior. If your User-Agent says “Chrome 120 on Windows 10,” they verify that your JavaScript features, WebGL capabilities, canvas rendering, and TLS handshake all match Chrome 120 on Windows 10.

A single mismatch anywhere flags the entire request. You can’t be Chrome in your User-Agent, Firefox in your TLS handshake, and headless Chrome in your canvas fingerprint. Anti-bot systems create composite fingerprints combining dozens of properties, then compare them against databases of known legitimate and bot patterns.

Machine Learning Pattern Recognition

Modern anti-bot systems use ML models trained on billions of requests. They learn what “normal” looks like for each type of visitor. This means that consumer browsers from residential IPs have different behavioral patterns than datacenter scrapers.

For ML models, statistical anomalies trigger investigation. Perfect timing intervals, impossible mouse movements, or timing patterns that don’t match human variance distributions are scored as anomalous. These models adapt continuously, so when new stealth techniques emerge, the models retrain on that data. This means that what works today might fail tomorrow.

Progressive Trust Scoring

Anti-bot systems block or allow requests, but they also score. This means that lower trust scores receive degraded service: slower response times, rate limits, or CAPTCHA challenges before blocking.

Also, scores accumulate across sessions. If you leak information across multiple visits, the system builds a profile associating your various identities. In other words, one leak can poison future requests, and even fixing the leak might not restore trust if your IP or fingerprint is already marked.

Check the TWSC YouTube Channel

Building the Defense: A Layered Approach to Stealth

Building a defense from data leaks in web scraping requires addressing each layer systematically. Your stealth stack must work from the inside out: browser → network → infrastructure → behavior. Each layer must remain consistent with your claimed identity.

Defense Layer 1: Hardening the Browser

The goal at this layer is to make the browser fingerprint indistinguishable from a real user’s browser and ensure every property is consistent with your claimed identity.

Step 1: Mask Automation Signals

Start with stealth libraries that patch the most common detection vectors:

• For Puppeteer: Use puppeteer-extra-plugin-stealth to automatically override navigator.webdriver, DevTools Protocol signatures, and plugin arrays.

• For Selenium: Use undetected-chromedriver, which patches automation signals and uses real Chrome binaries instead of ChromeDriver.

• For Playwright: Leverage native evasion features that handle many detection vectors out of the box.

Additionally, disable automation flags at launch. For example, in Playwright:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(
        headless=True,
        args=[
            '--disable-blink-features=AutomationControlled',
            '--disable-dev-shm-usage',
            '--no-sandbox'
        ]
    )

But remember: Stealth libraries handle the most common 20-30 leak vectors but miss advanced fingerprinting techniques. They’re your foundation, not your complete solution.

Step 2: Spoof Hardware Signatures

Cloud server canvas and WebGL fingerprints are obvious red flags. AWS, GCP, and Azure rendering signatures are well-known to anti-bot systems.

You have two approaches for your defense here:

• Add consistent noise: Inject deterministic noise into canvas operations so the fingerprint remains stable across sessions but doesn’t match your server’s real hardware. Override canvas methods to modify pixel data slightly before it’s read back. Keep noise minimal: just enough to mask the real hardware signature without appearing obviously manipulated.

• Emulate common consumer hardware: Spoof WebGL parameters to mimic common consumer GPUs. Override vendor and renderer strings returned by WebGL APIs to match your chosen hardware profile. Use existing libraries designed for canvas fingerprint defense or implement your own parameter overrides.

Step 3: Ensure Version Consistency

This is where most scrapers fail, even with stealth libraries. Your User-Agent string must match your actual browser engine behavior precisely. Consider the following rules of thumb:

• Use real browser binaries instead of spoofing: Tools like Playwright can launch actual Chrome, ensuring perfect consistency between claimed version and actual behavior.

• If you must spoof, maintain complete version profiles: Track which JavaScript features, WebGL capabilities, and API behaviors correspond to each browser version. Every property must align.

• Never mix components from different versions: If you claim Chrome 120 on Windows 10, every single API, from JavaScript features to WebGL renderers, must behave exactly like Chrome 120 on Windows 10.

Defense Layer 2: Hardening the Network Stack

Your goal at this layer is to make your TLS handshake and HTTP traffic indistinguishable from the browser you’re claiming to be.

Step 4: Match TLS Fingerprints to Your Browser Identity

Standard HTTP libraries can’t mimic browser TLS fingerprints because they use different SSL/TLS implementations. The solution requires specialized libraries that replicate browser behavior at the protocol level:

• For Python: Use curl_cffi or similar wrappers. These libraries use libcurl compiled with BoringSSL, which is the same SSL library Chrome uses. This creates identical JA3 fingerprints to real browsers.

• For Node.js: Use cycletls or equivalent libraries that allow you to specify exact JA3 fingerprint strings matching real browsers.

Critical requirement: Your TLS fingerprint must match your User-Agent. Chrome 120’s JA3 fingerprint is different from Firefox 115’s fingerprint. The browser identity must be consistent across all layers.

Step 5: Match HTTP/2 Fingerprints

Beyond TLS, HTTP/2 frame ordering creates additional fingerprints. Libraries like curl_cffi handle this automatically when you specify a browser to impersonate, but verify that:

• Settings frames match your target browser.

• Window update sequences align.

• Priority headers follow the correct pattern.

In Python, you can do so with the following code:

response = requests.get(
    '<https://tls.peet.ws/api/all>',
    impersonate='chrome120'
)
print(response.json()['http2']['sent_frames'])

Need help with your scraping project?

Defense Layer 3: Hardening Infrastructure

Your goal at this layer is to ensure your network traffic originates from legitimate-looking IPs and doesn’t leak your real location or identity.

Step 6: Choose the Right Proxy Type

IP reputation is the first filter that anti-bot systems check. This means that your proxy choice determines your baseline trust score. Consider the following guidelines:

• Datacenter IPs = instant red flag: Requests from AWS, Google Cloud, and Azure IP ranges receive instant higher suspicion scores.

• Residential proxies = highest legitimacy: These IPs come from real ISP connections, so they look legitimate because they are legitimate consumer connections.

• Mobile proxies = premium legitimacy: These IPs originate from cellular networks (4G/5G) and receive the highest trust scores. Mobile IPs rotate naturally as devices move between cell towers, making them appear even more organic than static residential connections.

Step 7: Prevent DNS Leaks

Force all DNS resolution through your proxy tunnel. For SOCKS5 proxies, use the SOCKS5h protocol variant, which forces DNS resolution on the remote proxy server instead of locally.

For example, in Python, write the following:

import requests

proxies = {
    'http': 'socks5h://proxy.example.com:1080',
    'https': 'socks5h://proxy.example.com:1080'
}

response = requests.get('<https://example.com>', proxies=proxies)

For browser automation, configure DNS-over-HTTPS to prevent local DNS leakage. The following is an example that applies to Playwright:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(
        args=[
            '--dns-over-https-server=https://cloudflare-dns.com/dns-query'
        ]
    )

Step 8: Disable WebRTC Completely

WebRTC will expose your real IP unless you completely disable it in browser automation. For example, in Playwright, you can do so as follows:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    
    # Remove WebRTC entirely
    await page.add_init_script("""
        delete window.RTCPeerConnection;
        delete window.RTCSessionDescription;
        delete window.RTCIceCandidate;
        delete navigator.mediaDevices;
    """)

When youìve done this, verify it’s actually disabled before deploying your scraper. Visit browserleaks.com/webrtc with your scraper. You should see “WebRTC is not supported by your browser”, or only your proxy IP should be visible. Never your real IP.

Defense Layer 4: Mimicking Human Behavior

Your goal at this layer is to make your interaction patterns indistinguishable from those of real human users.

Step 9: Add Timing Jitter and Randomization

Humans are inconsistent. Perfect patterns are robotic. The solution here is not to just add randomness. You also need to match the statistical distribution of real human behavior. To do so, consider the following example in Python:

import numpy as np
import time

# Wrong example (do not use this)

# Fixed interval
time.sleep(5)  # Always 5 seconds - DETECTABLE

# Random uniform
time.sleep(random.uniform(3, 7))  # Still doesn't match human patterns

------------

# Correct example (use this!)

# Log-normal distribution (matches real human reaction times)
delay = np.random.lognormal(mean=1.5, sigma=0.5)
time.sleep(delay)

For improving randomization, model different action types with appropriate distributions. Use the following rules of thumb:

• Clicks: 0.3-2 seconds (short delays)

• Reading: 5-45 seconds (high variance)

• Scrolling: 1-8 seconds (irregular intervals)

Step 10: Implement Realistic Mouse and Scroll Behavior

High-security sites like banking, ticketing, and heavily protected e-commerce websites track interaction patterns in real-time. To defend from leaking your information on such websites, you have to define mouse movements and scrolling for your automated scripts.

For mouse movements, you can:

• Use Bezier curves to create natural arcing movements between points.

• Add slight randomness to destination coordinates.

• Include hover delays before clicking.

• Vary the number of intermediate steps based on distance.

The following is an example you can try in Python:

import numpy as np
from playwright.sync_api import sync_playwright

def bezier_curve(start, end, control_points, num_steps=20):
    """Generate points along a Bezier curve for natural mouse movement"""
    t = np.linspace(0, 1, num_steps)
    points = []
    
    # Simplified cubic Bezier
    for t_val in t:
        x = (1-t_val)**3 * start[0] + \\
            3*(1-t_val)**2*t_val * control_points[0][0] + \\
            3*(1-t_val)*t_val**2 * control_points[1][0] + \\
            t_val**3 * end[0]
        y = (1-t_val)**3 * start[1] + \\
            3*(1-t_val)**2*t_val * control_points[0][1] + \\
            3*(1-t_val)*t_val**2 * control_points[1][1] + \\
            t_val**3 * end[1]
        points.append((x, y))
    
    return points

async def human_like_click(page, selector):
    element = await page.query_selector(selector)
    box = await element.bounding_box()
    
    # Add slight randomness to destination
    target_x = box['x'] + box['width']/2 + np.random.normal(0, 2)
    target_y = box['y'] + box['height']/2 + np.random.normal(0, 2)
    
    # Move mouse along curve
    current_pos = await page.mouse.position()
    control_points = [
        (current_pos['x'] + np.random.uniform(-50, 50), 
         current_pos['y'] + np.random.uniform(-50, 50)),
        (target_x + np.random.uniform(-20, 20), 
         target_y + np.random.uniform(-20, 20))
    ]
    
    points = bezier_curve(
        (current_pos['x'], current_pos['y']), 
        (target_x, target_y), 
        control_points
    )
    
    for x, y in points:
        await page.mouse.move(x, y)
        await page.wait_for_timeout(np.random.uniform(5, 15))
    
    # Hover briefly before clicking
    await page.wait_for_timeout(np.random.uniform(100, 300))
    await page.mouse.click(target_x, target_y)

For scrolling, you can:

• Pause between scroll actions for variable amounts of time (simulating reading).

• Scroll in chunks of varying size, not uniform pixels.

• Occasionally scroll backwards (humans re-read).

• Don’t scroll in perfect increments or at constant speeds.

Use the following Python code to try such scrolling behaviour:

async def human_like_scroll(page, total_distance):
    """Scroll with human-like patterns"""
    scrolled = 0
    
    while scrolled < total_distance:
        # Vary chunk size
        chunk = np.random.randint(100, 400)
        
        await page.mouse.wheel(0, chunk)
        scrolled += chunk
        
        # Pause to simulate reading
        pause = np.random.lognormal(mean=1.2, sigma=0.8)
        await page.wait_for_timeout(pause * 1000)
        
        # Occasionally scroll backwards (humans re-read)
        if np.random.random() < 0.15:
            await page.mouse.wheel(0, -np.random.randint(50, 150))
            await page.wait_for_timeout(np.random.uniform(500, 1500))

Step 10: Maintain Persistent Session State

Stateless scrapers look like stateless bots. Real browsers, instead, accumulate state over time because:

• Cookies persist across requests and sessions.

• LocalStorage accumulates tracking data over time.

• Session IDs remain stable across days or weeks.

To mimic real browser states, you can use the following Python code:

import pickle
import requests

# Save cookies to disk after each session
session = requests.Session()

# ... perform scraping ...

with open('cookies.pkl', 'wb') as f:
    pickle.dump(session.cookies, f)

# Before next scraping session
with open('cookies.pkl', 'rb') as f:
    session.cookies.update(pickle.load(f))

In case you use a browser automation tool:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    
    # Save browser storage state
    context = browser.new_context()
    # ... perform scraping ...
    context.storage_state(path='state.json')
    
    # Reload in next session
    context = browser.new_context(storage_state='state.json')

As a final note, consider keeping sessions alive for weeks to allow third-party tracking cookies to build up. Long-lived sessions with accumulated tracking data appear more legitimate than constantly refreshed clean states.

Conclusion

In this article, you learned that, if you don’t want your data to be leaked while scraping, you have to take several defensive measures, as no single technique makes you invisible. Anti-bot systems analyze multiple signals simultaneously, and any inconsistency across layers triggers detection and blocks your scrapers.

Also, detection methods evolve. So, what works today might fail tomorrow. This means you should also monitor the defenses you implemented and test new ones.

Now, let us know: How do you prevent data leaks in your scrapers? Did we miss some technique?

Rayobyte, the proxy and web scraping solutions provider, has taken a different approach. They recently released rayobrowse, a closed‑source yet Docker‑based, self‑hostable stealth browser built for local browser automation and web scraping.

In this post, I’ll take a deep look at this solution and walk you through everything you need to know about it. By the end, you’ll understand what rayobrowse is, how its stealth browser approach works, how to set it up, and whether it’s actually worth paying attention to.

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

An Introduction to rayobrowse

Let me introduce you to the world of rayobrowse, helping you understand what it is and what makes this project special.

What is rayobrowse?

rayobrowse is a self-hosted, Chromium-based stealth browser engineered for web scraping, AI agents, and automation workflows. It’s available as a Docker image, with optional support via a Python SDK (rayobrowse on PyPI) for simplified connection. The project is developed and maintained by Rayobyte.

The stealth browser runs inside Docker and is available via the Chrome DevTools Protocol (CDP). That means tools like Playwright, Puppeteer, and Selenium (or any other tool that speaks CDP) can natively connect to it for automation purposes.

What makes it noteworthy is its approach to device fingerprinting. User agents, screen size, WebGL, fonts, timezone, and other signals are tuned so each session looks like a real browser. That way, it helps your automation avoid detection on protected websites.

Core Principles Driving the Solution

These are the core principles and goals behind the project:

1. It should run on Linux server environments without GPUs or a GUI/desktop interface.

2. It should patch Chromium at the C++ level, rather than at higher layers like CDP, which are easier for anti-bot systems to detect.

3. It should work with Playwright, a common framework in browsing automation stacks.

4. It should support both headful mode (via Xvfb) and headless mode.

5. It should emulate fingerprints from real-world devices across different regions.

6. It should be self-hostable, so you can run it locally without relying on cloud infrastructure.

7. It should be free to test and use for certain user segments.

8. It should reliably bypass major anti-bot systems and scraping targets, including complex ecommerce and SERP platforms.

Note: If you’re not familiar with Xvfb, that’s an in‑memory display server for Unix-like systems that implements the X11 display protocol without requiring a physical display or input devices. In simpler terms, it allows GUI applications to run in headless environments. rayobrowse relies on it to launch headful browser sessions even on servers without a graphical interface (that’s beneficial as headful sessions are harder to detect than purely headless ones).

Main Features for Stealth Browsing and More

Here is a list of the most relevant rayobrowse features:

• Fingerprint spoofing: Each browser session comes with a real-world realistic device fingerprint drawn from a database of thousands of profiles. Signals include user agent, OS metadata, screen resolution, fonts, WebGL, hardware concurrency, and timezone.

• Human‑like mouse movement: Optional human‑style cursor behavior (inspired by HumanCursor) makes automation appear more natural. When using standard Playwright actions like page.click() or page.mouse.move(), the library applies realistic curves and timing.

• Proxy Integration: Traffic can be routed through any HTTP proxy, including authenticated and rotating proxies.

• Headless and headful Support: rayobrowse supports both execution modes, even on GUI-less Linux servers.

• Live session viewer: A built‑in noVNC interface (available at http://localhost:6080) lets you watch browser sessions in real time directly from the browser. This is particularly useful for debugging scraping flows and visually verifying fingerprint behavior.

• Official integrations: The browser integrates with common automation frameworks, namely Playwright, Puppeteer, Selenium, and Scrapy (via scrapy-playwright), as well as emerging AI‑driven tools such as OpenClaw. As of this writing, additional integrations (e.g., Firecrawl and LangChain) are planned.

• Remote/Cloud mode: rayobrowse can run as a remote browser service. Your server requests new browser instances through a REST API, and workers connect directly to the returned CDP WebSocket endpoint. This is still a beta feature.

• API‑driven browser management: The daemon exposes REST endpoints for creating, listing, and deleting browser sessions, allowing you to orchestrate multiple browsers across a distributed scraping infrastructure.

Technical Details About the Project

Now that you know what the project is and the features it provides, you’re ready to dive into the technical aspects.

How rayobrowse Works

At a high level, rayobrowse follows these steps:

1. Chromium patching: The project tracks upstream Chromium releases and applies a focused set of patches (relying on an approach similar to Brave’s “plaster” model). These patches normalize exposed browser APIs, reduce fingerprint entropy leaks, improve automation compatibility, and preserve native Chromium behavior whenever possible.

2. Fingerprint assignment: When a browser session starts, rayobrowse assigns a realistic device fingerprint.

3. Automation integration: Browser automation libraries connect to rayobrowse through the native CDP.

Architecture

Architecturally, rayobrowse follows a clean separation between the browser runtime and the automation code.

In particular, the system runs as a Docker container that bundles three core components:

1. A daemon server that manages browser sessions.

2. A browser manager that downloads and retrieves the correct version of Chromium, a fingerprint engine that injects realistic device profiles, and a stealth browser layer containing a custom Chromium build with stealth patches.

3. A noVNC viewer, which lets you watch browser sessions in real time. This is useful for debugging and demos.

As you can see, the automation scripts don’t run inside the container. Instead, they run on the host machine and connect to the browser remotely through the Chrome DevTools Protocol.

When a new session starts, rayobrowse assigns a real-user-looking fingerprint from a large database of actual devices, containing thousands of permutations collected from websites Rayobyte owns.

Requirements

The rayobrowse project is designed to run on Linux servers without GPUs (which is a common deployment environment).

These are the required prerequisites:

• Docker, as the browser runs entirely inside a container.

• ~2GB of available RAM, as each browser instance uses ~300MB.

The main benefit of this Docker-based approach is that you don’t need to install Chromium locally, configure fonts, or set up Xvfb manually. All of those dependencies live inside the container, which keeps the host machine clean, portable, and reproducible.

It also makes the project well-suited for self-hosted environments without exposing its internal Chromium patching logic, making it much harder for anti-bot solution providers to reverse engineer how it works.

In terms of compatibility, rayobrowse works on Linux, Windows (native or WSL2), and macOS. The supported architectures are x86_64 (amd64) and ARM64 (Apple Silicon and AWS Graviton). Still, you don’t have to worry about the architecture, as Docker automatically pulls the correct image for the host machine.

Optional: If you plan to use the stealth browser through the Python SDK, an additional requirement is Python 3.10+.

Check the TWSC YouTube Channel

How to Access rayobrowse

There are two main ways you can access rayobrowse:

1. The /connect endpoint.

2. The built-in Python SDK.

Method #1: Use the /connect Endpoint

The first rayobrowse usage method involves connecting directly to the /connect endpoint. This allows any CDP‑compatible tool (including Selenium, Playwright, and Puppeteer) to open a browser session simply by pointing to a WebSocket URL like ws://localhost:9222/connect.

For instance, take a look at the Playwright connection example below:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    # Connect to rayobrowse via CDP
    browser = p.chromium.connect_over_cdp("ws://localhost:9222/connect")
    page = browser.new_context().new_page()

    # Automation logic...

    browser.close()

Keep in mind that the WebSocket browser connection URL can be customized using query parameters, as follows:

ws://localhost:9222/connect?headless=false&os=android&proxy=http://user:pass@host:port

This URL creates a rayobrowse Chromium browser session in headful mode, using Android-based fingerprints, while routing all requests through the proxy http://user:pass@host:port.

Explore all /connect query parameters in the docs.

Method #2: Use the Python SDK

You can also interact with rayobrowse through the built-in Python SDK. This exposes a create_browser() function that returns a CDP WebSocket URL for a newly created browser instance. From there, connect using Playwright or another automation framework, as shown below:

from rayobrowse import create_browser
from playwright.sync_api import sync_playwright

# Configure the rayobrowse connection to run in headful mode 
# while simulating a Windows-based fingerprint
ws_url = create_browser(headless=False, target_os="windows")

with sync_playwright() as p:
    # Connect to rayobrowse with the configured URL via CDP
    browser = p.chromium.connect_over_cdp(ws_url)
    page = browser.contexts[0].pages[0]
 
    # Automation logic...

    browser.close()

This approach gives you more control over the browser lifecycle, but it also involves more configuration and setup.

For more examples (e.g., proxy integration, multi-browser management, etc.), check out the docs.

Get Started with rayobrowse: Step-by-Step Guide

In this guided section, I’ll show you how to build a simple Playwright script that connects to rayobrowse.

For the sake of simplicity, I’ll assume you already have:

• A Unix-based system (Linux, macOS, or Windows via WSL).

• Docker installed and running on your machine.

• Git installed locally.

• A Python environment set up with Playwright installed.

Follow the instructions below!

Step #1: Clone the rayobrowse Repository

The first step is to clone the rayobrowse repository to your machine:

git clone https://github.com/rayobyte-data/rayobrowse

Then, enter the project folder with:

cd rayobrowse

The cloned folder already includes everything you need to get started, including:

• docker-compose.yml: For running the browser container.

• requirements.txt: For installing the Python SDK.

Step #2: Set Up the Environment

rayobrowse requires a .env file that contains the configuration needed to run the browser daemon. For a full list of available environment variables and what they enable, explore the official documentation.

Start by creating a .env file as a copy of the .env.example file coming with the repository:

cp .env.example .env

Then open the .env file and make sure it contains:

STEALTH_BROWSER_ACCEPT_TERMS=true

This confirms that you accept the project’s LICENSE. Without that setting, the daemon will refuse to create browser sessions.

Step #3: Start the Docker Container

Launch the rayobrowse Docker container:

docker compose up -d

Docker will automatically pull the appropriate image for your system architecture (x86_64 or ARM64). Then, it’ll start the container, as explained earlier.

Step #4: Connect via CDP and Apply the Automation Logic

You can now connect to the running rayobrowse instance through the /connect endpoint using any CDP-compatible client. In this example, I’ll use Playwright with Python:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    # Connect to the rayobrowse browser through the CDP WebSocket endpoint
    browser = p.chromium.connect_over_cdp(
        "ws://localhost:9222/connect?headless=false&os=windows"
    )

    # Create a new browser context and page
    page = browser.new_context().new_page()

    # Navigate to the target (sample) page
    page.goto("https://quotes.toscrape.com/")

    # Print the page title to verify the session is working
    print(page.title()) # Output: "Quotes to Scrape"

    # Add your scraping logic here...

    # Close the browser session
    browser.close()

At this point, write your scraping or automation logic, which will run inside the stealth Chromium browser provided by rayobrowse.

For debugging, you can watch the browser session live through noVNC at http://localhost:6080/vnc.html. While the script is running, you should see a headful Chromium session opening and navigating to the target page specified in the script:

As you can tell, the server creates a headful Chromium session (due to the headless=false query parameter) and connects it to the page requested by the script.

Optional: If you want more control over the browser lifecycle, install the Python SDK with:

pip install -r requirements.txt

Take a look at the official examples in the repository for more guidance.

Need help with your scraping project?

Pricing and Limitations

This is how the rayobrowse pricing model works:

What matters most for us, developers, is that you can run rayobrowse for free via self‑hosting. In practice, the only real cost comes from proxies, which are necessary for scaling scraping workloads and avoiding IP bans (something that’s standard in most production scraping setups).

The main thing to keep in mind is that rayobrowse is still in beta. Rayobyte already uses it to scrape millions of pages per day, but results can vary depending on the target site and configuration.

Fingerprint coverage is currently strongest for Windows and Android, while macOS and Linux profiles are less mature. In addition, Canvas and WebGL fingerprinting are still evolving, which means some websites may detect the current implementation.

Benchmarks and Final Comment

To put rayobrowse to the test, I ran a simple script against a single page for each of the most popular anti‑bot detection systems. These are the results I obtained:

Note: These tests were performed on my local machine using my ISP’s IP address.

As you can see, in this simple experiment rayobrowse achieved a 100% success rate, while Playwright failed consistently in headless mode and even struggled in some headful scenarios.

This suggests that the project is definitely worth keeping an eye on, especially thanks to its self‑hosted nature.

To be honest, and this is just my personal opinion as an expert who works in this field, I don’t usually get very excited about projects like this…. In my experience, many libraries of this type either get cracked down on or simply don’t receive the long‑term support they deserve. In this case, however, things are a bit different. The project is closed‑source and backed by a well‑known company in the industry, which makes the expectations for its future understandably much higher!

Here, I covered what the project is about, what it offers, how it works, and how to use it. As always, remember to use rayobrowse only for legal and ethical web scraping. Until next time!

FAQ

Why is rayobrowse based on Chromium and not Chrome?

rayobrowse is based on Chromium simply because Chrome is closed-source. Plus, tests performed on difficult websites show no meaningful difference in detection rates between Chrome and Chromium. Using Chromium also avoids false positives and reflects the broader ecosystem of Chromium-based browsers like Brave, Edge, and Samsung Internet.

Is rayobrowse open source?

rayobrowse isn’t open-source to prevent anti-bot companies from reverse-engineering it. Similar projects, like Camoufox, were quickly studied and countered once their code became public. Rayobyte decided to keep the project closed-source to help maintain its effectiveness and reliability over the long term.

Can everyone use rayobrowse?

No, not all companies can use rayobrowse. Its license prohibits organizations listed in Rayobyte’s restricted list from using the software. For everyone else, the project is free to download and run locally.

Does rayobrowse support proxy integration?

Yes, Rayobrowse fully supports proxy integration. You can route traffic through any HTTP proxy using the proxy query parameter on the /connect endpoint or via the proxy option exposed by the create_browser() function from the Python SDK. The proxy support includes authentication and rotating proxies.

We have seen this pattern in every scraping operation we have worked with. The knowledge exists. It is just scattered across wikis, Slack threads, internal repos, and people’s heads. The real problem is not documentation; it is retrieval. People write things down. They just cannot find them when it matters.

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

In THE LAB #77, we explored the concept of RAG (Retrieval-Augmented Generation) applied to scraped data and showed how to build a basic knowledge assistant using FAISS. That was a proof of concept. This time we are going deeper. We are showing the production system we actually built and use daily, and we are explaining the reasoning behind each design choice: why markdown, how embeddings work, which chunking strategy actually performs better, and what role auto-tagging plays in retrieval.

After reading this article, we hope you will understand the mechanics well enough to build the same system for your team.

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

What we are building and why

At TWSC, we have published around 300 articles over the past four years. Tutorials, reverse-engineering deep dives, tool comparisons, anti-bot analysis. When we sit down to write a new article, we need to remember what we have already covered, find previous work to link to, and check whether a technique we are about to describe was already explained in a past issue. Doing this by memory or by searching Substack’s archive stops working after the first hundred articles.

We also follow what the broader community publishes. Projects like Crawl4AI, which appeared on Hacker News, show that the need to ingest web content into structured, LLM-ready knowledge bases is shared across the industry. The tools for crawling and extracting content keep getting better, but the retrieval side, finding the right piece of information in a growing archive, still requires a purpose-built system.

So we built one. Here is what the complete pipeline looks like:

Sources                                  Processing              Storage & Retrieval
─────────                                ──────────              ───────────────────
Substack articles                   ──┐
                                      ├──> HTML-to-Markdown ──> Frontmatter + Tagging ──> Markdown files
Hacker News and other sources       ──┘

Markdown files ──> Chunker ──> Embedder (e5-large-v2) ──> PostgreSQL + pgvector

Search query ──> Query embedding ──> Cosine similarity search ──> Ranked results

Three stages, each independent and replaceable. You scrape content from your sources. You process and embed it. You search it.

If your team writes in Confluence instead of Substack, you swap the scraper. If you prefer Qdrant over pgvector, you swap the vector store. The architecture remains the same.

And here’s the hardware used for most of the steps, from embedding to the storage and retrieval: my DGX Spark.

Yes, I know, probably an overkill.

Check the TWSC YouTube Channel

The tools

Playwright handles browser-based scraping for our own Substack articles. Substack serves content dynamically and requires authentication for premium posts, so a plain HTTP client is not an option.

Algolia API (via Hacker News) provides structured search over HN stories. No scraping needed: HN exposes its full search index through public endpoints.

ScrapegraphAI and Firecrawl convert external article URLs into clean markdown. ScrapegraphAI is the primary extractor, Firecrawl is the fallback.

sentence-transformers with the intfloat/e5-large-v2 model generates 1024-dimensional embeddings. We will explain why we chose this model later in the article.

PostgreSQL with pgvector stores embeddings and handles similarity search. We chose it over dedicated vector databases because we already need PostgreSQL for metadata, and pgvector with HNSW indexing handles our scale without adding infrastructure.

Docker Compose ties everything together as three containers: PostgreSQL, the API server, and the indexer.

You can find the code in our GitHub repository reserved to paying users, inside the folder 101.KNOWLEDGE_BASE.

Need help with your scraping project?

Why markdown as the universal format

The first design choice we had to make was what format our knowledge base would store. We had content from Substack (HTML), Hacker News links (various formats), and potentially Confluence, Google Docs, or Slack in the future. We needed a common representation.

We chose markdown for three reasons.

First, markdown preserves document structure without carrying rendering noise. An HTML page contains navigation bars, ad slots, JavaScript, CSS classes, and layout dividers. None of that is content. When you convert to markdown, you keep headings, paragraphs, code blocks, links, and lists. Everything the embedding model needs, nothing it would choke on.

Second, markdown is readable by humans and machines alike. When something goes wrong in the pipeline, you can open a markdown file and immediately see what the system is working with. Try doing that with a serialized HTML DOM or a JSON blob from an API response.

Third, YAML frontmatter is a natural fit for markdown and gives us a structured metadata header without mixing it into the content. Each file gets an `id`, `type`, `title`, `publish_date`, `topics`, and `visibility` field. This metadata drives filtering at search time and never enters the embedding model. The separation is important: embeddings capture meaning, frontmatter captures facts.

There are two paths to get content into markdown. You can build your own converter using open-source libraries, or you can use commercial services that handle extraction and conversion for you. In this article we show both approaches deliberately. For our own Substack articles, we built a converter from scratch with BeautifulSoup and markdownify. It costs nothing, we control every detail, and it works because we know the source HTML structure intimately. For external content discovered on Hacker News, we use commercial services like ScrapegraphAI and Firecrawl instead, because every URL leads to a different site with a different HTML structure. Building custom converters for thousands of unknown domains would be impractical. The trade-off is clear: when you control the source, build your own; when you are scraping the open web, commercial extraction services save an enormous amount of development time.

Our Substack HTML-to-markdown converter is deliberately simple. It strips scripts, styles, buttons, navigation, and footers, then converts the remaining HTML:

def html_to_markdown(html: str) -> str:
    soup = BeautifulSoup(html, "lxml")
    for tag in soup.find_all(["script", "style", "button", "form", "nav", "footer"]):
        tag.decompose()

    md = markdownify(
        str(soup),
        heading_style="ATX",
        bullets="-",
        strip=["script", "style", "button", "form", "nav"],
    )
    md = re.sub(r"\n{4,}", "\n\n\n", md)
    return md.strip()

The final output for each document looks like this:

---
id: a1b2c3d4e5f6...
type: twsc_article
title: "THE LAB #94: Using Cookies and Session Persistence"
slug: the-lab-94-using-cookies-and-session
canonical_url: https://substack.thewebscraping.club/p/the-lab-94-using-cookies-and-session
publish_date: 2025-11-15
visibility: premium
topics:
  - browser-automation
  - cloudflare
  - scraping-infra
---

[article body in markdown]

Scraping your own content

The first source we built was a scraper for our own Substack articles. The pattern applies to any CMS: discover URLs, authenticate if needed, extract content, convert to markdown with frontmatter.

URL discovery and authentication

Most publishing platforms expose a sitemap. We fetch it, filter for article URLs (Substack uses /p/ in the path), and track the lastmod date to detect changes:

def fetch_sitemap(sitemap_url: str) -> list[dict]:
    req = Request(sitemap_url)
    req.add_header("User-Agent", "Mozilla/5.0 ...")
    with urlopen(req) as response:
        content = response.read()

    root = ET.fromstring(content)
    ns = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}

    articles = []
    for url_elem in root.findall("sm:url", ns):
        loc = url_elem.find("sm:loc", ns)
        lastmod = url_elem.find("sm:lastmod", ns)
        if loc is not None and "/p/" in loc.text:
            articles.append({"url": loc.text.strip(), "lastmod": lastmod.text or ""})
    return articles

Substack gates premium content behind authentication. We handle this with a persistent Playwright browser context that stores cookies across runs. On the first run you log in manually; after that, the saved session keeps you authenticated. For cron jobs, we verify the session by loading a known premium article and checking if the full content appears.

We try multiple CSS selectors for extraction because Substack has changed its DOM structure over time. The extracted HTML goes through the markdown converter we showed earlier.

Ingesting external sources: Hacker News

Read more

The traditional way to answer that question involved surveys, focus groups, and expensive reports from firms that charge you a fortune for data that’s already a few months old by the time you read it. Today, the data you need is sitting on public web pages: You just need to collect it.

In this article, we’ll discuss how to scrape data for market research, what sources actually matter, how to build a pipeline that doesn’t fall apart after a week, and where the legal lines are.

Let’s dive into it!

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

What “Market Research” Actually Means Web Scraping Professionals

Market research needs to answer three questions:

• “What are our competitors doing?”

• “What are our customers saying?”

• “How is the market moving?”

That’s it. Everything else is a variation of those three. And if you think about it, the web gives you access to all three, if you know where to look.

In practice, scraped market intelligence sits on three pillars:

• Competitive data: Pricing, product catalogs, feature changes, hiring signals. This is the “what are they doing?” pillar.

• Customer sentiment: Reviews, forum discussions, social media posts. This is the “what are people saying?” pillar.

• Market signals: Job postings, regulatory filings, trend volumes, new product launches. This is the “where is the market going?” pillar.

Now, why scraping instead of traditional research? Because scraping is real-time, it’s continuous, and it doesn’t depend on people filling out forms. A survey tells you what 500 people said last month. A scraper tells you what thousands of customers are saying right now, every single day, without anyone having to opt in.

That’s the competitive advantage. And it’s a big one.

For your scraping activity, you need IPs with good reputation. For this reason, we’re using a proxy provider like our partner Ping Proxies, that’s sharing with TWSC readers this offer.

💰 - Use TWSC for 15% OFF | $1.75/GB Residential Bandwidth | ISP Proxies in 15+ Countries

Where to Scrape: Sources That Actually Matter

Not all sources are worth your time. You could scrape the entire Internet and still end up with nothing useful if you’re not targeting the right places. Below is a list of high-value targets for market research and what you can extract from each:

• Competitor websites: Pricing pages, product pages, feature matrices, changelog, and blog posts. This is your primary source for understanding what competitors are offering and how they position themselves. Pricing pages, in particular, are gold. They change more often than you’d think, and tracking those changes over time tells you a lot about a competitor’s strategy.

• Review platforms (G2, Trustpilot, Amazon, Yelp): Customer pain points, feature requests, sentiment shifts. Reviews are unfiltered customer feedback. Nobody writes a G2 review because they were asked nicely in a survey. They write it because they feel strongly about something—and that’s exactly the kind of signal you want.

• Job boards (LinkedIn, Indeed): Hiring patterns reveal where a company is investing. If a competitor suddenly posts 20 machine learning engineer roles, that tells you something no press release will. Job postings are one of the most underrated market research signals out there.

• Social media and forums (Reddit, X, niche communities): Unfiltered opinions, emerging trends, early complaints about products. Reddit threads and niche forums are where people say what they actually think, not what they’d say in a focus group.

• Government and public data portals: SEC filings, patent databases, import/export records. These are slower-moving signals, but they’re authoritative. A patent filing can tell you what a competitor is building 18 months before it ships.

Here’s the key question to ask yourself before adding a source to your scraper: “Does this data answer a specific research question, or am I just hoarding?”. If you can’t tie a source to a concrete insight, skip it. You’ll save yourself storage costs, maintenance headaches, and potential legal issues.

Building the Pipeline: From Raw HTML to Market Intelligence

A market research scraper is not a one-off script you run from your terminal. It’s a pipeline. And pipelines need structure. If you treat it like a quick script, you’ll end up with a mess of cron jobs, inconsistent data formats, and no idea whether your data is fresh or stale. So, build it properly from the start.

A scraping for market intelligence pipeline should have four stages:

1. Collection: Fetch the pages, extract the fields you need, throw the rest away. Don’t store raw HTML “just in case” (you’ll learn why in the legal section of this article).

2. Storage: Store facts and metadata (source URL, timestamp, extracted fields). Use a structure that makes deduplication and versioning easy. In practice, this means designing your schema around a composite key (for example: source + entity ID + scraped timestamp) so you can track how a data point changes over time without overwriting previous records.

3. Transformation: Normalize the data across sources, deduplicate records, and enrich with additional context (geocoding, industry classification, entity linking).

4. Analysis: Turn rows into insights. This is where the actual market research happens. And to be clear: “Analysis” doesn’t mean opening a CSV and scrolling through it. The goal is to turn your pipeline’s output into dashboards, scheduled reports, or Slack alerts that reach the people who make decisions. If the data sits in a database and nobody looks at it, the whole pipeline is wasted effort.

Scheduling Matters More Than You Think

Different data types have different freshness requirements. Getting this wrong means either wasting resources or working with stale data. The main ideas to consider when engineering the triggering times are the following:

• Price tracking: Daily or hourly, depending on the market. Consider that e-commerce prices can change multiple times a day. SaaS pricing pages, instead, change less often. But when they do, it’s significant.

• Review monitoring: Monitoring reviews daily is usually enough. Reviews don’t appear in real-time, and sentiment trends are measured in weeks, not minutes.

• Job postings: A weekly schedule works for trend analysis of the job market. Remember that you’re looking for patterns, not individual listings.

• Social media: This depends on your use case. If you’re tracking a product launch or a PR crisis, you might need near-real-time. For general trend analysis, daily or even weekly batches work fine.

Tools That Work Well for Market Research Scraping

You don’t need to reinvent the wheel. The software industry already provides you with the best tools for your market research scraping pipeline. Here’s a solid stack for a market research pipeline:

• Scrapy for structured crawling. Scrapy’s architecture is designed for exactly this kind of work: You define spiders per source, plug in middleware for proxy rotation and retry logic, and use item pipelines to clean and store data as it flows through. For market research specifically, Scrapy’s built-in feed exports let you dump results straight to JSON, CSV, or even S3 without writing custom I/O code. And if you need to coordinate multiple spiders (say, one per competitor), Scrapy’s project structure keeps things organized as your source list grows.

• Playwright or Puppeteer for JS-heavy pages. The key difference from Scrapy is that you’re running a real browser, which means you can handle dynamic content, infinite scroll, and client-side rendering. The trade-off is resource cost: Each browser instance eats memory and CPU, so you don’t want to use this for targets that serve static HTML.

• A task queue for scheduling and orchestration. This is what turns a collection of scrapers into an actual pipeline. Instead of running scripts manually or relying on cron jobs, a task queue lets you schedule scrapes per source at different intervals, retry failed jobs automatically, and control concurrency so you’re not overwhelming a target site with parallel requests. It also gives you visibility: you can see what’s queued, what’s running, what failed, and why.

• PostgreSQL for structured market data that needs querying and versioning. Relational databases shine here because market research data is inherently relational: competitors have products, products have prices, prices change over time.

The point is this: Pick tools that let you build a maintainable system, not just a working script. Every tool in this stack solves a specific problem, and none of them requires you to build infrastructure from scratch. The best market research pipeline is the one that’s boring to operate, because boring means reliable.

Check the TWSC YouTube Channel

Scaling Without Getting Blocked

If you’re scraping one competitor once a week, you don’t need this section. If you’re tracking 50 competitors daily across thousands of pages, you do.

Here’s the reality: The moment you start scraping at scale, you become visible. But sites don’t like bots, even polite ones. So you need to be smart about how you scale. Consider the following rules of thumb to avoid getting blocked:

• Proxy rotation: Residential proxies for sensitive targets (sites with aggressive anti-bot systems), datacenter proxies for everything else. Rotate per request or per session, depending on the site’s detection mechanisms. The key is to not send thousands of requests from the same IP in an hour.

• Rate limiting and backoff: Be a good citizen. If you hammer a site with concurrent requests, you’ll get blocked, and you’ll deserve it. Implement exponential backoff on failures, and set reasonable delays between requests. A 2-3 second delay between requests is a good starting point for most sites.

• Fingerprint management: Headers, TLS fingerprint, and browser-level signals matter on sites with serious anti-bot systems. Make sure your request headers look consistent and realistic.

• CAPTCHAs: If you’re hitting CAPTCHAs regularly, your approach is too aggressive. Fix the root cause (rate, fingerprint, proxy quality) before reaching for solver services. CAPTCHA solvers are a band-aid, not a solution.

The general principle is simple: Scrape at a pace that doesn’t degrade the target site’s performance.

Need help with your scraping project?

Turning Scraped Data into Actual Market Insights

Let’s be clear about something: Raw scraped data is not market research. It’s just data. A CSV with 50’000 rows of competitor prices is not an insight. A chart showing that competitor X has dropped their enterprise tier price by 15% over three months: That’s an insight.

Here’s where the value gets created:

• Price tracking and competitive benchmarking: Track changes over time, visualize trends, and set alerts for significant moves. The goal is not to know what a competitor charges today. It’s to understand their pricing trajectory. Are they moving upmarket? Are they running more frequent discounts? Are they simplifying their tier structure? This is where predictive analytics meets scraped data with the goal of predicting future moves from your competitors.

• Sentiment analysis on reviews: Use NLP to extract themes from customer reviews. This is powerful for product teams who want to understand what customers love and hate about competitors. But remember: You’re analyzing the data internally, not republishing the reviews.

• Hiring signal analysis: Aggregate job postings by role type, department, and location. A competitor suddenly posting 15 ML engineer roles tells you they’re investing in AI. A wave of sales hiring in EMEA tells you they’re expanding geographically. This is a signal that’s almost impossible to get from any other source.

• Trend detection: Time-series analysis on product launches, feature changes, pricing moves, or social media mentions. The goal is to spot patterns or anomalies before they become obvious. If three competitors all add the same feature within two months, that’s a market trend, not a coincidence.

Overall, the output of your scraping pipeline should be dashboards, reports, or automated alerts, not a database dump that someone has to manually dig through. If the insights don’t reach decision-makers in a usable format, the whole pipeline is wasted effort.

Legal and Ethical Considerations: Don’t Skip This Section

I know, I know. You’re a developer, not a lawyer. But here’s a thing I’m sure you know: Most legal problems in scraping are self-inflicted. They happen because someone scraped “everything on the page,” stored it “for later,” and only then asked: “Wait, can we actually use this?”

As discussed in detail in “How to Avoid Copyright Violations While Scraping”, let’s go through the key legal and ethical principles of web scraping shortly:

• Scrape facts, not expression: Copyright protects expression, not facts. Prices, SKUs, dates, availability, and job titles are facts. No one owns the fact that a SaaS product costs $49/month. On the other hand, product descriptions, review text, and blog posts are creative expressions.

• Don’t store raw pages by default: Storing the HTML of entire pages means creating copies of copyrighted content. Instead, parse in-memory, extract only the fields you need, and discard the rest. If you need to debug, store a small sample with short retention.

• Respect robots.txt: The robots.txt file is not the law, but ignoring it is evidence of bad faith if things go sideways. In disputes, it can be used to show that you knew you were unwelcome and kept going anyway.

• Terms of Service matter: If the ToS explicitly forbids scraping and you scrape anyway, you may have a breach-of-contract problem. This is often easier for the site owner to prove than copyright infringement, because the argument is straightforward: you agreed to a contract, then you violated it.

• Don’t scrape behind a login: Once you log in, you’ve affirmatively agreed to a contract. Breaking that contract to scrape is a fast track to legal trouble. If your plan requires authenticated access, treat it as a licensing problem, not an engineering challenge.

• GDPR/CCPA: If you’re scraping anything that could be personal data (usernames, reviewer names, profile information), you need to know which privacy laws apply. This is especially relevant for review scraping and social media monitoring.

Here’s the mental model that works: A price comparison tool that shows prices and links back to the source? Generally safe. A product catalog that copies descriptions, images, and reviews so users never need to visit the original site? That’s where you get into trouble, even if you don’t publicly display the results because you use them for internal analysis.

Keeping Your Scrapers Alive: Monitoring and Maintenance

Scrapers in production break for several reasons. Sites change layouts, add anti-bot measures, restructure their URLs, or just go down for maintenance. If you don’t monitor your scrapers, your data goes stale silently, and you won’t know until someone asks why the pricing dashboard hasn’t updated in three weeks.

Here’s a breakdown of what you need:

• Dead selector detection: Alert when a CSS selector or XPath returns empty across multiple consecutive runs. A selector that worked yesterday and returns nothing today means the site changed its HTML structure. The keyword here is “multiple consecutive runs”. A single empty result could be a transient issue, so consider not triggering alerts on the first failure. Instead, set a threshold, like three consecutive empty results, before flagging it. When it does fire, you need to inspect the current page structure and update your selectors. Alternatively, try to go beyond the DOM using AI and LLMs, to make your extraction more resilient to layout changes in the first place.

• HTTP status monitoring: A spike in 403s means you’re getting blocked. A spike in 429s means you’re hitting rate limits. A spike in 404s means URLs have changed. Each of these requires a different response. For 403s, check your proxy pool and rotation logic: You might need fresher IPs or a lower request rate. For 429s, back off and increase your delays between requests; the site is telling you exactly what the problem is. For 404s, the target has likely restructured its URL patterns, which means you need to update your URL generation logic, not just retry the same broken links. Log these status codes per source and per run so you can spot trends early. A gradual increase in 403s over a week is a warning sign that your current setup is losing effectiveness, even if individual runs still return some data.

• Data quality checks: Row counts, null rates, value distributions. If your price tracker suddenly shows all prices as $0 or your review scraper returns empty text fields, you want to know immediately. Build quality checks into your pipeline as a post-scrape validation step, not as something you run manually. Compare each run’s output against baseline expectations: If you normally get 200 rows from a source and today you got 12, something is wrong, even if those 12 rows look fine individually.

• Automated tests against fixture HTML: Save sample HTML pages from your targets and write tests against them. When a test fails, you know the site has changed before your production scraper breaks. Treat your scrapers like production code, because they are. In practice, this means saving a snapshot of a relevant section in the target page as a local HTML file. Then, write unit tests that run your extraction logic against that fixture and assert expected outputs. Store these fixtures in version control alongside your scraper code. When a site changes and your production scraper breaks, update the fixture with the new HTML. This gives you a repeatable workflow for handling site changes instead of scrambling every time something breaks.

The goal is simple: You should know when something breaks before your stakeholders do. A Slack alert that says “Competitor X pricing scraper returned 0 results” is infinitely better than a product manager asking why the dashboard is empty.

Conclusion

In this article, you learned that market research scraping is about building a reliable pipeline that collects the right facts, transforms them into insights, and doesn’t get you in legal trouble.

The competitive advantage of scraping for market research is in what you do with the data. Anyone can code a scraper. But building a system that delivers reliable, actionable market intelligence week after week? That’s where the real value is!

So, let us know: Are you using web scraping for market research? What sources have you found most valuable? How did you structure your scraping pipeline? Let’s discuss in the comments!

The Proxy Price Benchmark is now updated weekly

If you haven't checked it yet, the Proxy Price Benchmark is the tool I built to answer a simple but important question: how much should you actually be paying for your proxies?

Every week, I (or better, my fleet of agents) update the pricing data directly from the vendors, so you always have a reliable reference to compare offers or negotiate with your current provider.

This week, we added two new vendors: Dataimpulse and AnyIP, bringing the total number of monitored providers to 27.

Check the latest prices

If you use proxies at scale and would find API access to this data useful, I am considering a paid API plan. If you are interested, join the waitlist and tell me about your use case. I want to understand demand before I build it.

This week on Scraping News: stealth browsers are getting serious

The Scraping News feed has been tracking an interesting trend this week: two new stealth browser projects worth watching.

Owl Browser is a purpose-built browser engine for automation at scale. Not a Playwright wrapper but a full engine built on Chromium (CEF) with a custom C99 HTTP server, 256 parallel contexts, and sub-12ms cold start. Self-hosted, Docker-ready, with Python and TypeScript SDKs. If you are running high-volume scraping and hitting the limits of standard headless setups, this is worth a closer look.

Rayobrowse is Rayobyte's open-source stealth Chromium browser, released from their production scraping infrastructure. It handles fingerprint randomization at the browser level (user agent, WebGL, fonts, screen resolution, timezone) and connects via CDP, so it works with Playwright, Puppeteer, Selenium, or any custom script. Runs on headless Linux with no GPU required.

Both address the same problem from different angles: standard headless Chromium is detected, and the solution is now moving from patch-level evasion to full browser-level stealth. We will be covering both in depth on TWSC soon.

See all the latest news on Scraping News

Keep in mind that both the Proxy Price Benchmark tools and Scraping News are in an early version; feel free to suggest improvements and bug fixes.

You might think that, in a web scraping scenario, the solution is simply to connect directly to the WebSocket channels. Sure, that’s possible, but there are a few obstacles along the way. The main ones are WebSocket anti-bot techniques and bot detection measures.

In this post, I’ll walk through the most common ones, explain how they work, and share proven tips and tricks to help you avoid them.

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

A Quick Intro to WebSockets

Before diving into WebSocket bot detection, let me first provide some context about WebSocket as a protocol and its role in web scraping.

What Is the WebSocket Protocol?

WebSocket, also abbreviated as WS for short, is a web protocol standardized in RFC 6455 that enables full-duplex, bidirectional communication between clients and servers over a single, persistent TCP connection.

Unlike HTTP, which is stateless and request-driven, WebSockets establish a long-lived connection through an initial HTTP handshake. After the handshake, both client and server can send messages independently, with data transmitted in frames that can be text, binary, or control frames (ping, pong, close).

WebSockets support fragmentation, masking, and optional compression via extensions like per-message-deflate, while newer HTTP/2 and HTTP/3 mechanisms allow multiplexing, reduced latency, and better proxy traversal.

For your ethical scraping activity, you need IPs with good reputation. For this reason, we’re using a proxy provider like our partner Ping Proxies, that’s sharing with TWSC readers this offer.

💰 - Use TWSC for 15% OFF | $1.75/GB Residential Bandwidth | ISP Proxies in 15+ Countries

Why and When Web Pages Use WebSockets

The WebSocket protocol opens the door to live, bidirectional web communication. Unlike HTTP’s request-response model, it lets servers and clients exchange data continuously over a single, persistent connection.

In general, WebSockets are essential for any application where low latency and frequent updates are required. Common use cases include:

• Live streaming: YouTube Live, TikTok LIVE, Kick, Twitch, and similar platforms.

• Chat applications: Slack, Discord, and other messaging services.

• Collaboration tools: Google Docs, Figma, and online whiteboards.

• Gaming and multiplayer experiences: Browser-based MMO games, turn-based games, and PvP games.

• Financial data feeds: Stock tickers, cryptocurrency price updates, and trading dashboards.

• IoT and telemetry: Sensor updates, home automation, and device monitoring.

• Notifications and alerts: Push updates for social networks, dashboards, or monitoring systems.

In short, WebSocket comes into play wherever instant, continuous communication is necessary (and standard HTTP polling would be too slow or resource-intensive).

Main Challenges of Scraping Data from WebSockets

Connecting to a WebSocket server for collecting data isn’t as straightforward as spoofing API requests for web scraping. In particular, the main challenges of scraping data straight from WebSockets include:

• Finding the right client implementation: You must use a WebSocket client (and there are way fewer than HTTP clients…) that supports the correct protocol version and any negotiated extensions, such as compression or subprotocols.

• Limited documentation and examples: WebSocket scraping is less common than API scraping, so there are fewer guides, tools, and community resources available.

• Proxy integration complexity: Not all clients support proxy integrations, making IP rotation a challenge.

• No request–response model: You can’t simply send a request and receive a response, as with API scraping. Instead, you must send the right messages and then listen to a continuous stream of events.

• Real-time data handling: You require a system to collect, process, and store messages in real time, often dealing with high-frequency updates.

Main WebSocket Anti-Bot Techniques and Solutions

Now you’re ready to discover the most important WebSocket-specific bot detection techniques, along with practical tips to avoid and bypass them. The idea here is to target a WebSocket server from an automated script, relying on a WS client in Python, Node.js, or another programming language of your choice.

WebSocket Handshake Issues

The WebSocket handshake is a transition phase in which an HTTP connection is upgraded to a persistent WebSocket connection. During this step, both the client and the server negotiate the connection parameters, and either side can abort the process if the conditions aren’t acceptable.

Because the handshake is where the protocol upgrade happens, it’s also a pivotal security and bot-detection point. The server must carefully validate everything the client requests. Otherwise, protocol misuse or security issues may occur.

In detail, during the handshake, a WebSocket client must send a valid HTTP/1.1 GET request with specific headers, for example:

GET /live-data HTTP/1.1
Host: example.com:9000
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: JKjeFfYU8mti9re0prPQrw==
Sec-WebSocket-Protocol: chat, superchat
Sec-WebSocket-Version: 13

In practice, browsers also include additional headers such as Origin, User-Agent, Referer, Cookie, as well as authentication headers (e.g., Authorization). While these HTTP headers aren’t strictly required by the WebSocket specification, they are extremely valuable for fingerprinting and bot detection.

Now, the server should respond with 400 Bad Request and immediately close the connection if it encounters:

• An unknown or malformed header.

• An invalid Sec-WebSocket-Key.

• An unsupported WebSocket version.

Instead, if the WebSocket version is unsupported, the server should return a Sec-WebSocket-Version header listing the versions it supports (most modern servers only accept version 13).

In practice, repeated handshake failures or non-browser-like handshake patterns are often treated as a bot indicator. Those may result in blocking, particularly after repeated handshake attempts from the same IP or when fingerprinting enables identification even across IP changes.

📌 Tips:

• Always send a valid Origin header: All major browsers include it, and many servers automatically reject WebSocket requests without one.

• Replicate real browser handshakes as closely as possible: Inspect the WebSocket request made by a real browser and match all headers (e.g., User-Agent and similar extra headers).

• Avoid excessive handshake attempts from the same machine: Too many connection attempts in a short time window are a common bot signal.

• Use IP rotation carefully: Rotation can help avoid rate-based blocks, but it doesn’t protect against fingerprint-based detection if the handshake remains identical.

Honeypot WebSocket Events and Channels

If you’re familiar with common anti-bot techniques, you’ve probably heard of honeypots. A honeypot is a decoy mechanism designed to attract bots by exposing fake or hidden resources, allowing systems to detect automated behavior when those resources are accessed or interacted with (e.g., invisible links or fake pages created to study bots).

In the context of WebSockets, honeypot events are a possible anti-bot technique to detect automated clients. With this approach, the server deliberately sends fake, misleading, or non-actionable events over the WebSocket connection. Similarly, the server might expose channels that aren’t meant to be accessed by regular clients.

Yet, automated scraping bots may react incorrectly to WebSocket honeypots by:

• Processing incoming data that is fake or intentionally invalid.

• Requesting access to or subscribing to channels they aren’t supposed to use.

📌 Tips:

• Study real browser behavior carefully: Inspect WebSocket traffic in your browser’s DevTools (“Network” → “Socket”) and observe which server messages actually trigger data flow or UI updates.

• Avoid assuming every message is meaningful: Remember that reacting to every event can lead to detection.

Connection Lifecycle Anomalies and Patterns

Since WebSocket channels are stateful (unlike stateless HTTP requests), servers can detect bots by analyzing connection behavior over time. Scraping bots tend to prioritize speed over realistic user behavior, which can produce identifiable patterns.

In this regard, popular bot-like indicators include:

• Very short-lived connections: Opening and closing sockets rapidly to collect data.

• Immediate reconnections after closure: Reconnecting instantly without human-like delays.

• High connection churn per IP: Multiple connections from the same IP within a short period.

• Missing browser events: Typical browser WebSocket clients trigger events like proper socket closure, whereas bots often skip them.

• Unnatural latency patterns: Servers use ping frames as heartbeats to check responsiveness. Real users on home Wi-Fi or mobile networks exhibit variable latency (jitter), while automated scripts deployed on data centers generally show extremely stable, low-latency responses.

📌 Tips:

• Introduce some randomness: Introduce realistic delays between connections and reconnections.

• Replicate intended behavior: Emulate browser close events if testing automated clients.

• Add latency variation: Consider latency variation when sending and receiving frames to mimic real-world network jitter.

• Rotate connection IPs: Use proxies to distribute WebSocket connections across multiple IPs.

WebSocket Binary Data Transmission

WebSocket servers sometimes choose to send binary data instead of plain text or JSON. The main technical reasons for this are:

• Reduced bandwidth: Binary messages omit field names and whitespace, making packets smaller than JSON strings and supporting high-frequency updates.

• Faster parsing: Binary data can be read as typed arrays or fixed-size fields, avoiding JSON parsing overhead.

• Custom protocols: Web apps can define their own compact binary format for predictable, high-frequency data.

• Efficient number storage: Numeric values can be stored in 1–4 bytes rather than as multi-character strings, saving space.

For instance, TikTok LIVE pages use WebSockets to stream updates (e.g., chat messages, view counters, and other statistics) in binary format:

Sure, binary data can be converted to text. So, you may think that’s not a problem…

Well, keep in mind that most web applications using binary data implementations include some form of compression or encryption. This adds significant complexity!

Reverse-engineering these systems is technically possible by inspecting browser WebSocket clients, analyzing request headers for compression hints, or trial-and-error with common compression methods. Still, that’s time-consuming and error-prone. Plus, encryption keys, salts, or other details can easily change with each deployment.

📌 Tips:

This time, the only piece of advice I have is to look for alternative data sources. Many WebSocket-based pages, including TikTok LIVE, use regular HTTP APIs to retrieve initial data.

Note: Why aren’t these APIs called server-side when the HTML page is generated? In the case of live data, it’s more reliable to fetch it on the client, because even a single second of latency could result in outdated or inconsistent information.

Thus, polling over those RESTful APIs instead of the WebSocket data streams can allow you to retrieve the information of interest without dealing with binary encoding, compression, or encryption challenges.

Check the TWSC YouTube Channel

WebSocket-Based Bot Detection Measures

The WebSocket protocol is built on top of HTTP, so they inherit many anti-bot techniques commonly used for HTTP requests. At the same time, due to its stateful and persistent nature, anti-bot solutions like WAF (Web Application Firewalls) can leverage WebSockets to detect automated behavior even more effectively…

As a result, WebSocket-based anti-bot measures are not only relevant when connecting directly to WS servers, but also when interacting with web pages through browser automation tools like Playwright and Selenium. That’s why you must know them!

Advanced TLS Fingerprinting

Traditional HTTP fingerprinting checks headers and TLS details. WebSockets extend this by combining the TLS handshake with WebSocket-specific framing, which is much harder to spoof. Signals include JA3/JA4 fingerprints, unusual cipher suite ordering, frame fragmentation patterns, and incorrect masking behavior.

Need help with your scraping project?

Continuous Device Fingerprinting

HTTP allows basic fingerprinting on a per-request basis, but it can’t verify whether the client’s environment remains consistent. The stateful nature of WebSockets enables servers to continuously validate device fingerprints over time. For example, servers can request Canvas/WebGL renders, available fonts, and other browser characteristics repeatedly. Any inconsistency can lead to an immediate block.

Real-Time User Behavior Monitoring

WebSockets allow live streaming of mouse, keyboard, and scrolling events back to the server. This enables a much deeper level of user behavior analysis compared to static HTTP requests.

After all, most browser automation scripts produce perfectly straight mouse movements or instantaneous clicks, while human interactions naturally include slight jitter, variable speed, and reaction delays. These differences make automated clients easier to detect when behavior is constantly monitored over a WebSocket connection.

Conclusion

Here, I introduced the WebSocket protocol and explained why and when it comes in handy. Specifically, you learned that it powers live data updates on web applications. Want to access that data? Well, it’s not as straightforward as you might think due to WebSocket anti-bot techniques.

In this post, I explored the most relevant WS bot detection methods, along with useful advice for bypassing them successfully. You also saw how WebSocket’s stateful, continuous data streaming can be used by WAFs and other advanced anti-bot systems for enhanced detection.

I hope you found this helpful and informative. If you have any questions or comments, drop them below. Until next time!

But a browser does everything the hard way. It downloads the full page, parses HTML, executes JavaScript, renders the DOM, loads images, fonts, and stylesheets. For each request, it allocates hundreds of megabytes of RAM and takes seconds to complete what an HTTP client could do in milliseconds. When a pipeline needs to scrape ten pages, this overhead is irrelevant. When it needs to scrape ten thousand pages, the browser becomes the bottleneck.

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

Consider a concrete scenario: we need to monitor the wishlist of an e-commerce account, pulling product data, stock levels, and price changes every hour across hundreds of items. Running Camoufox for every single API call would mean spinning up a full browser instance, navigating to each page, waiting for JavaScript to execute, extracting the data, and closing. For a hundred items, that is minutes of execution time and gigabytes of memory. The same API calls through an HTTP client would complete in seconds using a fraction of the resources.

As we measured in THE LAB #96, HTTP clients with TLS impersonation can be 27x faster than browsers on the same target. The difference is not marginal. It is the difference between a pipeline that runs on a single machine and one that requires a cluster.

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

The problem is that these two approaches are usually treated as mutually exclusive. Either you use a browser for everything, accepting the overhead, or you try an HTTP client and hope the anti-bot system does not block it. But many websites only need a browser at the gate: for the login, the initial challenge, or the session establishment. Everything after that is plain API calls.

If we can use a browser to earn a valid session and then hand it off to an HTTP client, we get the reliability of browser automation where it matters and the speed of HTTP everywhere else. That is the pattern we want to build. But the handoff is not as simple as copying a few cookies, and the traps along the way are worth understanding before building a pipeline around this idea.

The hybrid pattern

The idea is simple in principle. Many websites require a browser only at the gate: the login flow, the initial anti-bot challenge, or the session establishment. Once that gate is passed, subsequent requests are plain API calls or page fetches that do not require JavaScript execution. If we can extract the session state from the browser and replay it through an HTTP client, we skip the browser for 99% of the work.

The session state, in practice, means cookies. An authentication flow sets session cookies that the server trusts for subsequent requests. If we transfer those cookies from the browser to an HTTP client, the server should treat the HTTP client as the same authenticated user.

But cookies alone are often not enough. Modern anti-bot systems like Akamai do not just check whether you have the right cookies. They also check whether the client presenting those cookies looks like the same client that earned them.

This is where TLS fingerprinting enters the picture: if the browser that logged in was Firefox, but the HTTP client that reuses the cookies presents a Python TLS fingerprint, the server may reject the request or simply drop the connection without responding.

So the real challenge is not just transferring cookies. It is maintaining continuity across two different execution models: the browser and the HTTP client must look like the same entity to the server.

Check the TWSC YouTube Channel

Tool landscape

For this experiment, we used two tools.

Camoufox is a custom Firefox build designed for stealth. It spoofs fingerprints (WebGL, canvas, audio, navigator properties), patches headless detection vectors, and uses Playwright’s Juggler protocol for automation. We covered it extensively in THE LAB #65: Scraping Datadome-protected websites with Camoufox. Its role here is limited to one thing: logging in.

curl_cffi is a Python binding for curl-impersonate, a modified version of curl that mimics the TLS and HTTP/2 fingerprint of real browsers. It supports impersonating Chrome and Firefox at specific versions, which means it can present the same TLS fingerprint as the browser that established the session. Unlike a browser, it uses negligible resources per request and can process thousands of pages per minute.

The key property that makes this pairing work: Camoufox is Firefox-based, and curl_cffi can impersonate Firefox’s TLS fingerprint. The server sees a consistent Firefox identity across both steps.

You can find the code in our GitHub repository reserved to paying users, inside the folder 100.HYBRID_SCRAPING.

Need help with your scraping project?

The target: Net-a-Porter

We chose Net-a-Porter as our target. It is a luxury e-commerce platform protected by Akamai Bot Manager, with authenticated features (wishlists, account details) exposed through internal JSON APIs. This gives us a clean test case: the login requires a real browser (Akamai blocks automation tools at the login endpoint), but the authenticated API calls are plain HTTP requests that return structured JSON.

Please keep in mind that this is an experiment for study purposes, and we’re not inciting you to scrape Net-a-Porter or any other website, especially the part behind a login.

Before diving into code, we need to understand what we’re dealing with. Net-a-Porter’s architecture has three layers relevant to us:

Akamai Bot Manager sits in front of everything. It sets a cluster of tracking cookies (_abck, bm_sz, bm_s, ak_bmsc, and others) that are generated through JavaScript execution on the client side. These cookies prove that a real browser visited the page. Without them, API calls either fail or hang indefinitely.

The login API at /api/nap/wcs/resources/store/nap_il/loginidentity/v2 accepts a JSON payload with email and password. On success, it returns a 201 status with an Ubertoken in the response body. This token is the key to all authenticated endpoints.

Authenticated API endpoints like the wishlist API at /api/nap/wcs/resources/store/nap_il/wishlist/v2/{id} require both the session cookies and the Ubertoken passed as an x-ubertoken header. They return clean JSON with product details, stock levels, and metadata.

The experiment: what worked and what did not

We did not arrive at the final solution directly. The investigation path itself reveals the constraints of session handoff, so it is worth walking through each attempt.

Read more

Lots of tools have emerged to solve the issue of browser fingerprinting, which, for example, is one of the primary reasons for blocks even when using high-quality residential proxies. So, the need companies have for stable, scalable data collection makes anti-detect solutions essential for survival in the current status of the industry.

In this article, you’ll discover Dolphin{anty}: A powerful anti-detect browser that lets you orchestrate hundreds of unique, isolated browser profiles. You’ll learn its strengths, why you should consider it for your scraping or multi-accounting projects, and how it works with a practical guide.

Let’s dive into it!

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

What is an Anti-detect Browser?

An antidetect browser is a specialized web browsing tool designed to mask a user’s digital fingerprint, allowing them to appear as a distinct, unique visitor to websites and tracking systems. Standard browsers like Chrome or Firefox broadcast a user’s hardware and software data. An anti-detect browser, instead, enables users to customize and spoof these parameters for every session.

In the context of web scraping, web scraping professionals use this technology to bypass anti-bot measures that rely on browser fingerprinting to identify and block automated traffic. Anti-detect browsers can also be used in “multi-accounting” strategies. You can use them to create isolated browser profiles, each with its own unique fingerprint, cookies, and proxy IP. The common use case is that a single user can manage hundreds of social media, e-commerce, or ad accounts simultaneously without triggering security flags that would normally link the accounts together and lead to mass bans.

A successful data pipeline is made not only by the right tool to use, but also from the right IP address. Proxy providers like Decodo help you achieving your scraping goals.

Try Decodo Now

What is Dolphin{anty} and Why Consider it for Your Web Scraping Projects?

Dolphin{anty} is an anti-detect browser that allows you to manage hundreds of unique, isolated browser profiles for web scraping and multi-accounting. You can use it via its desktop application or programmatically, as it provides a flexible API for deep integration with your scripts.

The best part of using it is that you can orchestrate wide scraping operations without worrying about browser fingerprinting. Forget about immediate IP bans, CAPTCHAs triggered by suspicious metadata, or complex cookie management. Dolphin{anty} handles the masking of your digital identity for you very simply. Also, thanks to its built-in “Scenarios” builder and synchronizer, it can automatically replicate human-like actions across multiple profiles simultaneously. So, say goodbye to manual warm-up routines and the fear of losing accounts to anti-fraud systems.

The top reasons why you should consider it for your projects are the following:

• Advanced anti-detect capabilities: If you’ve been scraping for a while, you know that standard headless browsers often leak metadata that triggers anti-bot defenses. Dolphin{anty} solves this by providing real, unique digital fingerprints for every profile. It mimics user behaviors at a granular level, allowing you to bypass sophisticated detection systems without the constant headache of being blocked.

• Mass profile management: Managing a few accounts is easy, but scaling to hundreds or thousands is a different beast. Dolphin{anty} is built for scale. It allows you to orchestrate hundreds of isolated browser profiles from a single interface. Whether you are managing a massive farm of accounts for data collection or need to segment your scraping tasks, the tool provides the infrastructure to keep everything organized and efficient.

• Flexible API integration: For those who prefer code, Dolphin{anty} offers a robust API that integrates deeply with your existing Python or Node.js pipelines. This allows you to automate profile creation, launch browsers programmatically, and integrate the anti-detect capabilities directly into your custom scraping infrastructure.

For your ethical scraping activity, you need IPs with good reputation. For this reason, we’re using a proxy provider like our partner Ping Proxies, that’s sharing with TWSC readers this offer.

💰 - Use TWSC for 15% OFF | $1.75/GB Residential Bandwidth | ISP Proxies in 15+ Countries

Dolphin{anty}’s Main Features

Dolphin{anty} is packed with features designed to make multi-accounting and scraping easier. The main features you should know about it are the following:

• Real fingerprint generation: The core of Dolphin{anty} is its ability to provide genuine device fingerprints. Instead of just blocking trackers, it creates a unique digital identity for every profile you run. In practice, it manages over 20 parameters—from WebRTC to Canvas—so your scrapers look exactly like real users on real devices.

• Built-in Automation: You don’t always need to be a coding wizard to automate tasks. Dolphin{anty} offers a “Scenarios” builder that lets you create automated workflows visually. Whether it’s warming up accounts or parsing data, you can set these scripts to run automatically. And for those who prefer code, the flexible API allows you to integrate these profiles directly into your existing scripts.

• Profile synchronizer: This is a game-changer if you need to perform the same action across multiple accounts. The Synchronizer allows you to perform an action in a “master” profile, and the tool automatically repeats that exact action across all other selected profiles in real-time. This saves you a massive amount of time on routine interactions.

• Team collaboration: If you work in a team, you know that sharing browser sessions and cookies can be a nightmare. Dolphin{anty} simplifies this by allowing you to transfer profiles, cookies, and proxies to colleagues in just a few clicks. You can also manage permissions, ensuring that team members only have access to the functionality they need.

• Smart profile management: When you are dealing with hundreds of profiles, organization is key. The tool provides a highly intuitive interface where you can use tags, statuses, and notes to sort and find your profiles instantly. It’s built to help you navigate a large farm of accounts without getting lost in the chaos.

Check the TWSC YouTube Channel

Hands-on Dolphin{anty}: Step-by-step Scraping Tutorial

In this section, you will see how easy and fast it is to use Dolphin{anty}. Get ready for the tutorial!

Setting Up Dolphin{anty}

First of all, you need to create a new login. After creating a new account on Dolphin{anty}, the system will ask you to download the software. As you can see from the image below, it supports all the major Operating Systems:

Below is how Dolphin{anty}’s interface appears after you installed it on your machine:

Good. Everything is set up. Time to create new profiles!

Need help with your scraping project?

Create New Profiles

Before using Dolphin{anty}, you have to create a new profile. To do so, click on CREATE PROFILE and fill in the fields:

Profiles are the core of Dolphin{anty}. This is where, for example, you can change the fingerprinting for your anti-detect strategies. To do so, you only need to click on NEW FINGERPRINT, and the tool will change all the fingerprinting data for you. And if the standard fingerprinting is not sufficient, you can manage advanced configurations:

Also, if your use case needs to use a specific social media like Facebook, you can set Facebook’s URL as the starting page and the credentials to log in to a profile you need to manage:

When everything is set up, click on SAVE, and your profile is completed! You are now ready to use Dolphin{anty} via UI or code.

Use Dolphin{anty} Via The UI

The power of anti-detect browsers rely in allowing you to create different profiles and letting you use the browser with one instance, but different profiles. So, after you created the profiles, click on START to launch the instances:

Dolphin{anty} will launch a new browser instance, allowing you to manage as many profiles as you have created and activated. Below is the expected result:

That’s it for using Dolphin{anty} via UI!

Use Dolphin{anty} Via Code

Before using Dolphin{anty} via code, you have to create an API key. To do so, navigate through the API panel in the web app:

Now you can connect to a profile through a port generated at startup and automate the browser using tools like Puppeteer, Playwright, Selenium, and others.

Basic automation you can do includes the following:

1. Start a profile via API with DevTools Protocol enabled.

2. Connect to the profile’s port using a browser tool.

3. Run your own automation script through the open connection.

Dolphin{anty} allows you maximum flexibility, so you can use your favourite programming language. For example, below is how you can write an authorization script:

import requests
api_url = "<http://localhost:3001/v1.0/auth/login-with-token>"
token = "your-api-key"
request_data = {"token": token}
headers = {"Content-Type": "application/json"}

response = requests.post(api_url, json=request_data, headers=headers)
if response.status_code == 200:
	print("OK", response.json())
else:
	print("Error", response.status_code)

If the response is successful, you will receive a message like the following:

{"success": true}

Discover how to use Doplhin{anti} via API by reading the documentation!

Pros and Cons of Dolphin{anty}

Like any tool, Dolphin{anty} has its strengths and weaknesses. Here is a breakdown of what you need to know before deciding if it fits your stack.

👍 Pros:

• Top-tier fingerprinting: The ability to generate real, unique fingerprints for every profile is its biggest selling point. It goes beyond simple user-agents, making your scrapers look genuinely human.

• Built-in automation tools: The “Scenarios” builder and the Synchronizer are massive time-savers. You can automate routine warm-up tasks or replicate actions across dozens of profiles without writing a single line of code.

• Team-centric design: If you work with a team, the ability to transfer profiles and share them instantly is invaluable. It removes the friction of sharing session data manually via files or text.

👎Cons:

• REST API complexity: This is a significant friction point for developers. Unlike other solutions that offer native SDK wrappers, Dolphin{anty} relies only on REST API calls for automation. This adds “boilerplate” complexity compared to simply importing a library.

• Resource intensive: Running multiple browser profiles with full fingerprinting requires significant system resources. You will need a powerful machine if you plan to run dozens of concurrent sessions locally.

Conclusion

In this article, you discovered Dolphin{anty}, a flexible anti-detect browser that can be used both via UI and via code. As you’ve learned, it comes packed with interesting features that can speed up your processes. In particular, we found that the “Scenarios” feature is the one that actually makes it stand out.

So, let’s discuss in the comments: Were you already using Dolphin{anty} before reading this article? What’s your experience with it?

We are not talking about a cease-and-desist letter or a Terms of Service dispute. Google did not send SerpApi any communication before filing the lawsuit. No cease-and-desist, no attempt to resolve their concerns directly. SerpApi told us this was highly unusual, and that had Google reached out, they might have learned that their claims lack merit.

Google is invoking the Digital Millennium Copyright Act, specifically Section 1201, the anti-circumvention provision. The same statute originally designed to prevent people from cracking DVD encryption is now being pointed at a SERP scraping API.

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

We reached out to both Google and SerpApi for comment on this case. Google did not respond. SerpApi did, and we will include their statements throughout this article where relevant.

Let us break down what happened, why it matters, and what it could mean for anyone who scrapes the web for a living.

The Facts

Google’s complaint tells a straightforward story. SerpApi, founded in 2017 by Julien Khaleghy, operates a paid API that sends automated queries to Google Search and returns the results as structured JSON. Google estimates that SerpApi sends hundreds of millions of artificial search requests per day, and that this volume has increased by as much as 25,000% over the past two years.

In January 2025, Google deployed a technological protection measure called SearchGuard. SearchGuard works by sending JavaScript challenges to incoming search queries. For regular browser users, the challenge is invisible: the browser runs the JavaScript, sends back the expected response, and the search results load normally. For automated systems, the challenge is a wall. Bots that cannot execute JavaScript or that fail behavioral checks get blocked.

According to Google’s complaint, SerpApi’s response to SearchGuard was to build circumvention mechanisms. The complaint alleges that SerpApi creates “fake browsers using a multitude of IP addresses that Google sees as normal users,” misrepresents device and location information when solving challenges, and syndicates authorization tokens from legitimate requests to unauthorized machines around the world. Google also alleges that SerpApi uses automated means to bypass CAPTCHAs that SearchGuard deploys as a secondary verification layer. SerpApi disputes these factual allegations.

The complaint cites SerpApi’s own blog posts, where the company reportedly described SearchGuard as making “web scraping more difficult” but claimed to be “fortunate to be minimally impacted” because its services had “already pre-solved Google’s JavaScript challenge.”

The Legal Theory

This is where it gets interesting for the scraping industry, because Google chose not to sue under the Computer Fraud and Abuse Act (CFAA). That would have been the traditional route. Instead, Google went with the DMCA.

The context matters. The CFAA path has been significantly narrowed by the hiQ Labs v. LinkedIn case. In that landmark decision, the Ninth Circuit held that scraping publicly available data does not violate the CFAA, and warned against allowing companies to create “information monopolies.” The Supreme Court vacated and remanded the case under its Van Buren ruling, but on remand, the Ninth Circuit reaffirmed its original position.

After hiQ, the CFAA is a much weaker weapon against scraping of publicly visible content. Google needed a different legal framework. Section 1201 of the DMCA provides one.

Section 1201 has two relevant provisions. The first, Section 1201(a)(1)(A), prohibits the act of circumventing a technological measure that effectively controls access to a copyrighted work. The second, Section 1201(a)(2), prohibits trafficking in technology designed to circumvent such measures. Google’s complaint invokes both.

The argument chain goes like this: Google’s search results contain copyrighted content, specifically images in Knowledge Panels licensed from third parties, merchant-supplied product images in Google Shopping, and licensed content from Google Maps. SearchGuard is a technological measure that controls access to these search results pages (and therefore to the copyrighted works within them). SerpApi circumvents SearchGuard. Therefore, SerpApi violates Section 1201.

Each act of circumvention carries statutory damages of between $200 and $2,500. Google alleges billions of individual circumventions. Do the math, and the potential damages exceed what SerpApi could ever pay. Google itself notes in the complaint that SerpApi “reportedly earns a few million dollars in annual revenue, but already faces liability that is orders of magnitude higher and growing.”

Check the TWSC YouTube Channel

SerpApi’s Position

When we reached out to SerpApi, they were clear about their stance. On the fundamental legality of what they do, SerpApi told us: “We embrace the term ‘scraping,’ and we practice it legally and transparently. SerpApi accesses publicly visible search results, the same ones available to any browser, and delivers clean, structured JSON back to our customers. We’ve operated this way since 2017, serving developers, researchers, and businesses who need reliable access to public information at scale.”

On the legal boundaries of automated access to search results, their position is equally direct: “The law on this is clear, and we’re prepared to defend that position in court. Scraping is legal, and we stand behind our products and customers. Our API replicates real-time searches with no login, no bypass of any paywall, and no access to anything that isn’t already available to anyone with a browser. U.S. courts have upheld this repeatedly; hiQ Labs v. LinkedIn is a key precedent. The data Google surfaces lives on the open web. Google didn’t create it.”

In February 2026, SerpApi filed a motion to dismiss. Their arguments include the assertion that the DMCA is a copyright protection statute, not a website protection statute, and that Google is improperly trying to use it to control access to public portions of its website. They also argue that mimicking browser behavior to access publicly available pages is not the same as cracking encryption or disabling authentication, and that any ambiguity in the definition of "circumvention" must be given its narrowest reasonable reading, citing the "First Amendment interest in maintaining accessibility of the Internet as an open forum."

SerpApi also pointed out what they see as an absurdity in Google’s theory. If statutory damages were calculated at scale, the total “would exceed U.S. GDP.” Congress, they argue, never intended Section 1201 to be used this way.

On the DMCA claim specifically, SerpApi told us: “The DMCA’s anti-circumvention provision was designed to protect copyrighted works, full stop. Google is not protecting access to copyrighted works. Google is improperly attempting to use the DMCA to limit access to the public portions of its website. We believe that the law is on our side.”

Need help with your scraping project?

The Hypocrisy Argument

SerpApi is not shy about making this point. In a blog post about the lawsuit, they argue that Google’s case threatens access to public data on the open internet and this resonates widely in the scraping community. As they told us: “Google indexed the web without anyone’s permission. That’s how search works. Now it’s trying to pull up the ladder behind it, prohibiting the practices that it used, and still uses today, to build its business empire. That’s why SerpApi is standing up to Google. Not just to protect our business, but to protect legal competition and open access to public information on the internet.”

Google Search operates by crawling, indexing, and presenting content from billions of websites. Many of those website owners never explicitly consented to being indexed. Google’s position has always been that robots.txt provides the mechanism for opting out, and that the default state of the open web is crawlable. Now Google is arguing that its own search results should be exempt from the same logic.

The irony is not lost on legal commentators either. Above the Law( described the case as Google “pulling up the ladder after climbing it.” Eric Goldman’s blog published an extensive guest analysis arguing that Google’s DMCA strategy represents an attempt to relitigate hiQ Labs through a different statutory framework.

Why This Matters Beyond SerpApi

If Google’s legal theory prevails, the implications extend far beyond one API company. The core question is whether deploying an anti-bot system on a publicly accessible website is enough to invoke federal copyright law against anyone who bypasses it.

Think about what that means in practice. Every CAPTCHA, every JavaScript challenge, every behavioral analysis system deployed on a public website could potentially become a “technological protection measure” under Section 1201. Any scraper that solves a CAPTCHA, executes JavaScript to render a page, or rotates IP addresses to avoid detection could be committing a federal offense.

This is not hypothetical. The legal theory applies to any website that hosts copyrighted content (which is almost all of them) and deploys some form of bot detection (which is increasingly all of them).

Eric Goldman’s blog highlighted this exact concern. The guest analysis by Kieran McCarthy warns that accepting Google’s theory would allow any website deploying anti-bot technology to invoke federal law against circumvention, “transforming speed bumps and CAPTCHAs into federally enforceable copyright perimeters.”

The Electronic Frontier Foundation has also weighed in. Staff attorney Tori Noble stated that “the right to scrape publicly available information keeps the Internet free and open,” cautioning that overly broad DMCA interpretations undermine innovation and research.

SerpApi made a similar point when we asked about the impact on consumers: “Scraping-powered services benefit all kinds of consumers who use the web every day. Scraping helps to maintain the free and open flow of information across the internet, ultimately encouraging things like price transparency, competition, and informed decision-making, all to benefit consumers. Expanding the DMCA as Google has suggested would only benefit the largest tech incumbents and hinder transparency and healthy competition.”

The Emerging Legal Pattern

Google’s lawsuit does not exist in isolation. In October 2025, Reddit filed a 41-page complaint against SerpApi, Perplexity AI, Oxylabs, and AWMProxy in the Southern District of New York. The complaint is far more aggressive than Google’s, both in tone and in scope: six legal counts including three separate DMCA claims, unfair competition, unjust enrichment, and civil conspiracy.

Reddit’s framing is vivid. It describes the defendants as “similar to would-be bank robbers, who, knowing they cannot get into the bank vault, break into the armored truck carrying the cash instead.” AWMProxy is characterized as “a former Russian botnet.” Perplexity is compared to “a North Korean hacker.” The language is clearly designed to make scrapers look like criminals.

The underlying theory is similar to Google’s. Reddit has signed licensing deals with both Google and OpenAI to grant them programmatic access to its data. Companies that want Reddit content at scale are expected to pay for it. But when scrapers circumvent SearchGuard to harvest Google’s search results, they also harvest Reddit content without paying a cent. According to data Reddit obtained through a subpoena to Google, the three scraping defendants accessed almost three billion Google SERPs containing Reddit content in just two weeks during July 2025. SerpApi alone accounted for over 1.8 billion of those page accesses. Like Google, Reddit did not send SerpApi any communication before filing suit. SerpApi disputes these figures and the other factual allegations in Reddit’s complaint, and has filed a motion to dismiss in that case as well.

Reddit also produced a piece of evidence that reads like a detective novel. It created a hidden “test post” that could only be crawled by Google’s search engine and was not otherwise accessible anywhere on the internet. Within hours, the contents of that post appeared in Perplexity’s “answer engine.” The only way Perplexity could have obtained that content was through scraping Google’s search results. Reddit calls this technique the equivalent of “marked bills” in a bank robbery investigation.

The Reddit complaint also reveals a detail that connects directly to our industry: after Reddit sent a cease-and-desist letter to Perplexity in May 2024, Perplexity’s citations to Reddit content did not decrease. They increased forty-fold.

And in December 2025, in Ziff Davis v. OpenAI, a federal judge in the Southern District of New York ruled that robots.txt files do not “effectively control access” under Section 1201. Judge Sidney Stein compared robots.txt to a “keep off the grass” sign that “relies on readers to decide to comply rather than enforcing any kind of access control itself.” The ruling is important because it sets a baseline: passive, voluntary measures are not enough to trigger DMCA protection.

But SearchGuard is not robots.txt. It is an active system that executes JavaScript, performs behavioral analysis, deploys CAPTCHAs, and makes real-time decisions about whether to grant access. Whether this kind of system meets the “effectively controls access” standard is the open legal question. The answer will likely set the direction for the entire industry.

Legal commentators have identified what they call the “DMCA 1201 scraping strategy”: platforms deploy technological protection measures specifically to create legal standing under Section 1201, then sue when those measures are circumvented. The sequence is intentional. Deploy, document, sue. Whether courts view this as legitimate copyright protection or as strategic rent-seeking will determine the outcome.

There is also a relevant doctrinal debate. The Lexmark case in the Sixth Circuit introduced the “front door/back door” argument: if a house’s front door is unlocked, putting a lock on the back door does not mean the house is “access-controlled.” Applied here: if anyone with a regular browser can access Google Search results, does deploying SearchGuard against automated systems meaningfully “control access” to the copyrighted works within those results?

The AI Angle

There is one more layer worth noting. As Search Engine Land reported, OpenAI used SerpApi to scrape Google Search results for ChatGPT responses on current events, after Google declined to provide direct access to its search index. SerpApi listed OpenAI as a customer on its website as recently as May 2024 before removing the listing. Other reported customers include Meta, Apple, and Perplexity.

This context matters because Google already has a massive structural advantage in the AI race when it comes to fresh web data. Cloudflare CEO Matthew Prince put numbers on it: “For every one page that OpenAI sees, Google is seeing 3.2 pages.” Against Microsoft, the ratio is 4.8 to 1. The reason is simple. Publishers cannot block Googlebot without disappearing from search results. So Google gets access to the web at a scale that no competitor can match, and it can use that data not just for search but also for training and running its AI products.

In this context, suing companies that make it easier for competitors to scrape Google’s search results is not just about protecting copyrighted images in Knowledge Panels. It is also an act of defense of a competitive advantage. If OpenAI or any other AI company can get structured search data through SerpApi, they partially close the gap that Google’s crawler monopoly creates. Shutting down that channel through litigation serves Google’s position in the AI race, even if the complaint is framed purely in terms of copyright protection.

What Happens Next

The case is still in its early stages. SerpApi filed its motion to dismiss on February 20, 2026. According to the court docket, the initial case management conference before Judge Yvonne Gonzalez Rogers is scheduled for March 30, 2026, and a hearing on the motion to dismiss is set for May 19, 2026.

If the motion to dismiss fails and the case proceeds to discovery and trial, it will force courts to answer questions that have been left open since hiQ. Is a JavaScript challenge a “technological protection measure” under the DMCA? Can anti-bot systems on publicly accessible websites invoke federal anti-circumvention law? Does the DMCA protect the act of accessing a public webpage, or only the copyrighted works behind genuine access controls like encryption and authentication?

For the scraping industry, the stakes are high. A ruling in Google’s favor would give any website with copyrighted content and a bot-detection system a federal cause of action against scrapers. A ruling in SerpApi’s favor would confirm that the DMCA was not designed to protect public webpages from automated access, regardless of the technical measures deployed.

We will follow the case closely. Whatever happens, the days of operating in a legal gray area are coming to an end. The courts will have to draw a line, and that line will define the rules for the next decade of web scraping.

*Disclaimer: We are not lawyers. This article represents our analysis of publicly available court filings and legal commentary. Consult legal counsel for advice specific to your situation.*

The scraping industry is well aware of this problem. A recent analysis by ScrapeOps found that even though proxy prices have dropped by 67% over the past five years, the cost per successful payload has actually increased by 133%, mostly because anti-bot defenses now require heavier infrastructure. When each request costs more, wasting them on unchanged pages hurts even more.

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

Several approaches try to solve this. Tools like changedetection.io monitor pages for visual or structural changes and alert you when something is different. On the more technical side, recently explored using SimHash as a client-side fingerprint to determine whether a document has changed since the last crawl, without downloading the full body. These are valid strategies, but they all share one trait: they require you to build and maintain the change detection logic yourself.

What you might not know is that the HTTP protocol already has a native mechanism for this, and it has been part of the spec since 1999. It is called conditional requests, and it lets the server itself tell your scraper “nothing has changed” by responding with a 304 status and zero bytes of body. No diffing, no hashing, no client-side state management beyond storing a single header value.

We have written about proxy cost optimization before in articles like Optimizing Proxy Usage for Large-Scale Scraping and Analyzing the Cost of a Web Scraping Project, but we have never covered this technique. In this article, we will test it against real e-commerce sites and measure exactly how much bandwidth and money it can save.

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

How HTTP caching works (the short version)

When a web server responds to a request, it can include headers that describe the freshness and identity of the content. Two of these headers are relevant for our purposes.

The first is ETag, short for Entity Tag. It is a string that uniquely identifies a specific version of a resource. Think of it as a fingerprint of the page content. When the content changes, the ETag changes.

The second is Last-Modified, a timestamp indicating when the resource was last updated.

These two headers enable what HTTP calls conditional requests. The idea is simple. After your first request, you store the ETag (or the Last-Modified value) returned by the server. On the next request to the same URL, you send it back using the `If-None-Match` header (for ETags) or `If-Modified-Since` (for timestamps). The server compares your stored value with the current one. If they match, the server responds with status 304 Not Modified and an empty body. If they do not match, you get a regular 200 response with the fresh content.

A 304 response contains zero bytes of body. For a proxy billed per GB, that is a request that costs almost nothing in bandwidth.

The tools we used

The HTTP caching technique itself is protocol-level and works with any HTTP client that allows setting custom headers. You could implement it with Python’s `requests`, httpx, or even raw curl.

For this article, we used curl_cffi, a Python HTTP client built on top of curl-impersonate. Its main strength for our purposes is TLS fingerprinting: it can impersonate the TLS handshake of real browsers (Chrome, Firefox, Safari), which prevents e-commerce sites from blocking the request before we even get to test caching behavior. Without TLS fingerprinting, some of the e-commerce targets we wanted to test would have returned 403 immediately, making it impossible to evaluate their caching support.

Then later in the article, we’ll see if we can use the same approach with Scrapy.

Check the TWSC YouTube Channel

The audit methodology

Before attempting conditional requests, we need to check whether a target supports them. We wrote a simple audit function that makes two requests to any URL.

The first request is a standard GET. We capture the ETag, Last-Modified, and Cache-Control headers from the response, along with the response body size.

If an ETag or Last-Modified header is present, we make a second request with the corresponding conditional header (If-None-Match or If-Modified-Since). If the server responds with 304, the site supports conditional requests and we measure the bandwidth saving.

import time
from curl_cffi import requests


def audit_caching(url: str) -> dict:
    headers = {
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        "Accept-Language": "en-US,en;q=0.9",
        "Accept-Encoding": "gzip, deflate, br",
    }

    resp = requests.get(url, headers=headers, impersonate="chrome", timeout=30)

    resp_headers = {k.lower(): v for k, v in resp.headers.items()}
    etag = resp_headers.get("etag")
    last_modified = resp_headers.get("last-modified")
    cache_control = resp_headers.get("cache-control")
    response_size = len(resp.content)

    result = {
        "url": url,
        "status": resp.status_code,
        "etag": etag,
        "last_modified": last_modified,
        "cache_control": cache_control,
        "response_size_bytes": response_size,
        "supports_304": False,
    }

    if etag or last_modified:
        time.sleep(2)

        cond_headers = dict(headers)
        if etag:
            cond_headers["If-None-Match"] = etag
        if last_modified:
            cond_headers["If-Modified-Since"] = last_modified

        cond_resp = requests.get(
            url, headers=cond_headers, impersonate="chrome", timeout=30
        )

        result["conditional_status"] = cond_resp.status_code
        result["conditional_size_bytes"] = len(cond_resp.content)
        result["supports_304"] = cond_resp.status_code == 304

    return result

You can find the code in our GitHub repository reserved to paying users, inside the folder 99.CONDITIONAL_SCRAPING.

Need help with your scraping project?

Shopify stores: full conditional request support

We focused our testing on Shopify stores because, while working on various scraping projects, we came across several Shopify-hosted sites that had this caching system enabled. Shopify powers hundreds of thousands of online stores and is one of the most common scraping targets in e-commerce, so the finding felt worth investigating systematically. The results were clear: Shopify stores with the native page cache enabled support conditional requests out of the box.

Allbirds, Kylie Cosmetics, and Brooklinen all returned 304 responses consistently. Here is what we measured on Allbirds:

URL: https://www.allbirds.com/products/mens-tree-runners.json
Status: 200
Response size: 7,961 bytes

Caching headers:
  ETag: "page_cache:11044168:ProductDetailsController:de822deb7906aa6f9932541f4fe3dae9"
  Last-Modified: not present
  Cache-Control: not present

Conditional request support:
  304 Not Modified: YES
  Conditional response size: 0 bytes
  Bandwidth saving: 100.0%

The saving is 100% because the 304 response body contains exactly zero bytes. The only cost is the request/response headers, which are a few hundred bytes.

This behavior was consistent across three types of Shopify endpoints. The Product HTML page is the standard storefront URL that a browser would load (e.g. /products/mens-tree-runners), which includes the full rendered page with images, reviews, and theme assets. The Product JSON endpoint is the same URL with .json appended (e.g. /products/mens-tree-runners.json), which returns only the structured product data: variants, prices, inventory, and metadata. The Catalog JSON endpoint (/products.json) returns the first page of the store’s entire product catalog in a single response.

We ran repeated conditional requests on each endpoint and confirmed that all returned 304 consistently. The ETag stayed stable as long as the product data did not change.

Read more

Lots of tools and libraries for the main programming languages have indeed driven web scraping to significant growth. The need companies have for data also makes such growth the actual reason for existing.

In this article, I’ll talk about Kadoa: A tool that lets you create resilient scraping workflows in minutes. I’ll show you its strengths, why you should consider it, and how it works, with a practical guide.

Let’s dive into it!

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

What is Kadoa?

Kadoa is a web scraping tool that automatically and programmatically extracts web data at scale. You can use it either via the UI or via code, as it has SDKs and provides you with REST APIs.

The best part of using it is that you can just paste the target URL and the tool retrieves the data for you. Forget about anti-bot measures, fingerprinting issues, or proxy management: Kadoa does all of that for you very simply. Also, thanks to its AI engine, it can automatically recognize the structure of the data you want to scrape from a target website. So, say goodbye also to CSS selectors and any other strategy you use to go beyond the DOM using LLMs.

Using the right tool is just the first steps for a successful data extraction pipeline. Having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

Why Consider Kadoa for Your Web Scraping Projects?

The top reasons why you should consider Kadoa are the following:

• Scrape via workflows: Kadoa’s UI is built to help you set scraping workflows step-by-step. Insert your target URL(s), define the data schema (or let AI make the work for you), and choose to scrape all the available pages or to remain on page and see the agent work for you.

• Write code only if you need it: Other than the UI, Kadoa provides you with Python and JavaScript SDKs and a wide set of REST APIs you can call. This allows you to create workflows via UI, but to manage and call them via code if you need to.

• Integrated data quality management: Before starting the scraping process of your target data, Kadoa allows you to manage data quality. In practice, it allows you to set data quality rules or to manage the rules it provides you, thanks to its AI agent.

• Easy proxy management: If you’ve been scraping for a while, you know that you have low chances of successfully scraping the majority of the content you need without using proxies. Using proxies is not a very big issue if you are used to it and if you already have a favourite provider. However, Kadoa simplifies proxy management. It already provides you with a list of countries you can choose from and, under the hood, it manages everything that’s needed to integrate proxies in your workflow.

• Scheduling feature: There are cases where you need to scrape the same target data from time to time. Or, eventually, you’d like to be notified when data in a target page has changed. Kadoa provides both these features. You can choose to schedule your workflow to scrape at precise time intervals. You can also choose among different notifications, one of which is getting notified when data is changed.

Kadoa’s Main Features

Below is a list of Kadoa’s top features to help better understand its potential:

• Simple and intuitive UI: Kadoa’s UI is simple and intuitive. It allows you to create workflows in minutes. Every scraping workflow is subdivided into steps, and Kadoa provides you with different screens. In a matter of a few minutes, you can define your preferred setup, insert the target page(s), and leave it scraping for you.

• Chrome extension: Other than the UI, Kadoa provides you with a Chrome extension. If you are a Chrome user, this feature allows you to define everything you need directly on the target page, then trigger the workflow to let Kadoa’s agent start scraping.

• Code integrations: If you are a developer or if you simply need to invoke your workflows via code, Kadoa offers you two possibilities. It provides you with Python and JavaScript SDKs in an open-source repository, so that you can use custom code to invoke your scrapers. Also, if you like to use code but prefer REST APIs, Kadoa provides you with several endpoints.

• Scraping suitable for structured or unstructured data: One of the difficult aspects you may encounter when manually scraping websites is defining how to grab unstructured data. This is one of the typical use cases where you could use AI to detect patterns in data in your scraping projects. The good news is that you don’t need to come up with imaginative solutions. Kadoa automatically retrieves unstructured data for you thanks to its AI engine.

• Data schemas definition: The tool provides you with a feature that allows you to define recurrent data structures. This can be helpful when you retrieve similar data from different websites. If you leave its AI engine to automatically define the data structure, in such cases, you could lose consistency across similar data.

• Proxy and anti-detection features: Forget about anti-bot measures and proxy management. Kadoa manages anti-bot solutions under the hood. It also provides you with a predefined list of locations you can choose from, and it will automatically set coherent proxies.

• Error handling: It provides you with advanced error handling management. Common cases are when the target site goes offline, is under maintenance, or encounters a technical issue. When this happens, Kadoa detects the problem, it notifies you, and automatically retries the data extraction. If recovery still fails, its support team is notified and investigates.

• Integration capabilities: The software allows you to integrate with several third parties. One interesting one is the integration between n8n and Kadoa, which allows you to get your scraping automation workflow a step forward.

• Pricing model and usage graphs: Kadoa offers a free tier option, for which you can use 500 credits. Its pricing model is based on credit consumption, and it provides you with a UI section where you can see a graph of the consumption.

• Extensive docs: Kadoa has extensive documentation that covers both UI and API usage.

Check the TWSC YouTube Channel

Hands-on Kadoa: Step-by-step Scraping Tutorial

In this section, I’ll show you how to use Kadoa on an actual scraping task via the UI. The workflow will retrieve Intel’s historical price from Yahoo Finance:

In this scraping workflow, I will:

• Set the target web page.

• Define the data schema.

• Set scheduling options and notifications.

• Retrieve the actual data.

Before starting the actual workflow, log in to Kadoa. Below is the first access page you will see:

Perfect! You are now ready to create your first scraping workflow with Kadoa.

Step #1: Create a New Workflow

From the main page, click on Add workflow to create a new one and paste the target URL. The Proxy location box allows you to select a country where proxies are localized; leave it to AUTO to let the tool automatically manage it. Click on Continue to proceed with the next step:

Note that inside the Enter one or more URLs box, you have to insert the target page. If the target page is more than one, you can insert all the target pages you are interested in.

Alright, you created a new workflow in Kadoa. Let’s proceed with the next step and customize it!

Step #2: Define the Data Schema

As the next step, define the data schema:

If you want to insert the schema manually, Kadoa already provides you with some predefined schemas. For this tutorial, I’ve chosen to let AI do the job. So I selected AI Suggest Fields.

The system, then, asks you how you want to navigate the data. For the sake of this example, I decided to scrape only the current page from the target one, but you can also choose among three different options:

After clicking on Continue, the agent will start doing its job:

Step #3: Review Extracted Fields and Schedule the Workflow

Because I let AI work, the agent automatically tries to extract the data from the target page. But before proceeding, Kadoa asks for your review:

As you can see from the previous image, the agent has correctly detected the data to extract from the target page. Also, this job is finely improved as the tool provides you with a screenshot of the data it will extract, so that you can visualize it even better.

In the next step, you have to define the scheduling:

For the sake of this example, I decided to run the workflow only once. But, as you can see, you can choose among several scheduling options.

Step #4: Set Notifications and Final Details

As the next step, define the way you want to be notified:

In this case, I decided to be notified via email if the workflow fails. You can add different notification channels by clicking on Add channel:

Next, define the latest details of your scraping workflow:

Before starting with the actual scraping, the system asks you to approve the sample data it proposes to you or to review the data quality rules:

By clicking on Review rules, the tool provides you with automated data quality rules. You can select them if you think this will improve the quality of the scraping result:

When you are done reviewing quality rules, click on Approve. The actual scraping workflow will start and will be queued:

Et voilà! You have launched your first scraping workflow with Kadoa.

Need help with your scraping project?

Download Data, See Logs and Statistics in Kadoa

The workflow section reports all the workflows you created, their status, and the token consumption for each scraper:

By clicking on one workflow, you can see the data it retrieved and can decide the format you want to download it:

The Activity log page reports detailed logs of every action occurred to your workflows:

The Usage page reports graphs of the trend in terms of active workflows and the number of rows extracted for workflow, as well as the remaining total tokens on your plan:

Manage Kadoa’s Workflows via APIs

As introduced before, Kadoa provides you with several endpoints for making calls via REST APIs. The APIs allow you to perform several actions that are not strictly necessary for workflows already created. For example, you can start crawling sessions and create data schemas.

Before using the API, get your API Key under the Settings page:

If you want to manage already existing workflows, either created via the UI or APIs, you have to use the specific workflow’s ID via the UI.

Then you can perform several actions by invoking the REST endpoints. For example, you can schedule a particular workflow for later:

curl --request PUT \\
  --url <https://api.kadoa.com/v4/workflows/{workflowId}/schedule> \\
  --header 'Content-Type: application/json' \\
  --header 'x-api-key: <api-key>' \\
  --data '
{
  "date": "2025-02-07T10:00:00.000Z"
}
'

Where you have to insert the following:

• workflowId : Is the ID of the workflow you want to schedule.

• <api-key>: Is your KadoaAPI key.

• The actual date you want your workflow to start the scraping task. You have to use the ISO format for the date in UTC.

Kadoa: Final Comments

After analyzing and testing the tool, I can say the following are its main advantages and disadvantages:

👍 Pros:

• Ready for AI integration. You can download the scraped data or integrate it into your AI projects directly via API.

• Suits all the user needs, as it provides APIs, SDKs, and the UI.

• Supports structured output formats, including JSON.

• Offers virtually unlimited scalability on the side of infrastructure management and the number of URLS to scrape.

• Focuses on data quality before scraping, not later.

👎 Cons:

• Currently, it supports only 5 proxy locations.

• You can’t scrape all the websites you’d like:

Conclusion

In this article, I’ve presented Kadoa: An AI-powered scraping tool that helps you simplify your scraping projects. As you’ve seen, this is a ready-to-use tool that creates scraping workflows in minutes via UI and also supports code.

Let us know in the comments: Did you know this tool before? Have you already tested it?

In this article, I’ll break down the main reasons so you can understand why LLM-ready scrapers work this way—and how you could even build a simple one yourself!

A Brief Reminder About Web Scraping Tools for AI

With the recent rise of AI, some web scraping solutions have specialized in returning content optimized for LLM ingestion.

That means the content returned by AI-ready web unlockers or open-source scraping libraries isn’t just plain HTML. On the contrary, you often get an optimized Markdown version of the page. (Sometimes it’s even parsed JSON, but that’s a different story I won’t cover here.)

The Markdown content is then ready to be processed by an LLM as part of an AI agent, an AI workflow or pipeline, a multi-agent system, or similar system. In some cases, these web scraping tools are even accessed autonomously by AI agents, which decide when to use them to retrieve web content based on the task at hand.

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

Why AI-Ready Web Scrapers Choose Markdown in the First Place

Let me introduce you to Markdown, the language spoken by LLMs.

Note: I assume you already know what Markdown is, but if not, read its Wikipedia page (as it gives a quick overview with everything you need to know about its syntax).

A Bit of Context About Data Formats in LLMs

Most LLMs can handle pretty much any text-based format you throw at them, whether it’s plain text, HTML, JSON, CSV, XML, or others. Some even have vision capabilities and can understand images or other multimodal content.

Still, under the hood, most LLMs actually “speak” Markdown. That’s how they handle code blocks, tables, and other structured content, if you’ve ever wondered…

I’m sure I’m not the only one who has received a response from ChatGPT or Gemini in pure Markdown, even if I didn’t ask for it. Or sometimes, you can even catch the LLM responding in Markdown, and the page renders it in real time:

Note that the LLM is writing the “```” characters used in Markdown to signify code blocks.

Why Markdown Is a Perfect Data Format for LLMs (and AI-Ready Scrapers Use It Too)

So, cool, LLMs love Markdown and use it behind the scenes. But why? Well, because Markdown is a versatile format that hits all the sweet spots LLMs care about:

• Structured content: Markdown gives you hierarchy and organization out of the box (H1, H2, H3, lists, images, code blocks, etc.), making it easy for LLMs to parse and understand the structure of your content.

• Concise and LLM-friendly: Compared to raw HTML or JSON, Markdown is much more concise. Less unnecessary markup or structure means fewer tokens consumed, which also reduces the risk of hallucinations, truncations, or context overflows.

• De facto standard: While there’s no single formal Markdown standard, GitHub-flavored Markdown has become the widely adopted baseline, so most tools and scrapers default to it.

• Rich content support: Markdown supports images, links, tables, code snippets (and in some cases, such as with MDX/MarkdownX, even raw HTML or embedded React components), making it flexible for a wide range of content types.

• Alignment with training data: LLMs are trained on massive datasets like Common Crawl, where a huge portion of high-quality technical documentation (READMEs, wikis, Stack Overflow posts, etc.) is written in Markdown. This means most AI models don’t just “understand” Markdown. Instead, they learned to reason through its structure during training, giving them a natural intuition for the format.

Long story short, that’s why most web scraping solutions built for AI integrations return content in Markdown (or at least give you the option).

By converting a scraped HTML page directly into Markdown, AI-ready scrapers help the underlying LLM (whether it’s part of a machine learning pipeline, AI agent, RAG workflow, plugin, or other application) process the content efficiently and effectively while also saving on token usage.

Markdown vs HTML

Still not convinced? Take a look at an HTML-to-Markdown conversion of a sports news page from ESPN:

As you can see, the original HTML page contains 125.88 KB of content. After converting to Markdown, it drops to 35.84 KB. That’s a ~28% reduction in size just from a simple data format conversion, without any significant loss of actual content!

If we look at token usage, the difference can appear even more striking. The original HTML page translates to 40,125 tokens:

Meanwhile, the Markdown version corresponds to only 11,175 tokens:

Again, that’s roughly a 3.6× reduction in token usage (which usually translates directly into cost savings, since most AI providers charge based on LLM usage).

For a more direct comparison between data formats, explore the AI data format comparison research piece (which I wrote in collaboration with Bright Data on its Kaggle account).

But Raw Markdown Alone Isn’t Enough…

Now, you might be thinking: “Okay, I’ll just convert HTML pages to Markdown using one of the many HTML-to-Markdown libraries out there, and I’m done.” Well… not quite.

The problem is that a direct HTML-to-Markdown conversion isn’t enough, and below are the main reasons why (and how to address them).

1. Non-Content HTML Tags Get Treated as Content

HTML pages are full of blocks that are required for rendering, but are completely useless for understanding the page itself. Think <script>, <style>, inline JSON configs, and similar tags.

After all, those HTML blocks contain plain text. Thus, conversion libraries (rightfully) treat them like any other text node and include them in the Markdown output:

This clearly pollutes the output with noise the LLM doesn’t need, while also greatly increasing token usage (as <script> and <style> blocks can be surprisingly long!)

🎯 Solution: Use an HTML parser (or, in simpler cases, well-scoped regexes) to remove <script>, <style>, and similar rendering-only HTML tags before converting the page to Markdown.

For instance, simply removing <script> and <style> tags from the input HTML produces an impressive reduction, from 1.93 MB down to 68.37 KB, which also translates into huge token savings!

2. Ads and Promotional Content

Ads, sponsored sections, and “recommended for you” blocks might have nothing to do with the main content of the page. Leaving them in the converted Markdown can confuse the LLM or skew its understanding of what the page is really about.

🎯 Solution: Use proxies that support ad-blocking when retrieving the HTML page, enable OS-based ad blockers on your deployment server, or apply rules to remove ads after fetching the HTML and before converting it to Markdown.

3. Navigation, Headers, and Footers

Menus, breadcrumbs, and footer links are all technically “content,” but could be semantically irrelevant for your use case (particularly if you’re not interested in links for crawling or further exploration).

If those elements aren’t removed or downweighted, they increase token usage. Plus, the LLM may overemphasize them or mistake them for part of the main content.

🎯 Solution: Conveniently remove tags like <header> and <footer>, or design your HTML-to-Markdown system to accept only specific CSS selectors for the blocks you want to include in the conversion process (just like Crawl4AI does).

4. Repeated and Boilerplate Text and Content

Things like “Sign up,” “Log in,” newsletter popups, cookie banners, or legal disclaimers (like GDPR notices) appear on almost every page of a site. Including them wastes tokens and adds repetition, which can degrade reasoning quality and increase the risk of hallucinations.

🎯 Solution: This is a tricky problem, as there’s no easy way to identify and remove all of these elements automatically. I know for a fact that some industry leaders have trained small LLMs specifically for this task, letting them process the remaining HTML (after earlier cleaning steps) to filter out all irrelevant content.

How to Convert a Web Page from HTML to LLM-Optimized Markdown

I was recently asked by a client to analyze specific web pages from competitors’ websites. These included structured pages with hidden elements that required basic interactions (like dropdowns). Plus, some information was spread across badge images, links, etc.

Now, if you’re trying to get high-level insights from web pages, your first idea might be to just copy all the text on a page (CTRL+A + CTRL+C) and paste it into ChatGPT (or a similar AI solution), analyzing it with the right prompt. That’s far from optimal, because you lose structure, links, image URLs, and other important context.

Instead, I wrote a simple Python script that:

1. Reads HTML from an index.html file.

2. Keeps only the <body> tag with Beautiful Soup for restricting the content to what you’re typically interested in.

3. Remove <script>, <style>, <header> and <footer> nodes.

4. Converts it to Markdown using markdownify.

5. Writes the output to an output.md file.

Let me show you this script!

HTML to LLM-Optimized Markdown Script

Here’s the simple script for converting HTML files to LLM-ready Markdown outputs:

# pip install beautifulsoup4 lxml markdownify

from bs4 import BeautifulSoup
from markdownify import markdownify as md

def html_to_markdown(html_input_path: str, output_markdown_path: str):
    # Load the input HTML from a file
    with open(html_input_path, "r", encoding="utf-8") as f:
        html = f.read()

    # Parse the HTML (using lxml for high performance)
    soup = BeautifulSoup(html, "lxml")

    # Remove the undesired tags
    for tag in soup(["script", "style", "header", "footer"]):
        tag.decompose()

    # Keep only the <body> content (if present)
    body_html = soup.body.decode_contents() if soup.body else str(soup)

    # Convert the HTML to Markdown
    markdown = md(
        body_html,
        bs4_options="lxml" # Set the underlying HTML parser
    )

    # Write the Markdown output to disk
    with open(output_markdown_path, "w", encoding="utf-8") as f:
        f.write(markdown)


if __name__ == "__main__":
    html_to_markdown("index.html", "output.md")

Check the TWSC YouTube Channel

How to Use the Script

The script above still involves a few manual steps, but it greatly improves the transformation of a web page into content that’s ready to be sent to any LLM.

First, load the target page in your browser (ideally with an ad blocker enabled):

Once the page has fully rendered, right-click and select the “Inspect” entry. Locate the <html> tag, then use “Copy > Copy outerHTML” option to get the complete HTML of the rendered page:

Note: Copying the rendered HTML is better than copying the HTML from the “View page source” option. The latter misses all dynamic content (basically, anything that requires JavaScript execution and rendering in the browser won’t appear in the raw page source).

Next, in your project folder, paste the HTML into a file named index.html:

Run the Python script, and it’ll generate an output.md file:

You can now pass the resulting Markdown to an LLM for processing.

Compared to a traditional HTML-to-Markdown approach, the tweaks in this process save tons of tokens. In particular, the output produced by this method is just 41.2 KB (compared to 611.68 KB for the original HTML), which corresponds to 11,006 tokens.

If you applied a basic HTML-to-Markdown conversion, you’d end up with a 430.21 KB Markdown file, resulting in 154,191 tokens:

In other words, these basic tricks lead to over 14× token savings. Not bad!

Et voilà! Simple, manual, but highly effective.

Need help with your scraping project?

Next Step

The script I just shared achieves its goal, but it’s super straightforward. I presented it simply to prove how a basic HTML-to-Markdown conversion is essentially suboptimal.

For a more sophisticated result, you could integrate a similar process into your LLM-ready scraper, including CLI options for more control over which tags to remove, which nodes to select, and other conversion settings.

As a project idea, you could even turn this approach into a browser extension that converts rendered web pages in a user’s browser into LLM-ready Markdown output files. Clearly, if you go this route, make sure to follow ethical web scraping practices.

Is Markdown Always the Right Choice?

This is the final question to ask after all this discussion. Now, you might be thinking: “Okay, there are no good reasons to stick with plain HTML when passing web pages to an LLM.”

That’s not true since there are situations where having access to the raw HTML can make a difference. Think of when the HTML contains semantic attributes or metadata. This information would be lost during HTML-to-Markdown conversion.

In detail, these are some scenarios where sticking to HTML for LLM ingestion is beneficial:

• data-* attributes storing product IDs, prices, etc.

• ARIA attributes that convey accessibility or structural information.

• class or other HTML attributes that reveal context beyond the visible content on a node.

• HTML comments contain useful information about the page.

For example, consider this HTML node on an Amazon page:

This is just an empty <span>, but its data-state attribute contains information-rich JSON data that you would lose during the Markdown conversion (as the node doesn’t contain text).

Another common example is visual elements, which often carry semantic information not captured by visible text. For instance, based on the image below, you might think the rating is 5/5, but the aria-label attribute reveals it’s actually 4.3/5:

In short—let’s be honest, as it always happens in IT—converting to Markdown isn’t a one-size-fits-all solution. Therefore, it’s no surprise that most web scraping solutions built for direct AI integrations also offer the option to return raw HTML.

That said, based on my experience in the field and everything highlighted here, I highly recommend sticking to Markdown when feeding web pages to LLMs for processing or data parsing, as the benefits far outweigh the downsides in the vast majority of use cases.

Conclusion

In this post, I’ve outlined why Markdown is the language of LLMs and, consequently, the preferred output format for all web scraping tools that integrate directly into AI systems like workflows, pipelines, agents, and so on.

The reasons are intuitive: Markdown is concise and strips unnecessary markup (reducing token usage) while preserving structure, images, links, lists, tables, and more.

As highlighted, plain HTML-to-Markdown conversion isn’t always optimal, and you need to apply some extra tricks to get the best results.

If you have any questions or comments, drop them below. Until next time!

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

This makes SERP scraping fundamentally different from conventional web scraping. The data you collect is only as reliable as your control over these variables. Scrape from a datacenter IP in Virginia with a desktop Chrome fingerprint while logged out, and you will get one set of results. Scrape the same query from a mobile device in Milan while logged into a Google account, and you will get something entirely different. Both are “correct” Google results. Neither tells the full story.

In this article of The Lab, we wanted to understand how much these variables actually change the output, and more importantly, how to control them reliably.

Google does not want you scraping its results

Before we get into the technical setup, we need to acknowledge something that changed the landscape significantly in early 2025.

Starting in January 2025, Google began releasing SearchGuard, a technical protection measure designed to make scraping search results harder.

SearchGuard works by sending JavaScript challenges to search queries originating from unrecognized sources, as we covered on these pages when it started. When a query arrives, Google’s system transmits JavaScript code that requires the browser to compute and return a “solve”, a set of specific information about the browser environment and the user generating the request. For human users, the solution happens transparently in the browser. For automated systems, it is a wall.

This change in strategy put pressure on all “SEO tools” and the operators that needed to scrape Google search results, suddenly increasing their day-to-day operational costs.

Need public web data, not scraper headaches?

SerpApi turns search results into predictable JSON with built-in scale, location options, and speed. All with no maintenance.

Try for free

The change in this strategy, and especially its timing, prompted professionals to raise questions that will likely never receive an answer. Does this have something to do with the AI race? Is this a way to make it harder for other AI companies rely on Google searches for their answers?
We’ll probably never know the answers, but they’re legitimate questions: SERP scraping is as old as Google search, so why bothering stop bots in 2025 and not some years ago?
However, this is today’s reality, and we need to adapt to it. Let’s examine the specifics of SERP scraping on Google (as always, we’re showing this for educational purposes; be aware of current copyright and scraping laws).

What shapes a Google SERP response

To scrape Google Search reliably, we need to model the system we are interacting with. Google personalizes search results along several axes, each of which produces measurably different output.

Geographic location is one of the most impactful variables. Google determines your location through your IP address and, when available, browser geolocation permissions. A query for “pizza restaurant” from a New York IP returns local results for Manhattan. The same query from a Milan IP returns pizzerias in Milan. This extends beyond local searches: news results, shopping results, and even organic ranking order shift based on geography.

We’ll see in the test part of this article that changing location and mimicking another one is less trivial than expected, since not every proxy type works as expected.

Device type determines the structure and content of the SERP page itself. Mobile and desktop results are not just different layouts of the same data. Google serves genuinely different content. Mobile SERPs prioritize featured snippets, location-based answers, and nearby points of interest. Desktop SERPs give more space to organic links and Knowledge Panels. Some results appear exclusively on mobile or exclusively on desktop. For anyone collecting SERP data for analysis, this distinction is not cosmetic. It is structural.

Login state introduces personalization based on your Google account history. When you are logged in, Google uses your search history, location history, and account preferences to tailor results. When logged out, you get a more “generic” version of the results for your location and device. The difference can be subtle for generic queries and dramatic for anything Google considers personal.

Keywords, of course, are the main driver of change. But in addition to returning different results for different keywords, the answer layout also varies accordingly. If you look for “trousers”, you’ll see more shopping results and product data, while if you’re looking for “aspirin”, you’ll see a more traditional layout.

These four variables interact. A logged-in mobile user in Tokyo sees a fundamentally different page than a logged-out desktop user in London, even for the same query. Controlling all four simultaneously is what makes SERP scraping an infrastructure problem, not just a coding problem.

Check the TWSC YouTube Channel

Tools: An Anti-Detect browser and Selenium

Given the variables we need to control (device type and login state, specifically), and the fact that we are not building a massive scraping operation here, the best setup we can use is Playwright paired with an anti-detect browser.

We need a real browser, not just an HTTP request library like requests or httpx, because Google’s SearchGuard validates the browser environment through JavaScript challenges. A raw HTTP client has no JavaScript engine, no DOM, no window object. It cannot compute the “solve” that SearchGuard requires. The request simply fails or returns a challenge page. To pass these checks, we need something that renders JavaScript and exposes a complete browser environment.

But a standard browser is not enough either. Regular Chrome or Firefox, even when automated with Playwright or Selenium, carries detectable signals: the navigator.webdriver flag, predictable fingerprint values, and missing or inconsistent browser properties. Google’s systems can identify these inconsistencies and treat the session as automated.

That’s why we’re pairing Selenium with an anti-detect browser, which is a modified browser engine that spoofs the properties websites use for fingerprinting. Navigator properties, screen resolution, WebGL parameters, canvas behavior, AudioContext values, font lists, language headers, and device type. Instead of presenting the same default fingerprint every time, an anti-detect browser generates a consistent, realistic identity that looks like a genuine user on a specific device and operating system.

The critical feature for our use case is persistent profiles. An anti-detect browser manages browser profiles that survive across sessions. Each profile stores its fingerprint configuration, cookies, local storage, proxy, and device settings. When we start a profile, it resumes exactly where it left off. This means we can log into a Google account through one profile, close the browser, and reopen it days later with the session still active. Without persistent profiles, we would need to authenticate on every run, which is both impractical and a red flag for Google’s security systems.

For this article, we use Kameleo as our anti-detect browser. It runs as a local service (Kameleo.CLI) exposing a REST API on port 5050, controllable via a Python client. It supports Chromium-based profiles (Chroma) for Chrome and mobile device emulation, and Firefox-based profiles (Junglefox). Each profile is an isolated browser session with its own fingerprint, proxy, and cookies.

Need help with your scraping project?

Setting up the infrastructure: deploying Kameleo on AWS

Our Kameleo instance runs on a Windows EC2 in the US. This means that without a proxy, all traffic exits via a US-based AWS IP address. We will use this setup to demonstrate the difference between the instance’s own IP and a proxy claiming to be somewhere else. I’m sure you’ll be surprised by what we’ll find later.

Installing Kameleo on AWS

We installed Kameleo on a Windows EC2 instance using the standard graphical installer, no rocket science here. Once Kameleo is running on the AWS machine, it exposes its API on port 5050. Our Python scripts run locally and connect to the remote Kameleo instance over the network.

The architecture is straightforward: Kameleo manages browser profiles and runs the actual browsers on the AWS instance. Our local machine sends API commands (create profile, start browser, stop browser) and connects to the browser via WebSocket for Playwright automation. The AWS instance needs port 5050 open in its security group for this to work.

Every script in this article follows the same initialization pattern. We read the remote IP from an environment variable:

from kameleo.local_api_client import KameleoLocalApiClient
import os
kameleo_ip = os.getenv(’KAMELEO_IP’)

kameleo_port = os.getenv(’KAMELEO_PORT’, ‘5050’)

client = KameleoLocalApiClient(endpoint=f’http://{kameleo_ip}:{kameleo_port}’)

Test 1: setting the right location

As we said, one of the keys to extracting SERP data is setting the location we’d like to know more about. Our Kameleo installation is on an AWS US machine, so we expect to get SERP data from there. But if we want to change location?

We run the same query, “weather”, three times from the same AWS instance in the US. First, without any proxy, the traffic exits from the instance’s own IP. Then, through a residential proxy geolocated in Italy. Finally, through a datacenter proxy also claiming to be in Italy. For each run, we first visit whatismyipaddress.com to verify the exit IP, then navigate to Google, type the query in the search bar with randomized keystroke delays, and capture the results.
You can find the code in our GitHub repository reserved to paying users, inside the folder 98.SERP-DATA. If you’re one of them but cannot access the repository, please fill out this form.

In the file test_location_comparison.py, we’ll see how Google responds to us when we’re using different types of proxies.

Read more

In this article, we’ll discuss what copyright in the context of web scraping is, when it occurs, and how to avoid it.

Let’s dive into it!

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

What is Copyright Violation in the Context of Scraping?

Generally speaking, a copyright violation occurs when you reproduce, display, distribute, or create derivative works from someone else’s creative work without their permission (or without a valid legal exception). In the context of web scraping, the “creative work” involves (but is not limited to) the following:

• Articles.

• Images.

• Audio and video.

• Code (under particular conditions).

In other words, if you scrape and reproduce an article (even a small part of it) on your website without the author’s permission, you can be infringing copyright. Whether it is actually infringement depends on context (how much content you copied, how you used it, and which jurisdiction applies), but “a small part of a whole article” is not a safe harbor.

So here’s the thing to bear in mind: Just because some content is accessible on the Internet, it doesn’t mean you can take it. Even though some content is publicly accessible, ownership and reproducibility are not. This is why minding the data you scrape is one of the best practices for ethical scraping.

For your ethical scraping activity, you need IPs with good reputation. For this reason, we’re using a proxy provider like our partner Ping Proxies, that’s sharing with TWSC readers this offer.

💰 - Use TWSC for 15% OFF | $1.75/GB Residential Bandwidth | ISP Proxies in 15+ Countries

How Can Copyright Violations Occur While Scraping?

To avoid copyright infringements, you should know the common cases to take care of. Below is a list of common situations where copyright can be violated while scraping data from websites:

• Copying content: Technically speaking, scraping is copying. When you download a webpage’s HTML to your disk, you’ve made a copy. If that HTML contains creative expression, you have created a copy of copyrighted material. That does not automatically mean you are infringing, but this is the exact action copyright law regulates. And if you store, reuse, or republish that expression without permission (or a solid exception), you’re in infringement territory. Note that courts don’t need the copied content to be 1:1 identical. For them, “substantial similarity” can be enough.

• Copying images and media: Images are typically strongly protected. Scraping image URLs and hotlinking can still be risky, even if you report the source URLs while republishing the images. And, of course, downloading and rehosting is even more direct copying.

• Copying “creative fields” that look like “data”: Product descriptions, editorial blurbs, “about” sections, hotel/restaurant descriptions, FAQs, and similar content is often copyrighted text. While editorial blurbs and similar text are obviously copyrighted content, the others are not so obvious. The point to always take care of is in relation to “creative work”. A product description can be creative work when it contains original language, structure, or marketing copy. But not every description is protected. For example, a purely functional description text may have weak or no copyright protection, depending on the jurisdiction and the originality of the content itself.

• Scraping for training LLMs: Scraping web pages to get data for training LLMs is surely part of the evolving career of web scraping professionals. However, scraping data to train Large Language Models can trigger reproduction/derivative-work arguments in courts. This is still an evolving legal area, so you should not assume “transformative” automatically saves you from legal troubles, especially at scale. The issue between studio Ghibli and OpenAI on copyright violations due to LLMs’ training is one among the open ones, but keep in mind: allegations, investigations, and lawsuits are not the same thing as a final court ruling.

How to Avoid Copyright Violations While Scraping

Having legal issues is probably the worst nightmare for professional scrapers. So, how can you be sure you are not violating copyright while scraping? Below is a list of guidelines to take into consideration:

• Scrape facts, not expression: Copyright protects expression, not facts. Scraping the price of a stock, the temperature in London, or a flight arrival time doesn’t infringe any copyright because these are facts. No one owns the fact that today it is 20 degrees in London. On the other hand, scraping a journalist’s analysis about why the price of a stock moved in a certain direction, or a photographer’s image of London, is a creative expression.

• Transform, don’t replicate: When repurposing content (on your website or anywhere else), transform it. This is a general rule of thumb, but if you are in the US, one of your best defenses is “Fair Use”. But to claim this, your use must be transformative. For example, scraping Amazon reviews and posting them on your own e-commerce site is replicating, not transforming. Even summarizing reviews cannot be considered transformative in some cases, and even when it is, it’s not a guaranteed shield.

• Don’t store raw pages by default: As said before, storing the HTML of entire pages means creating copies. To solve this, you can follow two paths:

  • Parse in-memory.

  • Extract only the necessary content, not whole pages.

• Treat images as a separate “danger zone”: Images are a type of content that, during the whole Internet era, had the majority of copyright issues so far. The safest options are:

  • Using the website’s official APIs when scraping images, if available.

  • Scraping images under a Creative Commons license with compliance.

  • Asking and getting direct licensing from the owner.

Check the TWSC YouTube Channel

Standardized Processes to Stay Safe

So far so good, but let’s be honest: When you are taken by your daily job tasks, it’s easy to lose your compass. To avoid it, the best thing to do is to create standardized (and documented) processes and procedures so that you always operate under a guardrail. This section provides you with a couple of ideas you can implement as standardized processes to be sure you don’t violate any copyright while scraping.

Procedure #1 to Avoid Copyright Violations While Scraping: Develop a Copyright Risk Check

Most copyright problems in scraping are self-inflicted. This happens because developers often scrape “everything on the page,” save it “for later,” and only then do they ask: “Wait, can we ship this?”.

Before you add a field (or a selector) to your scraper, ask yourself the following questions:

• “Is this a fact, or is this someone’s writing?”: Prices, dates, SKUs, addresses, and opening hours are facts. A paragraph of an article is someone’s writing. Remember to treat those differently.

• “If I publish this, would it compete with the source?”: If your application lets users consume the content without clicking the original, you’re not “aggregating.” You’re substituting.

• “Am I copying just what I need, or am I copying the entire page?”: If the answer to this question is: “We only store it for debugging”, then you are building a copy.

• “How much am I taking?”: A single excerpt is one thing. Thousands of excerpts across a site start looking like a dataset designed to recreate the whole content.

• “What am I going to do with it later?”: Internal analysis is one risk profile. A public API that returns the scraped text is a completely different risk profile.

• “Is my plan defensible if someone sends a legal notice?”: If your only defense is “but the content is publicly available”, you don’t have a defense. As said before, public availability is different than ownership-

If answers to these questions feel shaky, the fix is usually boring: don’t collect it, collect less, or get permission.

Procedure #2 to Avoid Copyright Violations While Scraping: Build Your Scraper So It’s Hard to Do Something Dumb

If you want to stay out of trouble, don’t rely on “policy.” Rely on defaults and standards.

Here’s what I mean: The safest scraper is the one that can’t casually vacuum up article bodies, image files, and review text unless you deliberately build it that way.

Below is a process that works safely:

1. Fetch the page.

2. Extract only what you came for.

3. Store facts + metadata (source URL, timestamp).

4. Throw the rest away.

When you really do need to keep anything close to “content” (ie, media), treat it as a special case: short retention, locked-down access, and a reason written down somewhere if needed. Not “maybe we’ll need it later”: You must have a valid reason.

If you want a mental model, you can think of it like so: You’re not building a web scraper. You’re building a pipeline. And pipelines need guardrails.

Need help with your scraping project?

Examples: What “Safe-ish” Looks Like vs What Can Surely Get You in Trouble

Let’s be practical now and see some examples of what is generally safe and what is not. Of course: The following examples are not court outcomes. They’re the kind of setups that tend to be boring (safe-ish) or spicy (trouble-ish):

• Price tracker (safe-ish): You scrape SKU + price + availability + timestamp and show a price history chart. You don’t copy product descriptions or images. This is the classic “facts + original output” use case.

• Product catalog clone (risky): You scrape titles, descriptions, bullet points, images, and reviews, then you show them on your site. That’s not “data.” That’s content. You’re rebuilding their user experience.

• News aggregation (high risk): If you store headlines + links and add your own tags/filters, you’re closer to indexing. If you store full articles and users can read all the content as is without leaving your site, then you’re highly risking getting a trip to the nearest court.

• Review analytics (mixed): Using reviews internally to compute “top complaints this month” is one thing. Republishing reviews precisely as they are is another.

• Business directory (often safer, until you start copying the fluff): Name, address, phone, opening hours: These are usually factual. “About us” sections and photos, on the other hand, are where you cross over into copyrighted expression.

So notice the pattern: The moment your product starts looking like a substitute for the source, your legal risk goes up fast.

The Traps That Have Nothing to Do with Copyright (But Still Hurt)

Copyright is only one way scraping can go wrong. Plenty of scraping disputes are won on issues that are simpler to prove than infringement. Below are the big ones you should treat as “no trespassing” signs:

• Circumvention (DMCA Section 1201): If the site uses a login wall, CAPTCHA, paywall, anti-bot challenges, or IP blocking to stop you, and you write code to bypass those measures, you are potentially violating anti-circumvention laws. This is not “copyright infringement” in the traditional sense, but the practical takeaway is simple: If you have to defeat a technical barrier to get the data, you’re walking into a high-risk territory fast.

• Disregarding robots.txt: The robots.txt isn’t the law, but ignoring it has its implications. In disputes, it can be used as evidence that you knew you were unwelcome and kept going anyway. It can also be relevant to arguments about authorization and “bad faith,” even if it doesn’t create copyright liability by itself.

• Terms of service (contract risk): If the ToS explicitly forbids scraping (and most do), and you scrape anyway, you may be liable for breach of contract. This is often easier for the content owner to win than a copyright claim because the argument is straightforward: You agreed (explicitly or implicitly) to a contract, then you violated the agreement.

• Do not scrape behind a login: Once you log in, you have affirmatively agreed to a contract. Breaking that contract to scrape is a fast track to a lawsuit. If your plan requires authenticated access, treat it as a licensing/permission problem, not an engineering challenge.

Conclusion

In this article, we’ve discussed how copyright infringements can occur while scraping and how to avoid them. As said, it’s not always easy to understand when you are actually infringing copyright, as it depends on the governing laws which, often, are local ones. Still, the main ideas proposed can help you be conservative and stay pretty safe while scraping web pages.

So, let us know: Did you find those practices useful? Do you apply other frameworks to be sure you’re not violating copyrighted content? Let us know in the comments!tat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.”

Think about it: how would you convince several million people to share their spare internet connection with companies that you don’t know how they will use it?
Not an easy task, and some companies took shortcuts, as the IPIDEA case shows.
Let’s see it in detail: more than the takedown itself, we’ll use it as an opportunity to look under the hood: how do residential proxy networks acquire millions of IP addresses, what keeps them running, and why are they so difficult to shut down permanently? The IPIDEA case provides unusually detailed answers to all these questions.

What happens when Big Tech goes after the infrastructure that powers both scrapers and threat actors

On January 28, 2026, Google Threat Intelligence Group (GTIG) announced what they called the disruption of “one of the largest residential proxy networks in the world.” The target was IPIDEA, a name that most people outside the proxy industry had never heard. Yet according to Google’s analysis, IPIDEA’s infrastructure was being used by over 550 distinct threat groups in a single week, including state-sponsored actors from China, North Korea, Iran, and Russia.

This is not just a story about a takedown. It is a detailed look at how residential proxy networks actually work, how they acquire millions of IP addresses, and why disrupting them is harder than it sounds.

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

What Google Actually Did

The disruption involved three coordinated actions:

First, Google took legal action to seize the domains used to control enrolled devices and route traffic through them. Without these command-and-control (C2) domains, the SDK code running on millions of devices loses the ability to receive instructions and proxy traffic.

Second, GTIG shared technical intelligence about IPIDEA’s SDKs with platform providers, law enforcement, and research firms. The goal was to trigger ecosystem-wide enforcement, getting these SDKs flagged and removed across multiple app stores and platforms.

Third, Google updated Play Protect to automatically warn users and remove applications known to contain IPIDEA SDKs. This blocks the network’s ability to recruit new devices on certified Android devices.

Google claims these actions reduced IPIDEA’s available device pool by millions. Whether that number holds up over time is a different question, and we will get to that.

Not all residential proxy networks operate in gray zones. Decodo built theirs on user consent, ISO 27001 certification, and co-founded the Ethical Web Data Collection Initiative to prove the model works.

Try Decodo Now

The Anatomy of a Residential Proxy Network

To understand why this matters, we need to understand what a residential proxy network actually is and how it differs from datacenter proxies.

A datacenter proxy routes your traffic through IP addresses belonging to cloud providers or hosting companies. These IPs are easy to identify and block because they belong to known ASNs (Autonomous System Numbers) associated with commercial hosting.

A residential proxy routes traffic through IP addresses assigned by consumer ISPs to regular households. When you connect through a residential proxy, your request appears to come from someone’s home internet connection in Omaha, Tokyo, or Milan. This makes detection and blocking significantly harder because the traffic looks indistinguishable from a regular consumer browsing the web.

The challenge for proxy providers is obvious: they need access to millions of consumer devices to build a usable network. These devices need to be online, geographically distributed, and willing (or unwilling) to forward traffic.

There is an important nuance here. While Google’s report focuses on residential proxies, the same SDK installation on a mobile phone yields two distinct proxy classes. When the phone is connected to home WiFi, traffic exits through a residential IP assigned by the home ISP. When that same phone disconnects from WiFi and switches to 5G or LTE, traffic now exits through a mobile carrier IP. The device has not changed. The SDK has not changed. But the proxy class has shifted from residential to mobile.

This matters because mobile proxies are typically sold at a premium, sometimes 2-3x the price of residential proxies. Mobile carrier IPs are considered even harder to block than residential IPs because they are shared across thousands of legitimate mobile users through carrier-grade NAT. A single SDK deployment on mobile devices effectively generates inventory for two separate product lines.

Check the TWSC YouTube Channel

How Residential Proxy Providers Acquire IP Addresses

The IPIDEA takedown revealed the specific mechanisms used to build and maintain a large-scale residential proxy network. These methods fall into several categories, ranging from semi-legitimate to clearly deceptive.

SDK Integration

The primary method is embedding Software Development Kits into legitimate applications. IPIDEA operated multiple SDK brands: PacketSDK, HexSDK, CastarSDK, and EarnSDK. These SDKs are marketed to app developers as monetization tools. The pitch is simple: integrate our SDK, and we will pay you based on downloads or active users.

Once embedded, the SDK turns the device into an exit node for the proxy network. The device will accept incoming connections from the proxy infrastructure and forward requests to target websites. The app continues to function normally. The user has no obvious indication that their device is being used as a proxy.

Google’s analysis found that many applications containing these SDKs did not disclose this functionality to users. The SDK was hidden, not mentioned in the terms of service, and ran silently in the background.

Trojanized Applications

Beyond SDK integration, IPIDEA directly operated or controlled VPN applications that served as trojan horses. Galleon VPN, Radish VPN, and Aman VPN all provided genuine VPN functionality while simultaneously enrolling devices into the proxy network.

The logic is effective: users who install VPN applications expect their traffic to be routed through external servers. They are primed to accept unusual network behavior. The proxy functionality hides inside this expected behavior.

Google identified over 600 Android applications across multiple download sources with code connecting to IPIDEA’s C2 infrastructure. On Windows, they found 3,075 unique executables making DNS requests to IPIDEA’s Tier One domains, including applications masquerading as OneDriveSync and Windows Update.

Pre-Infected Devices

Researchers have documented cases of uncertified Android devices shipping with residential proxy payloads already installed. Set-top boxes, TV boxes, and other IoT devices from off-brand manufacturers have been found with hidden proxy software baked into the firmware.

This method bypasses the need for user installation entirely. The device arrives compromised.

The Technical Architecture

Google’s reverse engineering of the SDK code revealed a two-tier command-and-control system.

When an infected device starts up, it contacts a Tier One server. The device sends diagnostic information including OS version, device identifier, and a key parameter that appears to be used for affiliate tracking (determining which app developer gets paid for the enrollment). The Tier One server responds with timing configuration and a list of Tier Two server IP addresses.

The device then periodically polls a Tier Two server, checking for proxy tasks. When a task arrives, it contains a target FQDN (like www.google.com:443) and a connection ID. The device establishes a connection to the target, receives data payloads from the Tier Two server, and forwards them unmodified to the destination.

Google found approximately 7,400 Tier Two servers at the time of their analysis, hosted globally including in the United States. The number fluctuated daily, suggesting a demand-based scaling system.

The infrastructure analysis revealed something important: despite different brand names (PacketSDK, HexSDK, CastarSDK, EarnSDK), all the SDKs connected to the same pool of Tier Two servers. The brands were marketing fronts for a single unified network.

Need help with your scraping project?

The Brand Proliferation Problem

This brings us to one of the most interesting findings. IPIDEA did not operate under a single name. Google identified at least 13 ostensibly independent proxy and VPN brands controlled by the same actors:

• 360 Proxy

• 922 Proxy

• ABC Proxy

• Cherry Proxy

• Door VPN

• Galleon VPN

• IP 2 World

• Ipidea

• Luna Proxy

• PIA S5 Proxy

• PY Proxy

• Radish VPN

• Tab Proxy

These brands operated separate websites, separate marketing, and separate pricing. A customer comparing “922 Proxy” to “Luna Proxy” would have no obvious indication that they were buying access to the same underlying network.

This is not unique to IPIDEA. Industry analysis suggests there are only about 7 truly unique residential proxy networks globally, despite hundreds of brands competing in the market. The rest are resellers, white-labels, or, like IPIDEA, multiple storefronts for the same infrastructure.

Why Residential Proxies Are Hard to Block

The fundamental problem for defenders is that residential proxy traffic looks legitimate. When a request arrives from a Comcast IP in Chicago, there is no technical marker indicating whether it comes from an actual Comcast customer browsing normally or from a proxy network routing traffic through that customer’s compromised device.

The proxy networks exploit this by design. The value proposition they sell is precisely this difficulty of detection.

Security researcher Antoine Vastel published concrete data that illustrates the scale of this problem. By actively testing proxy endpoints, he verified more than 16 million unique IP addresses that were functional and associated with the IPIDEA network during the 30 days preceding the takedown. The breakdown by brand shows the relative sizes within the IPIDEA ecosystem: PY Proxy (PyProxy) accounted for 13.4 million IPs, PIA S5 Proxy for 2.2 million, and Luna Proxy for 549,000.

These are not theoretical numbers from marketing materials. These are IP addresses through which Vastel routed traffic and confirmed as working proxy endpoints. And here is the critical insight from his analysis: even with 16 million identified proxy IPs, defenders cannot simply block them.

The reason is that residential exit nodes mix traffic from automated tools and legitimate human users on the same IP. The device owner browses the web normally, while the SDK, in the background, forwards proxy traffic over the same connection. Blocking these IPs based on proxy activity would inevitably block real users who happen to share an IP or whose IP was previously used as an exit node.

Vastel’s recommendation is telling: use these IoCs for risk scoring, behavioral enrichment, and incident investigation, but not for direct blocking. The data is context, not a verdict. This fundamental asymmetry makes residential proxies valuable to attackers and frustrating for defenders.

His research also confirmed another pattern: IP addresses frequently appear across multiple proxy ecosystems simultaneously. IPIDEA did not rely exclusively on its own residential pool. Requests were routed through or resold from other networks. The same IP might be accessible through IPIDEA, a competitor, and a reseller all at once. This interconnection means that even identifying an IP as “IPIDEA-linked” does not tell you the full story of how it is being used.

Traditional IP reputation systems struggle with this complexity. Blocking known bad IPs works for datacenters where the IP assignments are stable, and the ASNs are identifiable. Residential IP addresses rotate frequently as ISPs reassign addresses, and blocking residential IP ranges blocks legitimate users.

Google’s Approach: Attacking the Infrastructure

Rather than trying to block individual IP addresses, Google attacked the control infrastructure. By taking down the Tier One C2 domains, they severed the connection between infected devices and the proxy operators. Without C2 connectivity, the SDK code on millions of devices becomes inert.

This approach has precedent. It is the same strategy used against botnets: identify the command-and-control infrastructure and take it down. The infected devices remain infected, but they can no longer receive instructions.

Google also partnered with Cloudflare to disrupt IPIDEA’s domain resolution, adding another layer of infrastructure disruption beyond the legal domain seizures.

Will It Work? The Persistence Problem

Here is where we need to be realistic about the limitations of this approach.

The takedown disrupted IPIDEA’s current infrastructure. The domains are gone. The C2 servers are unreachable. Millions of devices are no longer participating in the proxy network. But “no longer participating” is not the same as “cleaned up.”

The fundamental problem is that infected devices remain infected. The SDK code is still installed on millions of phones, tablets, TV boxes, and computers worldwide. Google can take down domains. Google can update Play Protect to block new installations. What Google cannot do is reach into millions of devices and uninstall the malicious code that is already there.

For the SDK to be removed, one of these things needs to happen: the user manually uninstalls the app containing it, the device gets factory reset, or the device gets replaced. None of these happens at scale. Most users are unaware that the SDK exists. The apps that contain it often provide real functionality, games, utilities, and VPNs that users want to keep. There is no mechanism to notify millions of people across dozens of countries that their flashlight app is secretly a proxy node.

This creates an asymmetry that favors the attackers. Google invested significant legal, technical, and coordination resources to take down IPIDEA’s infrastructure. The IPIDEA operators (or anyone who acquires their codebase) can spin up new C2 domains, update their DNS configuration, and potentially reactivate a substantial portion of the dormant network. The SDK code often includes fallback mechanisms and update capabilities precisely for this scenario.

The brand proliferation we discussed earlier is part of this resilience. IPIDEA operated 13+ brands. If some domains get seized, others may survive. If the entire IPIDEA operation is compromised, the operators can rebrand entirely and inherit a pre-installed base of millions of devices waiting for new instructions.

Google acknowledged this reality in its announcement, noting that “this industry appears to be rapidly expanding” and that “there are significant overlaps across providers.” The reseller and partnership agreements that connect different proxy brands mean that disruption propagates unpredictably through the ecosystem, but so does recovery.

This is not a battle that can be won definitively. It is a cost-imposition strategy. The goal is to make operating these networks so expensive and risky that some operators exit the market or shift to more legitimate practices. But as long as the infected device base exists, the infrastructure can be rebuilt. The realistic outcome is degradation, not eradication.

The Economics Behind Residential Proxy Networks

Understanding why these networks exist requires understanding the economics.

Residential proxy bandwidth sells for $4-8 per gigabyte or more to end customers. The cost to acquire that bandwidth through SDK partnerships is measured in cents per gigabyte. The gross margins appear enormous.

But running a proxy operation is not just bandwidth arbitrage. The real costs include:

Engineering and Infrastructure: Maintaining thousands of C2 servers globally, building rotation logic, handling the unreliable nature of consumer devices going online and offline unpredictably.

SDK Distribution: Paying app developers for integration, maintaining relationships with publishers, and navigating app store policies that increasingly scrutinize monetization SDKs. The silver lining is that mobile app SDKs generate dual inventory: residential IPs when devices are on WiFi, mobile IPs when they switch to cellular. This allows providers to sell the same underlying device pool across two product categories at different price points.

Customer Acquisition: Finding buyers for proxy services is expensive. The market is niche, competition is intense, and customers are price-sensitive.

Legal and Compliance: Or in IPIDEA’s case, the lack thereof. Operating in legal gray zones creates ongoing risk. The Google takedown demonstrates what happens when that risk materializes.

Industry estimates suggest customer acquisition costs consume 40-60% of revenue even at scale. The apparent margin compression means that most residential proxy providers operate on thin actual profits despite the high sticker prices.

This economic pressure explains the proliferation of brands. Running multiple storefronts for the same underlying network lets operators segment the market, test different price points, and spread legal risk across multiple corporate entities.

The Bigger Picture

The IPIDEA takedown is part of a larger pattern. Google previously took action against the BadBox2.0 botnet, which shared infrastructure with IPIDEA. Law enforcement agencies worldwide are paying greater attention to residential proxy networks, recognizing the role this infrastructure plays in facilitating activities ranging from credential stuffing to espionage.

The residential proxy industry has partially operated in a legal gray zone for years. The Google action, particularly the legal component, establishes a clearer precedent that enrolling devices without consent and facilitating malicious activity creates meaningful legal exposure.

This does not mean residential proxies are going away. The demand exists, and where demand exists, supply follows. However, the industry may be compelled toward more transparent practices: clearer consent mechanisms, improved disclosure in applications that embed SDKs, and more careful vetting of customers who purchase proxy access.

For those of us who work with web scraping, the takedown is a reminder that the infrastructure we rely on has a supply chain. Understanding the supply chain, including its technical architecture, its business model, and its vulnerabilities, helps us make better decisions about which providers to trust and how to build resilient scraping systems.

The IPIDEA network may be rebuilt, rebranded, or replaced by competitors. However, the detailed technical analysis Google published provides the entire industry with greater visibility into how these networks operate. That visibility, more than the takedown itself, may be the most lasting impact.