Well, not so fast.

Open data does not automatically mean free to use, free to redistribute, or free from privacy obligations. And scraping it without thinking through the implications can land you in legal trouble, get your IP banned from a public infrastructure that was never designed to handle aggressive crawlers, or cause you to expose people’s personal information.

In this article, we will go through a complete picture of the “open data” world: what the problem actually is, how to approach it correctly, and how to implement responsible open data scrapers in Python.

Let’s dive into it!

Before proceeding, let me thank NetNut, the platinum partner of the month. Their set of solutions cover all your needs for scraping.

Visit Netnut

What “Open Data” Actually Means Legally, Ethically, and Practically

“Open” is one of the most overloaded words in the data world. Depending on the license, the jurisdiction, and the type of data involved, the same publicly accessible dataset can be freely redistributable, commercially restricted, privacy-sensitive, or legally off-limits entirely.

So, before anything else, let’s establish what you are actually dealing with.

What “Open-Source Dataset” Actually Means (and What It Doesn’t)

Where a dataset sits on the licensing spectrum determines everything: whether you can redistribute it, whether you can use it commercially, and whether collecting it at all exposes you to liability. Here is how the spectrum breaks down:

• CC0 (Creative Commons Zero): Essentially, it is a public domain dedication. The author waives all rights. You can scrape it, redistribute it, use it commercially, and modify it.

• CC-BY (Creative Commons Attribution): It requires you to credit the original source. This means you must clearly state where the data came from, who created it, and link back to the original when you publish or redistribute it. This is the most permissive license after CC0, and it is generally easy to comply with.

• CC-BY-SA (Share-Alike): This carries the same attribution requirement as CC-BY, but adds a condition: any derivative work you publish must carry the same license. In practice, this means you cannot fold a CC-BY-SA dataset into a proprietary product and lock it down.

• CC-BY-NC (Non-Commercial): It also requires attribution, but restricts commercial use entirely. You can use the data for research, journalism, or personal projects, but the moment money is involved, you need a separate agreement with the data owner.

• ODbL (Open Database License), used by OpenStreetMap: It requires both attribution and share-alike, specifically for databases. It is worth noting that ODbL distinguishes between the database itself and the contents. Basically, you can use individual facts freely, but any public use of the database as a whole must comply with the license terms.

And then there is the grey zone, which is where most scraping engineers actually operate: data that is publicly accessible but carries no explicit license. Common cases are government portals, academic repositories, open court records, and municipal datasets. This is a huge portion of what people call “open data”. And here is the thing that matters for scraping professionals: no license does not mean free to use. In most jurisdictions, the absence of a license means the default copyright law applies. Which means the creator reserves all rights.

So before you write a single line of scraper code, the first question is not “Can I access this?” but “Under what terms am I allowed to use what I access?”

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

Where the Ethical (and Legal) Risks Hide

Once you have cleared the license question, there are still several risk areas that are easy to overlook:

• License violations: This is the most obvious one. If a dataset requires attribution and you redistribute it without crediting the source, you are in breach. If it has a non-commercial clause and you use it in a commercial product, it’s the same story. These are the kind of things that generate cease-and-desist letters.

• PII embedded in “open” datasets: This is a subtler and arguably more dangerous problem than license violation. Consider open court records: they are public by design, but they contain names, addresses, and sometimes sensitive personal details. Census microdata, even when anonymized at the aggregate level, can contain individual-level records. For example, the GitHub commit history is public, but it contains email addresses, which is personal data. So, the fact that data was made public by someone else does not strip it of its privacy implications when you collect, aggregate, and store it.

• Jurisdictional complexity: A dataset hosted on a European government portal carries GDPR obligations even if you are scraping it from the United States. The GDPR applies based on where the data subjects are located, not where the scraper is running. If you are collecting data about EU residents, you are in GDPR territory regardless of your own geography.

• The aggregation problem: This is probably one of the most underappreciated risks in the scraping industry. Individually, a dataset of names, a dataset of addresses, and a dataset of employment records might each be harmless and openly licensed. But combine them, and you have created a detailed profile of real people. This is something that privacy regulations were specifically designed to prevent.

The Infrastructure Problem: Open Data Portals Are Not Built for Scrapers

Many scraping engineers come to open data with habits built on commercial targets. That experience can be misleading, because the infrastructure behind open data portals is completely different.

When you scrape a large e-commerce website or a major social media platform, you are hitting servers that are engineered to handle millions of requests per day, backed by CDNs, load balancers, and dedicated anti-bot teams. In other words, they can take a (hard) hit.

On the other hand, a municipal open data portal, a university’s research repository, or a small NGO’s dataset hosting is an entirely different story. This means that a scraper that would barely register as noise on Amazon’s servers could genuinely degrade performance for a public data portal serving thousands of researchers.

This is why scraping open data portals aggressively is arguably more unethical than doing the same to a commercial target. You are not fighting a corporation’s anti-bot system. You are potentially taking down a public resource that other people depend on.

A Four-Step Framework for Scraping Open Datasets Without Breaking Rules or Infrastructure

Every risk outlined above has a straightforward mitigation, but only if you apply it at the right point in your workflow. The mistake most scraping engineers make is treating these as afterthoughts: checking the license after the scraper is already built, thinking about PII after the data is already stored. Let’s discuss a framework that inverts this.

License-First Workflow: Read Before You Scrape

The fix for the license problem is simple in principle, even if it requires discipline in practice: make license verification the first step of your workflow.

Most well-maintained open data portals provide license information in one of these three places: a LICENSE file in the dataset’s root directory, a metadata field in the dataset’s API response, or the dataset’s documentation page. Here is a quick reference for what the licenses described above mean for your use case:

When there is no license, the safe default is not to scrape and redistribute without seeking explicit permission from the dataset owner. A short email asking for clarification is a sign of professionalism.

Check the TWSC YouTube Channel

Prefer APIs and Bulk Downloads Over Scraping

This is a rule that experienced scraping engineers sometimes forget because they are so used to reaching for their scraper toolkit: always check for an official API or bulk download endpoint before writing a scraper.

Most serious open data portals expose REST APIs or provide direct bulk download links. Using these is better in every dimension: it is faster, more reliable, more respectful of the server, and often gives you cleaner, structured data than you would get from parsing HTML.

Your workflow should be:

1. Check the portal’s documentation for an API.

2. Check for a Sitemap or structured data endpoint (as discussed in our article on robots.txt and its implications).

3. Check for bulk download links (CSV, JSON, Parquet).

4. Only fall back to HTML scraping if none of the above exist.

Scraping should be your last resort, not your first instinct.

Responsible Scraping Behavior for Open Infrastructure

When scraping is genuinely the only option, the rules of polite scraping apply. But in the case of open data portals, you should apply a higher standard than you would on a commercial target.

As covered in “best practices for ethical web scraping”, respecting rate limits, introducing delays between requests, and using a descriptive User-Agent are baseline requirements. But for open data portals, you should go further because of their weaker infrastructure. Below are additional rules you should consider:

• Respect Crawl-delay strictly: Even if major crawlers ignore it, on underfunded infrastructure, that directive is a good signal about server capacity.

• Cache responses locally: If you need to re-run your scraper for testing or debugging, you should not be hitting the server again. Cache what you have already fetched.

• Scrape during off-peak hours: For public portals serving researchers and government users, off-peak typically means nights and weekends in the portal’s local timezone.

• Scrape only what you need: This sounds obvious, but it’s easy to over-collect data “just in case”. However, for open portals, remember that every unnecessary request is a cost imposed on a public resource that stays online on an underfunded infrastructure.

Handling PII in Open Datasets

PII stands for Personally Identifiable Information. This refers to any data that can be used, alone or in combination with other data, to identify a specific individual. Think names, email addresses, phone numbers, but also subtler things like IP addresses or device IDs.

The reality is that most well-maintained open data portals go through a review process before publication, and raw PII in open datasets is not as common as you might think. The most common cases where PII can slip through are quite specific: older government datasets published before modern privacy review processes, improperly anonymized academic research deposits, or crowdsourced datasets where contributors included personal details voluntarily.

In such specific cases, the real risk for most scraping engineers is at the aggregation level. A dataset of names, a dataset of ZIP codes, and a dataset of employment records might each be perfectly clean and openly licensed in isolation. But combine them, and you have built a detailed profile of real individuals. This is something that privacy regulations like the GDPR and CPRA were specifically designed to prevent. And once you collect, store, and process that combined data, you become responsible for it, regardless of where it originally came from.

The key principle remains the usual one: identify and handle PII at collection time. Here is a schema you can use to audit the fields that are likely to contain PII:

• Direct identifiers: names, email addresses, phone numbers, national ID numbers, passport numbers, and social security numbers. These are the clearest cases as they point to a specific individual on their own, without needing to be combined with anything else. If you see these fields in a dataset, there is no ambiguity: you are dealing with PII.

• Quasi-identifiers: dates of birth, ZIP codes, job titles, gender, ethnicity, and salary ranges. None of these identify a person on their own, but they become dangerous in combination. A classic example is aggregating just three fields—say date of birth, gender, and ZIP code: this is enough to uniquely identify a great portion of the population in a country.

• Sensitive categories under GDPR: health and medical data, political opinions, religious or philosophical beliefs, biometric data, genetic data, sexual orientation, and trade union membership. This is a legally distinct class that carries stricter obligations regardless of context. In other words, you cannot process this data based on legitimate interest alone. You need explicit consent or another specific legal basis, and the bar is significantly higher than for ordinary PII.

For each PII field, decide upfront: do you need it? If not, drop it at collection time. If you do need it, apply pseudonymization (replacing the identifier with a reversible token) or anonymization (irreversible removal or generalization) before storage.

Need help with your scraping project?

Python Implementation: Putting the Full Responsible Scraping Pipeline Into Code

Principles are only useful if they translate into implementation. Below are two concrete components you can adapt for your own pipelines:

• Checking a dataset’s license before downloading anything, using CKAN’s metadata API, with a practical fallback strategy for portals that don’t run CKAN.

• Running PII detection at collection time, using field-level schema classification, with an honest discussion of where that approach has limits.

Note that the examples below omit an API-first fetch pattern and a polite scraper skeleton, even though they are covered in the framework section above. This is because those are problems with well-known, straightforward solutions that every scraping engineer should be aware of. The idea of the following sections is to provide you with lesser-known solutions, to help you get ideas to apply to your pipelines.

Checking a Dataset’s License Programmatically

Many open data portals are built on CKAN, an open-source data management system used by governments and enterprises. CKAN exposes a REST API that includes license metadata, which makes programmatic license checking straightforward.

Here is how to query a CKAN-based portal and extract license information before proceeding:

import requests

def check_dataset_license(portal_base_url: str, dataset_id: str) -> dict:
    """
    Queries a CKAN portal API to retrieve license information
    for a given dataset before any scraping begins.
    """
    api_url = f"{portal_base_url}/api/3/action/package_show"
    params = {"id": dataset_id}

    response = requests.get(api_url, params=params, timeout=10)
    response.raise_for_status()

    data = response.json()
    result = data.get("result", {})

    license_info = {
        "dataset_name": result.get("title", "Unknown"),
        "license_id": result.get("license_id", "Not specified"),
        "license_title": result.get("license_title", "Not specified"),
        "license_url": result.get("license_url", "Not specified"),
    }

    return license_info

# Example: querying the UK government's open data portal
portal = "<https://data.gov.uk>"
dataset = "road-accidents-safety-data"

license_info = check_dataset_license(portal, dataset)

print(f"Dataset: {license_info['dataset_name']}")
print(f"License: {license_info['license_title']}")
print(f"License ID: {license_info['license_id']}")
print(f"License URL: {license_info['license_url']}")

Which outputs the following:

Dataset: Road Safety Data
License: UK Open Government Licence (OGL)
License ID: uk-ogl
License URL: <https://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/>

With this information in hand, you can make an informed decision before a single byte of dataset content is downloaded. Specifically, you can directly check the government license page. The image below partially shows the license page:

But what if the portal you need to scrape doesn’t run CKAN? Not all open data portals do… Socrata (used by many US city and state governments), DKAN, and custom-built portals each have different or no metadata APIs. In those cases, your fallback options are the following:

• Check for a LICENSE or METADATA file in the dataset’s root directory or bulk download package. Many portals include one.

• Look for a <link rel=”license”> tag in the dataset’s HTML page, which some portals emit as structured metadata.

• Check the portal’s documentation or “About” page, where license terms are often stated globally for all datasets.

If none of the above yield a clear answer, treat the license as unknown and do not redistribute without seeking explicit written permission from the dataset owner. A short email asking for clarification is a professional move.

PII Detection at Scrape Time

In this case, the approach depends heavily on what you actually know about the data you need to scrape. Two situations you will encounter in practice, each calling for a different strategy:

• You know the schema: If you are retrieving structured data, field-level detection is the right approach. You know which fields are likely to carry PII, so you can target them directly. This is faster, more precise, and produces far fewer false positives than running a general NER model over free text.

• You have no schema: For unstructured data, NER-based detection is a reasonable starting point, but go in with realistic expectations. A common solution is using spaCy’s en_core_web_sm, which is a small model trained on news text, so don’t expect it to do miracles for you. Another approach, which can give way better results, is using LLMs to give a structure to unstructured text.

For the structured case, here is a field-level PII detection pipeline:

import re
import hashlib
from dataclasses import dataclass, field
from typing import Any

# Fields that are unambiguously PII on their own
DIRECT_IDENTIFIER_FIELDS = {
    "name", "full_name", "first_name", "last_name",
    "email", "email_address",
    "phone", "phone_number", "mobile",
    "ssn", "national_id", "passport_number",
    "ip_address", "device_id"
}

# Fields that are not PII alone but dangerous in combination
QUASI_IDENTIFIER_FIELDS = {
    "date_of_birth", "dob", "birth_date",
    "zip_code", "postcode", "zip",
    "gender", "sex",
    "job_title", "occupation",
    "salary", "income",
    "ethnicity", "race"
}

# Regex patterns for validating suspected PII values at the content level
EMAIL_PATTERN = re.compile(r"[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\\.[a-zA-Z0-9-.]+")
PHONE_PATTERN = re.compile(r"\\b(\\+?\\d[\\d\\s\\-().]{7,}\\d)\\b")

@dataclass
class FieldAudit:
    field_name: str
    classification: str    # "direct", "quasi", or "clean"
    original_value: Any
    processed_value: Any   # pseudonymized, generalized, or original
    action_taken: str       # "pseudonymized", "generalized", "dropped", "kept"

def pseudonymize(value: Any) -> str:
    """
    Replaces a PII value with a consistent, reversible token.
    Using a hash means the same value always produces the same token,
    which preserves referential integrity across records (e.g., you can
    still count unique users without knowing who they are).
    In production, use an HMAC with a secret key instead of plain SHA-256.
    """
    return hashlib.sha256(str(value).encode()).hexdigest()[:16]

def generalize_date(value: str) -> str:
    """
    Reduces a full date of birth to a birth year only.
    A simple but effective generalization for quasi-identifiers.
    """
    # Handles common formats: YYYY-MM-DD, DD/MM/YYYY, MM/DD/YYYY
    match = re.search(r"\\b(19|20)\\d{2}\\b", str(value))
    return match.group(0) if match else "UNKNOWN_YEAR"

def audit_record(record: dict) -> tuple[dict, list[FieldAudit]]:
    """
    Processes a single structured record field by field.
    Returns a cleaned record and a full audit trail of what was done to each field.

    Strategy:
    - Direct identifiers: pseudonymize (preserve referential integrity)
    - Quasi-identifiers: generalize where possible, pseudonymize otherwise
    - Everything else: pass through unchanged
    """
    clean_record = {}
    audit_trail = []

    for field_name, value in record.items():
        normalized = field_name.lower().strip()

        if normalized in DIRECT_IDENTIFIER_FIELDS:
            processed = pseudonymize(value)
            audit_trail.append(FieldAudit(
                field_name=field_name,
                classification="direct",
                original_value=value,
                processed_value=processed,
                action_taken="pseudonymized"
            ))
            clean_record[field_name] = processed

        elif normalized in QUASI_IDENTIFIER_FIELDS:
            # Apply field-specific generalization where we can
            if normalized in {"date_of_birth", "dob", "birth_date"}:
                processed = generalize_date(value)
                action = "generalized"
            else:
                # For other quasi-identifiers, pseudonymize as a safe default
                processed = pseudonymize(value)
                action = "pseudonymized"

            audit_trail.append(FieldAudit(
                field_name=field_name,
                classification="quasi",
                original_value=value,
                processed_value=processed,
                action_taken=action
            ))
            clean_record[field_name] = processed

        else:
            # Field is not in either PII list — pass through, but still
            # run a regex check on string values as a safety net
            if isinstance(value, str):
                if EMAIL_PATTERN.search(value) or PHONE_PATTERN.search(value):
                    # Unexpected PII in a non-PII field: flag it and pseudonymize
                    processed = pseudonymize(value)
                    audit_trail.append(FieldAudit(
                        field_name=field_name,
                        classification="direct",
                        original_value=value,
                        processed_value=processed,
                        action_taken="pseudonymized (unexpected PII in non-PII field)"
                    ))
                    clean_record[field_name] = processed
                    continue

            audit_trail.append(FieldAudit(
                field_name=field_name,
                classification="clean",
                original_value=value,
                processed_value=value,
                action_taken="kept"
            ))
            clean_record[field_name] = value

    return clean_record, audit_trail

def process_records(records: list[dict]) -> list[dict]:
    """
    Runs field-level PII detection and handling across a list of records.
    Prints an audit summary for any record where PII was found.
    """
    clean_records = []

    for i, record in enumerate(records):
        clean_record, audit_trail = audit_record(record)
        pii_fields = [a for a in audit_trail if a.classification != "clean"]

        if pii_fields:
            print(f"Record {i}: PII detected and handled in {len(pii_fields)} field(s):")
            for audit in pii_fields:
                print(f"  [{audit.classification.upper()}] {audit.field_name} "
                      f"→ {audit.action_taken}")

        clean_records.append(clean_record)

    return clean_records

# Example: a batch of records from a scraped open dataset
records = [
    {
        "record_id": "A001",
        "name": "Jane Doe",
        "date_of_birth": "1985-03-22",
        "zip_code": "SW1A 1AA",
        "incident_type": "Road accident",
        "severity": "Slight"
    },
    {
        "record_id": "A002",
        "name": "John Smith",
        "date_of_birth": "1973-11-04",
        "zip_code": "EC1A 1BB",
        "incident_type": "Road accident",
        "severity": "Serious",
        # An email that slipped into a free-text notes field
        "notes": "Witness contact: witness@example.com"
    }
]

clean = process_records(records)

The output is the following

Record 0: PII detected and handled in 3 field(s):
  [DIRECT] name → pseudonymized
  [QUASI] date_of_birth → generalized
  [QUASI] zip_code → pseudonymized
Record 1: PII detected and handled in 4 field(s):
  [DIRECT] name → pseudonymized
  [QUASI] date_of_birth → generalized
  [QUASI] zip_code → pseudonymized
  [DIRECT] notes → pseudonymized (unexpected PII in non-PII field)

A few things worth calling out in this implementation:

• Pseudonymization preserves referential integrity: Because the same input always produces the same hash token, you can still count unique individuals, join records, or track entities across datasets, without storing the raw PII. In production, replace the plain SHA-256 with an HMAC keyed on a secret, so tokens cannot be reversed by someone who also has access to the hashing algorithm.

• The regex safety net on non-PII fields: This catches the common real-world case where PII slips into a free-text or notes field that your schema classification didn’t anticipate. It is not foolproof, but it catches the obvious cases.

• The audit trail is intentional: Every field-level decision is logged. If you are ever asked to demonstrate that your collection process handled PII responsibly, you have a record of exactly what was done to each field in each record.

Conclusion

Open data is a shared resource, and how you interact with it says something about you as a professional. In this article, you learned what “open” means in the context of data scraping and how you should treat it if you want to be an ethical scraper.

So, let us know: Did we miss something? What’s your approach to handling open datasets in your scraping projects? Let’s discuss in the comments.

No wonder nearly half a trillion dollars are spent every year by financial firms on technology. Now, you probably don’t have that kind of money in the first place (and if you do, you don’t need to invest much anyway), but you might still want to collect financial data for personal use, research, academic projects, backtesting, or even just for selling it to industry giants.

No matter what you want to do with scraped financial data, there are a few pivotal tips to understand before embarking on this journey, which is exactly what I will explain here!

In this blog post, I will show why web scraping and finance are a match made in heaven and cover everything you need to know about retrieving both historical and real-time financial data from the web.

Web Scraping + Finance: A Happy Marriage

Before diving into web scraping for finance, let me explain why this is such a powerful approach and the advantages you can gain from it.

Finance Runs on (Web) Data

If there’s one thing that’s become clear over the past decade, it’s this: finance runs on data!

Financial institutions process massive volumes of market, customer, and transactional data every single day. In finance, data powers everything, from investment strategies to risk management. And the stakes are high, as bad data alone costs organizations an average of $12.9 million per year!

Data drives real-time decision-making, predictive modeling, and scenario planning. Finance teams feed that data into pipelines built around statistical analysis, machine learning, and AI to identify patterns, forecast market movements, and manage uncertainty in increasingly complex environments.

Now, here’s the central question we, web scraping enthusiasts, are all interested in: where does most of that data actually come from? A big portion of it comes from the web (not that surprising, uh?).

I’m talking about news sites, financial portals, company pages, official exchange websites, regulatory filings, institutional reports, and more. The web is essentially the largest and most dynamic data source available for financial purposes.

That’s exactly why web scraping in finance isn’t just useful. It’s foundational!

Before proceeding, let me thank NetNut, the platinum partner of the month. Their set of solutions cover all your needs for scraping.

Visit Netnut

Benefits of a Data-Driven Approach in Finance

Keep in mind that it’s not just big corporations or financial firms that benefit from data. Even individual retail investors can leverage financial data scraping to gain an edge. In particular, the main advantages include:

• Informed decisions: Access to accurate historical data supports smarter investment decisions, while real-time data enables more solid trading choices.

• Market trend insights: Spot patterns and emerging trends before the wider market does.

• Risk management: Identify potential risks early and adjust strategies proactively.

• Portfolio optimization: Fine-tune asset allocation based on backtesting and up-to-date market and company data.

• Efficiency and speed: Automate data collection, reducing time spent on manual research.

I mean, financial firms wouldn’t be spending over $495 billion a year (yeah, you read that right!) on technology (mostly built around collecting, processing, and leveraging data) if it didn’t give them a real edge!

Getting vs Selling Financial Web Data: High-Level Overview

There’s no doubt that financial firms invest billions into data. But what about you, as a web scraping expert, how can you leverage financial data for potential gain? There are two high-level approaches:

1. For yourself or your company: Build custom web scraping pipelines to gather data from multiple sources. Use it to feed investment models, AI agents, trading algorithms, or analytics pipelines. This is usually highly tailored to your strategies, risk appetite, or operational goals.

2. To sell to financial services: Collect, aggregate, and potentially enrich data from various sources to sell. You can offer broad datasets for many clients or fully customized solutions for a specific customer’s needs.

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

How to Approach Financial Data Scraping: Historical vs Real-Time

When it comes to finance, the web is packed with countless data fields and categories (e.g., news, stock prices, filings, analyst reports, and more). It’s a huge industry, and almost anything can be scraped!

At a high level, though, the key distinction for web scraping is simple: the financial data you want to collect is either historical or real-time. That’s what actually makes a difference in the approach to data scraping.

In the following chapters, I’ll dive deeper into each of the two categories of financial data. I’ll cover which fields are most interesting to scrape, where to find them, and how to collect them efficiently and effectively.

For now, start with a brief introduction to historical and real-time financial web data scraping!

Historical Financial Web Data

This includes all past financial data collected from the web, from historical stock prices to inflation rates and archived news. It’s used for analysis supporting long-term investment decisions.

👍 Pros:

• Enables backtesting of investment and trading strategies.

• Easier to scrape, as it isn’t time-sensitive.

• Data itself is stable and doesn’t change over time…

👎 Cons:

• …but the web pages displaying it (e.g., in tables and static charts) can still change, breaking your static parsing logic.

• Misses recent market shifts or breaking events.

• Data completeness varies across websites, often requiring aggregation from multiple sources.

Real-Time Financial Web Data

This includes live financial data extracted from the web, such as stock prices, market news, order books, etc. It’s employed for trading and short-term investment decisions.

👍 Pros:

• Enables fast, data-driven trading decisions.

• Captures live market movements and breaking news.

• Can be passed to AI agents and pipelines directly, as it tends to require minimal preprocessing.

👎 Cons:

• Harder to scrape reliably due to latency constraints and rate limits.

• Requires robust infrastructure for real-time ingestion and analysis, as every second counts.

• Data storage can grow rapidly because new data arrives continuously.

Mastering Historical Financial Data Scraping

As promised, let me guide you through the world of scraping historical financial data from the web.

Main Types of Historical Financial Web Data

The most important types of historical financial data you can retrieve from websites are:

• Historical stock and commodity prices: Open, high, low, close (OHLC) prices and trading volumes for stocks, ETFs, indices, and commodities, used for time-series analysis, modeling, and predictions.

• Summary info and infographics: Stock profiles, key metrics, and past indicators (e.g., P/E, EPS, moving averages), presented in dashboards or visual charts for quick insights.

• Macroeconomic indicators: Inflation, GDP, interest rates, unemployment, CPI, and PCE data, essential for understanding economic cycles and long-term market behavior.

• Financial statements: Company filings (income statements, balance sheets, cash flow), utilized for fundamental analysis and valuation models.

• News data: Archived headlines and press releases analyzed via NLP to correlate past market movements with specific events and sentiment shifts.

• ESG scores and sustainability reports: Historical environmental, social, and governance metrics employed to assess how “green” or ethical a company has been over time.

• Alternative data: Non-traditional datasets like web traffic, social media, satellite imagery (e.g., new headquarters or production plants), or credit card data for early performance signals.

Most Popular Targets

Also, if you’re interested in how to scrape historical data from the Wayback Machine, read my previous guide for this newsletter!

Scraping Techniques

Typical examples of historical financial data include lists of open, high, low, and close prices for a given stock:

Or, another example, the historical returns of a specific index (.e.g, SP500) over time:

These cases fall into the category of table-based data scraping, one of the most common web scraping scenarios. You’re probably already familiar with it, so there’s no need to go too deep here. Scraping older news and media can be slightly more challenging due to the unstructured nature of the target data, but it’s still a simple task.

At a high level, the process for getting historical finance data via web scraping follows a standard workflow:

1. Visit the target web page, either via an HTTP client or a browser automation tool.

2. Parse the page using an HTML parser, either directly or after rendering in a controlled browser.

3. Select the HTML elements of interest and extract the data.

4. Store the scraped data in your desired format (e.g., XLS, CSV, JSON) or in a database.

The main challenges involve generic anti-scraping mechanisms, such as CAPTCHAs, WAFs, IP bans, as well as browser, TLS, and device fingerprinting.

Best Practices

Based on my experience with financial web scraping, especially when focusing on historical data, these are the tips you should apply:

• Normalize and validate data: Standardize formats (dates, currencies, units) and validate across sources to catch inconsistencies early.

• Be cautious with AI parsing: Avoid using AI for automatically parsing structured data (tables, metrics, structured fields). It can introduce subtle errors and hallucinations, so prefer deterministic parsing. Harness AI mainly for retrieving unstructured text like news.

• Store raw HTML snapshots: Always keep the original page HTML. It lets you re-parse data later and extract new signals without re-scraping.

• Avoid single-source bias: When scraping news or market analysis pieces, pull data from multiple sources to reduce bias and improve reliability.

• Handle pagination properly: Many sites split historical data across pages or date ranges. Make sure your scraper fully traverses them all.

• Respect rate limits and retries: Even for historical data, implement retries and throttling to avoid blocks and incomplete datasets.

Check the TWSC YouTube Channel

Understanding Real-Time Financial Data Scraping

This is where things get a bit more interesting. Let me introduce you to real-time financial scraping!

Main Types of Real-Time Financial Web Data

The most relevant types of real-time financial web data are:

• Live price tickers: Continuously updated “last trade” prices and bid/ask spreads for stocks, crypto, and forex, used to detect breakouts and short-term trading opportunities.

• Order book and market depth: Incoming buy/sell orders, liquidity levels, and spreads, fundamental for execution strategies and high-frequency trading.

• Breaking news: Immediate updates and announcements that trigger sentiment models as soon as key figures (CEOs, central banks, governments) release information.

• Corporate event triggers: Monitoring press releases or SEC feeds for earnings surprises, M&A rumors, or sudden executive changes.

• Social media signals: Tracking ticker mentions on platforms like Reddit or X to detect retail-driven momentum, hype cycles, or panic selling in near real time.

• Institutional “whale” activity: Observing large trades or major wallet movements (especially in crypto) to identify where significant capital is flowing.

• Alternative digital signals: Web traffic spikes, app store ranking changes, or “out of stock” alerts on retail sites as proxies for real-world demand.

As you can tell, this category is more varied than historical financial data, including social media tracking and other less conventional practices. Thus, the sources to monitor for live financial web scraping can be less standardized and intuitive.

Most Popular Targets

Scraping Techniques

Imagine applying a traditional scraping pattern to real-time financial data. You send a request to a target site, extract a stock price, and repeat the operation every few seconds or even milliseconds.

The problem is latency. By the time the server responds, the page is rendered or parsed, the target data field is collected, and stored or sent to your pipeline, that piece of data is already outdated.

On top of that, this approach requires a crazy number of requests in a very short time. That increases the risk of triggering rate limiting or even IP bans. You might think proxies solve that through IP rotation, but most proxy networks introduce additional latency, often 2/3/5 seconds per request. In real-time scenarios, that delay is simply not acceptable!

Even if you switch to faster or dedicated proxies, you may end up with a smaller IP pool, which increases the likelihood of those IPs getting blocked.

A more advanced idea is to rely on browser automation and keep a page open, capturing updates as they happen. This is smarter, but still problematic. Long-lived sessions with little or no user interaction are highly suspicious and can easily trigger anti-bot systems. Plus, browser automation at scale tends to be flaky, not really reliable for persistent connections.

Long story short, scraping real-time financial data this way quickly turns into a losing game.

The solution? Stop targeting the data presentation layer in HTML and instead go directly to the data source!

API/WebSocket Scraping as The Solution

Web pages showing real-time financial data aren’t doing anything magical. Behind the scenes, they either poll APIs at regular intervals or (more commonly) maintain a persistent connection via WebSockets to receive continuous updates. The page simply renders that incoming data.

As a result, a much better approach is to intercept and replicate those data flows. You can do this through AJAX/API request inspection or WebSocket sniffing. Open the browser developer tools, go to the “Network” tab, and check where the data is coming from.

If it’s an API call, you’ll see it under the “Fetch/XHR” tab:

If it’s a WebSocket, you’ll find it under the “Socket” tab:

Once identified, replicate those API calls or connect directly to the WebSocket in your scraping script. This gives you access to near real-time financial data in a structured format (typically JSON) without the overhead of parsing HTML.

Of course, that’s not trivial. WebSockets require proper anti-bot bypass, and APIs may still enforce rate limits, tracking, and TLS fingerprinting protections. However, this approach is generally faster, more reliable, and much easier to maintain than scraping rendered pages!

And What About Live News or Social Media Scraping?

When it comes to news, if available, it makes sense to connect to public RSS feeds exposed by websites to monitor updates. This allows you to trigger scraping only when new and relevant content is published, instead of constantly polling pages unnecessarily.

Otherwise, you can build a polling mechanism that periodically checks news sites, social media platforms, and similar sources to capture fresh data. In these cases, you usually can’t rely on techniques like API or WebSocket scraping, as that’s not how those platforms fetch data.

Instead, you need a solid and robust infrastructure built around speed and efficiency: fast connections, high-quality proxies, optimized parsing, and lightweight requests. The goal is to minimize latency while maintaining reliability at scale.

Best Practices

Scraping real-time financial data is a demanding art, but it becomes easier with the following best practices:

• Prefer APIs and WebSockets over HTML parsing: Whenever possible, save time by extracting data directly from the underlying APIs or WebSocket streams utilized by web pages instead of scraping data from rendered pages.

• Choose clean, structured sources: Prioritize endpoints that return well-formatted JSON to minimize preprocessing and reduce latency.

• Stream data into pipelines immediately: Send incoming data directly to processing pipelines for real-time insights, while storing it in parallel for later analysis.

• Use specialized AI for sentiment analysis: Prefer AI/ML models tuned for finance/social media, as Reddit and X content often include slang, memes, and non-standard language.

• Optimize browser automation: Configure Playwright, Selenium, or similar browser automation tools to block images, stylesheets, and fonts. This reduces bandwidth usage and significantly speeds up rendering time.

• Design for low latency: Optimize your stack (async requests, streaming ingestion, fast JSON parsers) to minimize delays, as even milliseconds matter.

• Prefer high-quality premium proxies: Count on proxy providers with a proven track record of fast, stable connections to minimize latency and avoid disruptions.

• Time-synchronize everything: Append timestamps to all scraped data to enable time-series analysis and accurately reconstruct events.

• Build fault-tolerant systems: Expect disconnections (especially with WebSockets) and issues, so add reconnection logic and configure fallback data sources.

Need help with your scraping project?

Top 5 Open-Source Financial Web Scraping Libraries

Below is a selected set of interesting, fully open-source libraries, packages, and projects for simplified financial web scraping:

Conclusion

Here, I’ve gone through the rabbit hole of financial web scraping, the task of collecting finance-related data from the Internet. This is one of the main use cases of corporate web scraping, powering enterprise data pipelines for decision-making and market analysis.

As you’ve seen, the main difference in the approach comes down to whether you’re targeting historical or real-time data. The first follows standard web scraping practices you’re likely already familiar with. The second is trickier and requires more advanced techniques.

I hope you found this helpful and insightful. If you have questions, feel free to share them in the comments below!

This time, we wanted AWS WAF on its own, in front of a target with nothing else in front of it, and we wanted to see what changes when the 2024 Scrapy-Playwright stack is replaced with the 2026 toolbox.

The target we picked is imdb.com. It is an Amazon subsidiary, runs a standard AWS WAF deployment, and Wappalyzer confirms that there are not others antibot on the website. That makes IMDB a perfect use case for our article.

Before proceeding, let me thank NetNut, the platinum partner of the month. Their set of solutions cover all your needs for scraping.

Visit Netnut

Today we’ll test three Python HTTP clients with strong TLS fingerprint impersonation: curl_cffi, the newer httpx-curl-cffi, and Rust-backed rnet. Each one produces a TLS handshake indistinguishable from real Chrome. Is that enough to scrape an AWS WAF target without spinning up a browser? And if not, what is the smallest browser step that gets us past the gate so the rest of the work can run on a cheap HTTP client?

The tools we used

Four libraries are in scope. Three are HTTP-only, one runs a real browser.

curl_cffi is a Python binding for the curl-impersonate patched curl. It exposes a requests-like API and ships impersonation profiles for recent Chrome, Firefox, and Safari builds and works at the TLS layer. JA3 and JA4 fingerprints match the impersonated browser, along with HTTP/2 settings and header order. We tested with chrome142, the latest Chrome profile in version 0.14.0.

httpx-curl-cffi is a transport for httpx that delegates the actual HTTP work to curl_cffi. While it does not add new fingerprinting capability, it implements the httpx programming model: sync Client, async AsyncClient, event hooks, the same response object you get from the rest of an httpx-based codebase. We tested with the Chrome profile and default_headers=True.

rnet is a Rust HTTP client with Python bindings. It implements its own impersonation stack rather than wrapping curl-impersonate. The enum rnet.Impersonate exposes a wide range of Chrome, Firefox, Safari, Edge, Opera, and OkHttp profiles. We tested with Chrome137.

Scrapling is the only browser-driven tool in the set. Our Scrapling: A Complete Hands-On Guide goes through the library in depth, with Cloudflare as the test target. Its StealthyFetcher drives a stealth-patched Chromium that runs JavaScript and applies fingerprint countermeasures. The library README only advertises Cloudflare Turnstile, but the same machinery handles AWS WAF’s challenge too.

Your scraping workflows deserve a proxy infrastructure that just works. With Swiftproxy on your side, consistency is built-in.

Try Swiftproxy today

How AWS WAF protects IMDB

A quick intro of the system helps interpret the results that follow. AWS WAF is not a dedicated anti-bot platform like DataDome or Kasada. It is a general-purpose web application firewall with a bot-control module that operators can enable per rule. When the bot-control rule is in challenge mode, AWS WAF inserts a single JavaScript gate at the start of a session.

A request without a valid cookie returns HTTP 202 with x-amzn-waf-action: challenge and a short HTML body. The body contains window.gokuProps containing three base64 blobs (key, iv, context), a <script src> pointing to a customer-specific URL on *.token.awswaf.com, and a small inline script that calls AwsWafIntegration.saveReferrer(), AwsWafIntegration.checkForceRefresh(), and AwsWafIntegration.getToken(). The remote challenge.js tests the browser environment, posts a validation payload back to AWS, and on success, the response sets Set-Cookie: aws-waf-token=.... The inline script then reloads the page, and the second request, now carrying the token, gets the real content.

This works very differently from systems that score every request. Once the token is in our jar, AWS WAF lets us through with no further behavioral checks beyond IP reputation and rate limits.
What we want to discover with this article is if we’re able to bypass AWS WAF with “convincing” requests, with a proper TLS fingerprint and set of headers, or if we need a JS rendering engine.

Check the TWSC YouTube Channel

Test setup

As always, the code can be found in our GitHub repository reserved to paying users, inside the folder 104.IMDB. If you’re not able to access the repository, please use this form to request access.

The libraries we pinned at the time of writing are curl_cffi==0.14.0, httpx==0.28.1, httpx-curl-cffi==0.1.5, rnet==2.4.2, scrapling==0.4.7. Python is 3.11.

Each HTTP test creates a GET against two URLs: the IMDB home page

https://www.imdb.com/

and a title page https://www.imdb.com/title/tt0111161/. We use two URLs to confirm the challenge fires the same way on both, not only on one entry point. We do not follow redirects (follow_redirects=False) because the AWS WAF response is a 202 with content rather than a redirect, and we want to see it raw.

We capture status code, HTTP version, the full response headers, any cookies, body length, and the first 600 characters of the body, and we saved everything to JSON under aws_waf_imdb/responses/ for later inspection.

The baseline probe in probe_plain.py uses an unmodified httpx.Client(http2=True) with a generic Chrome User-Agent header and the standard Accept headers. This is the control: no TLS impersonation, no fingerprint trickery, just a normal Python HTTP client.

Read more

The web contains an extraordinary volume of information, the majority of which is in textual form. Blogs, forums, and newsletters alone generate millions of words of domain-specific knowledge every week. And they’re not the only sources of text on the web.

When you want to get insights from that kind of data, successfully extracting it from the web is only half of the battle, even now that LLMs can use vision to scrape complex visual layouts. The second part of the challenge is structuring this data to get it ready for analytics. Why? Because when you point a scraper at a news article, you get back a wall of text. But you cannot query it. You cannot aggregate it. You cannot feed it reliably into a machine learning pipeline or a database without significant preprocessing.

This article addresses the preprocessing problem of unstructured text when you scrape it from the web. It traces the evolution of solutions from classical NLP to large language models, identifies where each approach breaks down, and proposes a practical architectural solution.

Let’s get into it!

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

What “Unstructured” Really Means in Practice

Unstructured text refers to content that carries no machine-readable schema. The information exists in the data you retrieved from the web, but no field boundaries exist, no consistent labels, and no guaranteed position for any given fact.

The following schema represents the difference between unstructured and structured text (machine-readable schema):

Let’s consider three concrete scraping targets to illustrate what this costs you in practice.

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

News Articles: When Signals Are Buried in Noise

Consider you scraped a Reuters article about an ECB rate decision. The text you get back from the scraper could be something as follows:

European Central Bank decides on rates.
Listen to this article. 2 min audio. 
You might also like: Eurozone inflation hits 3-year low. 
Christine Lagarde announced Thursday a 25 basis point reduction, bringing the main 
refinancing rate to 3.40%. 
SPONSORED: Track macro events with Bloomberg Terminal. 
The decision was widely anticipated after last month's CPI print. Share this article. 
4 comments. John M. writes: this was priced in already

Your raw text contains the article body, a teaser for a related story, a sponsored insertion, and reader comments. The fact you want is buried in there: the ECB cut its main refinancing rate to 3.40% on a specific date. But your extractor gets the full content.

Such a wall of text, which, generally speaking, is way bigger than this and is useless for analytics purposes without preprocessing.

Financial Newsletters: When “Just Under Two Percent” Breaks Your Aggregation

Suppose you scrape a financial newsletter to extract an updated macroeconomic forecast. You need to capture a specific fact. Something like “Goldman Sachs revised its 2026 US GDP growth forecast down to 1.8%”. Your scraper captures the entire page output, which is similar to an article. Similarly to the previous example, the resulting raw text mixes the core facts with boilerplate and unrelated news:

Market Daily Newsletter. November 12.
Jan Hatzius (Goldman Sachs) and his team were out with a note early Tuesday.
SPONSORED: Get 50% off your trading fees today. 
They see tariffs shaving roughly 0.7 points off the baseline. Meanwhile, 
European markets rallied on ECB news. 
Read our full coverage of the Eurozone here.
The revised number now sits just under two percent for the full year.
Subscribe for premium insights.

The text distributes the target fact across the entire document. Also, the wording “just under two percent” requires numerical understanding to say that the text refers to the actual number you were searching for, that is, an exact 1.8%.

Now, imagine generalizing this after scraping hundreds of financial news and newsletters to regroup the information to summarize the numbers. Getting insight would be impossible. Why? Because some sources will give you the actual information you want (growth forecast down to 1.8%), others will use different phrasing to define the trend (”An expected growth under 2 percent”, “a slightly shrinking trend”, etc).

Without a way to create a structure for such data, you can’t get any insights from it.

Job Posting Offers: They Are Always Messier Than They Look

Consider the case when you want to scrape job offers to get an idea of what the market is paying on average for a specific position, given the expected technical skills, and considering the same day-to-day activity. Job offers can have the following ambiguities:

• A sentence might read “3+ years of experience with Python”. This establishes a floor and ignores a ceiling. Alternatively, the text might read “Senior-level candidates only”. This uses qualitative seniority as a proxy for an exact quantitative number.

• Salary breaks in a different direction. One posting can say “$120,000 - $145,000 base”. Another can be “competitive compensation commensurate with experience”. A third could be“€100,000”, which you need to convert to dollars to make an actual comparison.

• Employment type can introduce further ambiguity and difficulties. “Full-time”, “FTE”, “permanent”, and “direct hire” basically mean the same thing but are written differently. Also, the text might specify the role is “Hybrid”, which means multiple different things across companies. It could mean two days in the office. It could mean occasional travel with headquarters-optional rules.

When sites get tough, skip the heavy lifting. Get clean, structured CSV datasets, ready for Excel, BI or your apps

Find your dataset

How Classical NLP Tried to Solve This (and Where It Stopped)

Before large language models were released, the standard answer to this problem was Natural Language Processing. The classical NLP toolkit gave developers a set of tools that could, with enough effort, extract meaningful structure from text using different, but often interconnected, processes like the following:

• Named Entity Recognition (NER): NER is a process used in NLP to extract entities from text corpora. It can particularly identify spans of text as persons, organizations, locations, or dates. An NLP model trained on news corpora, for example, is able to scan an article and tag “Jane Doe” as a person and “Washington D.C.” as a geopolitical entity.

• Part-of-speech tagging: Is a process in which NLP models can identify nouns, verbs, and adjectives. This enables the downstream logic to focus on the right parts of a sentence.

• Dependency parsing: Maps grammatical relationships between words, helping to extract which subject performed which action on which object.

• Relation extraction: Identifies when two co-occurring entities have a specific relationship. For example, a person who was affiliated with an organization, or an event that occurred in a specific location.

Libraries like spaCy, Stanford NLP, and NLTK made these processes largely accessible. But they work well for well-defined, narrow tasks on consistent text domains. The problems and limitations of this solution appear quickly at the edges:

• Domain shift breaks everything: A NER model trained on news articles performs poorly on scientific abstracts. A model tuned for English financial text fails on multilingual content. In other words, every new domain requires retraining, re-labeling, and re-evaluation. These processes are very costly, both in terms of money and time.

• Context is invisible: Classical NLP models operate at the token and sentence level. They have no mechanism for understanding that “Apple” in a technology article refers to a corporation, while “apple” in a nutrition blog refers to a fruit. Disambiguation requires hand-crafted rules or separate classification layers bolted on top (which, again, is costly).

Before NLP, you could basically only use regex (with all the difficulties associated with manually cleaning data, standardizing it, and…using regex!). So, NLP was a genuine (big) step forward: it made large-scale text analysis possible in ways that pure pattern matching never could (which is a way to find patterns in scraped data using AI). But it still required substantial domain expertise, constant maintenance, and produced results that were narrow, fragile, and difficult to generalize.

Check the TWSC YouTube Channel

The Modern Solution: LLMs as Universal Structure Extractors

Large language models fundamentally changed the extraction problem. On the side of the underlying technology, a classical NLP model learns the statistical patterns inside the text. An LLM, instead, learns to understand language. This distinction matters enormously because it opened the doors to the following:

• Context disambiguation that works out of the box: Feed an LLM with a paragraph from a technology article containing the word “Apple” and it will correctly identify it as a company. Feed it with a paragraph from a recipe blog, and it will correctly identify it as a fruit. No separate disambiguation layer. The model resolves ambiguity the same way a human reader does: by reading the surrounding context.

• Semantic equivalence that is understood, not computed: An LLM knows that “$40,” “forty dollars,” “40 USD,” and “forty bucks” all express the same value. You don’t need to instruct it to understand that.

• Implicit information that becomes accessible: A sentence like “the study, conducted over three months at a Boston hospital, found no significant effect” contains a location, a duration, and a finding. An LLM can extract all three without requiring the text to follow any particular structure.

• Domain generalization that requires no retraining: The same LLM that extracts entities from political news articles can extract findings from scientific abstracts, event mentions from cultural journalism, and source attributions from investigative reporting. You just need to change the prompt, not the model.

The practical workflow becomes straightforward:

• You scrape unstructured text from the web.

• You pass the content to an LLM with a prompt that describes what you want to extract.

• The model returns a response.

• You use that response downstream.

This process works. But using LLMs alone introduces a different class of problems:

• Output format is not guaranteed: Ask an LLM to return a price, and it might return $40 in one run, 40 dollars in another, and 40 USD in a third. The model understands the value when it retrieves it from scraped content. But it does not guarantee how it expresses that value unless you explicitly constrain it.

• Required fields can go missing: If the article you extracted the content from does not mention a publication date, the model might omit the field, return null, or return "not mentioned", or invent a plausible date (which is way worse). Each behavior is different, and none of them is predictable without enforcement.

• Hallucination is a real risk: When the model is uncertain, it always generates a plausible answer. For extraction tasks, that means it can invent entity names, fabricate statistics, or fill in missing information with confident-sounding fiction. Without validation, these errors pass into your data, creating issues at the analytics level.

Generalizing all of this, you also get scalability issues because you have no consistency guaranteed. A pipeline processing 10,000 articles requires every output to follow the same schema. But a model that returns slightly different structures across runs cannot feed a database reliably without significant error handling.

In other words, LLMs provide you with the understanding that NLP lacked. But they do not, on their own, provide the structural guarantees that production pipelines require.

Need help with your scraping project?

How to Get Semantic Power and Structural Guarantees at the Same Time: A Practical Approach

One possible solution to the unpredictability of LLM outputs is to separate the two concerns that these models conflate: semantic understanding and structure enforcement.

To do so, you can:

• Use the LLM for what it does well: reading text, resolving ambiguity, extracting meaning, and normalizing inconsistent expressions.

• Use specific libraries to define schemas, enforce types, validate outputs, and reject malformed data before it enters your pipeline.

Below is how this solution works, at a high level:

Let’s see how to implement this process and how the two approaches differ in practice.

The Baseline Approach: A Direct LLM Call (and What It Gives You)

Consider the following content that can come from scraping a news article:

Fed Signals Caution as Inflation Data Disappoints

By Sarah M. Connelly | April 14, 2026 | Economics & Policy

The Federal Reserve signaled on Monday that it remains in no rush to cut interest rates,
after fresh inflation data showed consumer prices rose more than expected last month.
Jerome Powell, speaking at a conference in Washington, said the central bank needs
"greater confidence" that inflation is moving sustainably toward its two-percent target
before reducing borrowing costs.

The Consumer Price Index climbed 3.5 percent in March, up from 3.2 percent in February,
according to the Labor Department. Economists polled by Reuters had forecast a reading
of 3.3 percent. Core CPI, which strips out food and energy, rose 3.8 percent year-over-year.

Markets reacted sharply. The S&P 500 fell nearly one point five percent by midday,
while the yield on the 10-year Treasury note jumped to four point six percent.
Goldman Sachs revised its forecast, pushing back its expected first rate cut from June
to September. JPMorgan analysts said two cuts in 2026 now look "optimistic."

Powell emphasized that the Fed is not considering rate hikes at this stage, but stressed
that the path back to two percent inflation "may take longer than previously thought."
The next Fed meeting is scheduled for the first week of May

To directly pass it to a GPT model, asking it for a precise output, you can use the following code:

import os
import json
from openai import OpenAI

# Scraped content
SCRAPED_TEXT = """
Fed Signals Caution as Inflation Data Disappoints

By Sarah M. Connelly | April 14, 2026 | Economics & Policy

The Federal Reserve signaled on Monday that it remains in no rush to cut interest rates,
after fresh inflation data showed consumer prices rose more than expected last month.
Jerome Powell, speaking at a conference in Washington, said the central bank needs
"greater confidence" that inflation is moving sustainably toward its two-percent target
before reducing borrowing costs.

The Consumer Price Index climbed 3.5 percent in March, up from 3.2 percent in February,
according to the Labor Department. Economists polled by Reuters had forecast a reading
of 3.3 percent. Core CPI, which strips out food and energy, rose 3.8 percent year-over-year.

Markets reacted sharply. The S&P 500 fell nearly one point five percent by midday,
while the yield on the 10-year Treasury note jumped to four point six percent.
Goldman Sachs revised its forecast, pushing back its expected first rate cut from June
to September. JPMorgan analysts said two cuts in 2026 now look "optimistic."

Powell emphasized that the Fed is not considering rate hikes at this stage, but stressed
that the path back to two percent inflation "may take longer than previously thought."
The next Fed meeting is scheduled for the first week of May.
"""

# Define LLM client
raw_client = OpenAI(api_key=os.environ.get("YOUR_OPENAI_API_KEY"))

# Define prompt for the LLM
raw_prompt = """
Extract the following information from the article below and return it as JSON:
- title
- author
- publication_date
- mentioned_organizations
- cpi_march_value
- key_claim
- market_sentiment

Article:
""" + SCRAPED_TEXT

# Get response from LLM
raw_response = raw_client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": raw_prompt}]
)

raw_output = raw_response.choices[0].message.content

# Print results
print(raw_output)

The result will be as follows:

{
  "title": "Fed Signals Caution as Inflation Data Disappoints",
  "author": "Sarah M. Connelly",
  "publication_date": "April 14, 2026",
  "mentioned_organizations": [
    "Federal Reserve",
    "Labor Department",
    "Reuters",
    "Goldman Sachs",
    "JPMorgan"
  ],
  "cpi_march_value": "3.5 percent",
  "key_claim": "The Federal Reserve is in no rush to cut interest rates and needs greater confidence that inflation is moving sustainably toward its two-percent target before reducing borrowing costs.",
  "market_sentiment": "Negative"
}

Now, at first sight, this seems good. The prompt asked the GPT model to create a JSON file with specific values, and the model was able to do so. But two major problems affect the next steps when analyzing this data. They are:

• The publication date is reported as “April 14, 2026”. This is not represented in ISO 8601 format and will break any date parser.

• The CPI is reported as “3.5 percent”, which is a string. Not a number or a float, which is what is required for such data if you want to further analyze it (without any intermediate steps).

So, the LLM was able to give structure to an unstructured text, after being specifically prompted to do so. But it failed at providing the data in the right format. To do so, you have to provide specific guidance to the model.

What Changes When You Define The Schema

To have guarantees on the output format, you can use the following code:

import os
import json
import instructor
from openai import OpenAI
from pydantic import BaseModel, Field
from typing import Optional, Literal

# Scraped content
SCRAPED_TEXT = """
Fed Signals Caution as Inflation Data Disappoints

By Sarah M. Connelly | April 14, 2026 | Economics & Policy

The Federal Reserve signaled on Monday that it remains in no rush to cut interest rates,
after fresh inflation data showed consumer prices rose more than expected last month.
Jerome Powell, speaking at a conference in Washington, said the central bank needs
"greater confidence" that inflation is moving sustainably toward its two-percent target
before reducing borrowing costs.

The Consumer Price Index climbed 3.5 percent in March, up from 3.2 percent in February,
according to the Labor Department. Economists polled by Reuters had forecast a reading
of 3.3 percent. Core CPI, which strips out food and energy, rose 3.8 percent year-over-year.

Markets reacted sharply. The S&P 500 fell nearly one point five percent by midday,
while the yield on the 10-year Treasury note jumped to four point six percent.
Goldman Sachs revised its forecast, pushing back its expected first rate cut from June
to September. JPMorgan analysts said two cuts in 2026 now look "optimistic."

Powell emphasized that the Fed is not considering rate hikes at this stage, but stressed
that the path back to two percent inflation "may take longer than previously thought."
The next Fed meeting is scheduled for the first week of May.
"""

# Validation schema
class ArticleExtraction(BaseModel):
    title: str = Field(description="The article's headline")
    author: Optional[str] = Field(description="Full name of the author if explicitly mentioned")
    publication_date: Optional[str] = Field(description="Publication date in ISO 8601 format (YYYY-MM-DD)")
    mentioned_organizations: list[str] = Field(description="All organizations referenced in the article")
    cpi_march_value: Optional[float] = Field(description="CPI value as a float (e.g. 3.5)")
    key_claim: str = Field(description="The central argument or finding of the article in one sentence")
    market_sentiment: Literal["positive", "negative", "neutral"] = Field(
        description="Overall market sentiment expressed in the article"
    )

structured_client = instructor.from_openai(OpenAI(api_key=os.environ.get("YOUR_OPENAI_API_KEY")))

extraction = structured_client.chat.completions.create(
    model="gpt-4o",
    response_model=ArticleExtraction,
    messages=[
        {
            "role": "user",
            "content": f"Extract structured information from the following article:\\n\\n{SCRAPED_TEXT}"
        }
    ]
)

print(extraction.model_dump_json(indent=2))

print("\\n" + "=" * 60)
print("INSPECTION OUTPUT")
print("=" * 60)
for field, value in extraction.model_dump().items():
    print(f"  {field}: {repr(value)}  →  type: {type(value).__name__}")

The above code leverages two fundamental libraries:

• Pydantic: This is a Python data validation library. You define a schema as a Python class, declare the fields and their types, and Pydantic enforces that any data you put into that class matches what you declared.

• Instructor: This is the bridge between Pydantic and the LLM. The core problem it solves is that LLMs’ APIs return text, but Pydantic validates Python objects. So, something has to sit in the middle, take the LLM’s response, parse it into the structure your Pydantic model expects, and retry the call if the output doesn’t validate. That’s what Instructor does. Without Instructor, you would have to manually prompt the model to return JSON, parse that JSON yourself, handle malformed responses, write retry logic, and coerce types by hand.

By using these two libraries, the ArticleExtraction() class does the following:

• Type enforcement: Defines cpi_march_value as a float. This guarantees the model will return an actual number) instead of a string (3.5 instead of "3.5 percent" as the previous example).

• Controls formatting and vocabulary: The Literal type on market_sentiment restricts the LLM’s output to "positive", "negative", or "neutral". The model cannot invent new categories. Similarly, the description for publication_date explicitly demands the ISO 8601 format.

• Built-in prompting: The Field(description="...") parameters serve a dual purpose. First, they document the code for developers. Secondly, under the hood, the Instructor library feeds these exact descriptions to the LLM as targeted instructions. This ensures the model understands exactly what “key claim” or “publication date” means in this context.

• Graceful omissions: Wrapping fields like author in Optional[...] gives the model permission to safely return a null value if the information isn’t present in the scraped text. This highly reduces the risk of hallucinations.

The JSON output is as follows:

{
  "title": "Fed Signals Caution as Inflation Data Disappoints",
  "author": "Sarah M. Connelly",
  "publication_date": "2026-04-14",
  "mentioned_organizations": [
    "Federal Reserve",
    "Labor Department",
    "Reuters",
    "Goldman Sachs",
    "JPMorgan"
  ],
  "cpi_march_value": 3.5,
  "key_claim": "The Federal Reserve remains cautious about cutting interest rates because inflation has not yet shown sufficient progress toward its two-percent target.",
  "market_sentiment": "negative"
}

As you can see, now the CPI is a float, and the publication date is in ISO 8601.

The inspection output is the following:

============================================================
INSPECTION OUTPUT
============================================================
  title: 'Fed Signals Caution as Inflation Data Disappoints'  →  type: str
  author: 'Sarah M. Connelly'  →  type: str
  publication_date: '2026-04-14'  →  type: str
  mentioned_organizations: ['Federal Reserve', 'Labor Department', 'Reuters', 'Goldman Sachs', 'JPMorgan']  →  type: list
  cpi_march_value: 3.5  →  type: float
  key_claim: 'The Federal Reserve remains cautious about cutting interest rates because inflation has not yet shown sufficient progress toward its two-percent target.'  →  type: str
  market_sentiment: 'negative'  →  type: str

This validation helps immediately see that the data types are correct.

Conclusion

In this article, you learned what unstructured text actually costs a data pipeline. You saw how classical NLP made structured extraction possible but fragile, and how LLMs removed the domain constraints that NLP never solved. You also learned why LLMs alone are not enough and saw a practical solution to provide “guardrails” for LLMs so that their output follows a defined schema.

So, let us know: how are you managing unstructured text after you scraped it?

In this article, you’ll find out this and more. I’ll walk you through a complete guided example of how to use it, and break down its (Spoiler: undoubtedly serious) limitations.

An Introduction to the Cloudflare Crawl Endpoint

Before exploring the technical aspects behind the Cloudflare /crawl endpoint and seeing it in action, let me first give you some context!

What Is the Cloudflare /crawl Endpoint?

The /crawl endpoint is a new addition to Cloudflare’s REST APIs. Its goal is to crawl an entire website (or just a portion of it) starting from a single URL.

Note: The Crawl endpoint is currently in beta and was introduced on March 10, 2026, as highlighted in the Cloudflare changelog.

Under the hood, it automatically discovers and visits new pages, rendering them in a headless browser. It returns the discovered content as HTML, Markdown, or structured JSON, making it ideal for RAG pipelines, monitoring, or dataset creation.

As I’ll dive into later, it respects robots.txt and doesn’t bypass bot protection or captchas. Thus, it’s designed as a compliant approach to web crawling!

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

How It Works at a High Level

At a high level, the /crawl endpoint involves a two-step flow:

1. You kick off an asynchronous crawl job, passing a starting URL. Cloudflare returns a job ID.

2. You use that job ID to periodically check the job’s status or fetch results as they become available, following typical polling behavior.

Important: A crawl job can run for up to seven days! Results remain available for 14 days after completion, after which the job data is deleted.

Behind the scenes, the crawler expands outward from the starting URL. By default, the API follows a clear order:

1. The initial page.

2. Sitemap URLs.

3. Links discovered within pages.

Still, you can tweak that depending on whether you want to prioritize sitemaps, page links, or both.

Supported Use Cases

The officially promoted use cases for the Cloudflare /crawl API are just two:

• Creating knowledge bases or training AI systems (like RAG applications) using up-to-date web content.

• Collecting and analyzing content across multiple pages for research, summarization, or monitoring purposes.

Pricing

Compared to most other web crawling or discovery APIs on the market, Cloudflare’s /crawl API doesn’t charge by the number of pages. Instead, costs are based on resource usage, which depends on whether you enable the headless browser rendering feature.

If headless rendering is active, pricing follows the Browser Rendering model:

If rendering isn’t active, pricing follows the Workers model:

Yeah, I know… It’s honestly a bit confusing, and it’s almost impossible to predict the exact cost of a crawling task. The good news? You can test it for free!

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

Cloudflare Crawl Endpoints: Technical Analysis

Now that you know what Cloudflare is and what it brings to the table, it’s time to better understand its functioning, strengths, and limitations.

Endpoint Presentation

The Cloudflare Crawl API is built around two main endpoints. Both share the same base URL:

https://api.cloudflare.com/client/v4/accounts/<ACCOUNT_ID>/browser-rendering/crawl

Where <ACCOUNT_ID> is your Cloudflare account ID.

1. Initiate the Crawl Job (POST)

To start a new crawl, you need to send a POST request with the target URL (and optional parameters like depth, rendering mode, etc.) as below:

curl -X POST 'https://api.cloudflare.com/client/v4/accounts/<ACCOUNT_ID>/browser-rendering/crawl' \
  -H 'Authorization: Bearer <CLOUDFLARE_API_TOKEN>' \
  -H 'Content-Type: application/json' \
  -d '{ "url": "https://example.com" }'

Keep in mind that the endpoint supports several parameters, allowing you to greatly customize the crawling behavior, output format (JSON, HTML, or Markdown), rendering options, caching, and more. Check out the full list of supported body parameters for all available options.

Cloudflare immediately returns a job ID that you’ll use to track or retrieve results. A possible response looks like this:

{
  "success": true,
  "result": "9f1c2d3a-4b5e-6f7a-8c9d-0e1f2a3b4c5d"
}

The UUID in the result field is the Crawl job ID you’ll use to poll for updates.

2. Request Crawl Results (GET)

Once the crawl is running, make a GET request with the job ID to check the status or fetch results:

curl -X GET 'https://api.cloudflare.com/client/v4/accounts/<ACCOUNT_ID>/browser-rendering/crawl/<JOB_ID>' \
  -H 'Authorization: Bearer <CLOUDFLARE_API_TOKEN>'

Here, the <JOB_ID> placeholder is the UUID retrieved before from the result field.

The response either includes a status field like this:

{
  "result": {
    "id": "3e7a1c92-b4d8-4f6e-9a21-6c0d5b8e2f14",
    "status": "running"
    // ...
  }
}

The possible status values are: running, completed, errored, or one of several cancellation states (cancelled_due_to_timeout, cancelled_due_to_limits, cancelled_by_user).

Or, once the job is completed, calling the API returns the full results in the records field:

{
  "result": {
    "id": "3e7a1c92-b4d8-4f6e-9a21-6c0d5b8e2f14",
    "status": "completed",
    "browserSecondsUsed": 98.3,
    "total": 12,
    "finished": 12,
    "records": [
      {
        "url": "https://example.com/",
        "status": "completed",
        "markdown": "# Example Domain\nThis domain is for use in illustrative examples...",
        "metadata": {
          "status": 200,
          "title": "Example Domain",
          "url": "https://example.com/"
        }
      },
      {
        "url": "https://example.com/about",
        "status": "completed",
        "markdown": "## About\nLearn more about this example site...",
        "metadata": {
          "status": 200,
          "title": "About - Example Domain",
          "url": "https://example.com/about"
        }
      }
      // additional entries omitted for brevity...
    ],
    "cursor": 10
  },
  "success": true
}

Note that the response will vary based on the specified query parameters. For example, you can filter by specific statuses, limit the number of results, and navigate through them using a pagination system.

Features

Below is a list of the main, most relevant capabilities provided by the Cloudflare Crawl API:

• Asynchronous crawl jobs: Trigger crawling jobs and poll results when they are ready, enabling non-blocking, large-scale crawling workflows.

• Automatic URL discovery: Finds pages from the starting URL, sitemaps, and in-page links, with configurable source control.

• Flexible output formats: Returns HTML, Markdown, or structured JSON. JSON leverages Workers AI for schema-driven data extraction.

• Headless browser rendering: Control JavaScript execution with render: true or perform fast static HTML fetches with render: false.

• Fine-grained crawl control: Configure limit, depth, and URL inclusion/exclusion with the includePatterns/excludePatterns fields.

• Incremental and cache-aware crawling: Use modifiedSince and maxAge parameters to avoid re-fetching unchanged content, optimizing performance and cost.

• Advanced filtering and pagination: Retrieve results using limit, cursor, and status filters to handle large datasets efficiently.

• Authentication and custom headers: Supports HTTP auth, cookies, and custom headers for crawling protected or API-driven content.

• Dynamic content handling: Wait for JS-rendered content using gotoOptions and waitForSelector, ideal for SPAs and interactive pages.

• Resource skipping for performance: Optionally block images, media, fonts, or stylesheets to speed up crawling and reduce unnecessary bandwidth usage.

Your scraping workflows deserve a proxy infrastructure that just works. With Swiftproxy on your side, consistency is built-in.

Try Swiftproxy today

Limitations

Cloudflare’s /crawl API also comes with several important limitations, such as:

• Respects bot protection: The crawler can’t bypass CAPTCHAs (including Turnstile challenges) or Cloudflare bot protections. As a rule of thumb, sites protected via Cloudflare Bot Management or other WAFs tend to block crawling tasks entirely, limiting automated access and leading to incomplete datasets.

• Fixed User-Agent: The /crawl endpoint sets a non-customizable User-Agent value (CloudflareBrowserRenderingCrawler/1.0). You can’t change it, which may cause sites to block requests or serve different content based on the User-Agent.

• Content Signals enforcement: If a site disallows AI usage via Cloudflare Content Signals, crawl requests for those purposes are rejected with a 400 Bad Request error. Even if the site allows other uses, attempts to crawl disallowed categories will fail, limiting AI-specific data collection.

• Rate limiting and crawl pacing: Sites with strict rate limits can slow down crawling. The crawler respects the robots.txt Crawl-delay directive and implements backoff. Large crawls may need to be split into smaller jobs to avoid throttling or skipped URLs.

• Browser usage limits and job cancellation: Accounts on Workers free plans are capped at 10 minutes of browser time per day. Exceeding this limit results in a cancelled_due_to_limits status. To avoid that, you can upgrade to a paid plan.

How to Use the Cloudflare Crawl Endpoint: Step-by-Step Tutorial

In this guided section, I’ll show you how to use the Cloudflare Crawl Endpoint to crawl a website in Python. The target site will be the “Quotes to Scrape” sandbox. The goal here is to demonstrate how to use the API, rather than actually collecting relevant data.

Follow the instructions below!

Prerequisites

To follow this tutorial section, make sure you have:

• Your Cloudflare account ID at hand.

• A Cloudflare API token with the “Browser Rendering - Edit” permission.

For the sake of simplicity and to keep this tutorial concise, I’ll assume you already have a Python project set up with requests installed. That said, you can use any programming language and any HTTP client, because the high-level logic remains the same.

Step #1: Set Up the Configurations

Start by importing the required libraries and reading the necessary secrets (your Cloudflare API token and account ID). Use these secrets to prepare the Cloudflare Crawl base URL and headers. Also, specify the starting target URL as a constant.

import requests
import time
import json

# The Cloudflare secrets required for authenticating Crawl API calls
CLOUDFLARE_API_TOKEN = "<YOUR_CLOUDFLARE_API_TOKEN>"
CLOUDFLARE_ACCOUNT_ID = "<YOUR_CLOUDFLARE_ACCOUNT_ID>"

# The base URL used for all Crawl API calls
CLOUDFLARE_CRAWL_BASE_URL = f"https://api.cloudflare.com/client/v4/accounts/{CLOUDFLARE_ACCOUNT_ID}/browser-rendering/crawl"

# Common headers shared by all API calls
HEADERS = {
    "Authorization": f"Bearer {CLOUDFLARE_API_TOKEN}",
    "Content-Type": "application/json"
}

# The URL to crawl
START_URL = "https://www.ssense.com/en-us/men/product/acne-studios/silver-folded-leather-wallet/18169981"

Tip: In a production script, read the Cloudflare API token and account ID from environment variables rather than hardcoding them.

Step #2: Trigger the Crawling Job

Define a start_crawl() function to send a POST request to Cloudflare’s Crawl API:

def start_crawl(start_url):
    # Customize according to your needs
    payload = {
        "url": start_url,
        "limit": 20,
        "depth": 2,
        "formats": ["markdown"],
        "render": True
    }

    response = requests.post(CLOUDFLARE_CRAWL_BASE_URL, headers=HEADERS, json=payload)
    response.raise_for_status()

    job_id = response.json()["result"]
    return job_id

This creates a new crawling job for the target URL. Then, it returns a job_id that identifies this specific crawl.

Tip: In a production-level script, make the payload object configurable via function input arguments for greater flexibility and reusability.

Step #3: Poll Over the Job

Next, add a wait_for_completion() function to repeatedly check the job status every few seconds until the crawl finishes or times out:

def wait_for_completion(job_id, max_attempts=60, delay=5):
    status_url = f"{CLOUDFLARE_CRAWL_BASE_URL}/{job_id}?limit=1"

    for attempt in range(max_attempts):
        res = requests.get(status_url, headers=HEADERS)
        res.raise_for_status()

        status = res.json()["result"]["status"]
        print(f"Attempt {attempt+1}: {status}")

        if status != "running":
            return status

        time.sleep(delay)

    raise Exception("Timeout waiting for crawl job")

This makes GET calls to the Cloudflare /crawl endpoint. It ensures you’re waiting for the task to complete processing before fetching the crawled records.

Tip: The limit=1 query parameter is recommended to restrict the number of retrieved records, keeping the response lightweight. After all, at this stage, you’re only interested in checking the job status, not in retrieving the actual output data.

Check the TWSC YouTube Channel

Step #4: Get the Crawled Content Pages

Build a fetch_records() function to collect all crawled pages:

def fetch_records(job_id):
    print("Fetching results...")

    all_records = []
    cursor = None

    while True:
        url = f"{CLOUDFLARE_CRAWL_BASE_URL}/{job_id}"
        # Getting 10 records at a time
        params = {
            "limit": 10
        }

        if cursor:
            params["cursor"] = cursor

        response = requests.get(url, headers=HEADERS, params=params)
        response.raise_for_status()

        result = response.json()["result"]
        records = result.get("records", [])

        all_records.extend(records)
        print(f"+{len(records)} records (Total: {len(all_records)})")

        cursor = result.get("cursor")
        if not cursor:
            break

    return all_records

This handles pagination using a cursor, accessing records in batches (10 per request) until all results are returned.

Step #5: Put It All Together

Finally, in the main() function, orchestrate the workflow:

1. Start the crawl

2. Wait for completion

3. Fetch all results

Then, you can export the crawled records to a local JSON file for further use, store the retrieved data in a database, process it there, etc.

def main():
    job_id = start_crawl(START_URL)
    status = wait_for_completion(job_id)

    if status != "completed":
        raise Exception(f"Crawl failed: {status}")

    print("Crawl completed!")

    records = fetch_records(job_id)

    # Export the crawled pages to an output JSON file
    with open("records.json", "w", encoding="utf-8") as f:
        json.dump(records, f, indent=2, ensure_ascii=False)

if __name__ == "__main__":
    main()

Step #6: Complete Code

This is what your Python script for interacting with the Cloudflare Crawl API will look like:

# pip install requests

import requests
import time
import json

# The Cloudflare secrets required for authenticating Crawl API calls
CLOUDFLARE_API_TOKEN = "<YOUR_CLOUDFLARE_API_TOKEN>"
CLOUDFLARE_ACCOUNT_ID = "<YOUR_CLOUDFLARE_ACCOUNT_ID>"

# The base URL used for all Crawl API calls
CLOUDFLARE_CRAWL_BASE_URL = f"https://api.cloudflare.com/client/v4/accounts/{CLOUDFLARE_ACCOUNT_ID}/browser-rendering/crawl"

# Common headers shared by all API calls
HEADERS = {
    "Authorization": f"Bearer {CLOUDFLARE_API_TOKEN}",
    "Content-Type": "application/json"
}

# The URL to crawl
START_URL = "http://quotes.toscrape.com/"

def start_crawl(start_url):
    """
    Triggers the Cloudflare Crawl API job
    """

    # Customize according to your needs
    payload = {
        "url": start_url,
        "limit": 20,
        "depth": 2,
        "formats": ["markdown"],
        "render": True
    }

    response = requests.post(CLOUDFLARE_CRAWL_BASE_URL, headers=HEADERS, json=payload)
    response.raise_for_status()

    job_id = response.json()["result"]
    return job_id

def wait_for_completion(job_id, max_attempts=60, delay=5):
    """
    Waits for the crawling task to complete
    """

    status_url = f"{CLOUDFLARE_CRAWL_BASE_URL}/{job_id}?limit=1"

    for attempt in range(max_attempts):
        res = requests.get(status_url, headers=HEADERS)
        res.raise_for_status()

        status = res.json()["result"]["status"]
        print(f"Attempt {attempt+1}: {status}")

        if status != "running":
            return status

        time.sleep(delay)

    raise Exception("Timeout waiting for crawl job")

def fetch_records(job_id):
    """
    Collects all records from the paginated results
    """

    print("Fetching results...")

    all_records = []
    cursor = None

    while True:
        url = f"{CLOUDFLARE_CRAWL_BASE_URL}/{job_id}"
        # Getting 10 records at a time
        params = {
            "limit": 10
        }

        if cursor:
            params["cursor"] = cursor

        response = requests.get(url, headers=HEADERS, params=params)
        response.raise_for_status()

        result = response.json()["result"]
        records = result.get("records", [])

        all_records.extend(records)
        print(f"+{len(records)} records (Total: {len(all_records)})")

        cursor = result.get("cursor")
        if not cursor:
            break

    return all_records

def main():
    job_id = start_crawl(START_URL)
    status = wait_for_completion(job_id)

    if status != "completed":
        raise Exception(f"Crawl failed: {status}")

    print("Crawl completed!")

    records = fetch_records(job_id)

    # Export the crawled pages to an output JSON file
    with open("records.json", "w", encoding="utf-8") as f:
        json.dump(records, f, indent=2, ensure_ascii=False)

if __name__ == "__main__":
    main()

Step #7: Test the Script

Launch the script, and it’ll produce an output like this:

The polling mechanism required 5 attempts (~25 seconds), and the API discovered and retrieved 22 pages.

A records.json file will appear in your project directory. Open it, and you’ll see:

Notice how the “Quotes to Scrape” entries contain a markdown field with the Markdown version of the page. Instead, external links like Zyte’s homepage and Goodreads.com are skipped, since includeExternalLinks is set to false by default. In other words, the Cloudflare Crawl API doesn’t automatically attempt to fetch data from different domains than the target source URL.

Et voilà! Implementation complete.

Need help with your scraping project?

Benchmark Against Protected Websites

Cool! The Cloudflare Crawl endpoint works like a charm and is easy to use. However, I was particularly concerned about its documented limitations and wanted to verify whether they actually hold up in practice…

So, I ran tests against several well-known sites protected by common WAF and anti-bot solutions (from different providers). Here’s a summary of the results:

As you can tell, the limitations are very real. The results are quite discouraging: the Cloudflare Crawl API failed against all anti-bot–protected websites I tested.

So, is this solution reliable for web scraping? When (and how) should you actually use it? Let me break that down in a final comment!

Final Comment

In this article, I introduced you to one of the newest tools in Cloudflare’s growing ecosystem: the Crawl API! This endpoint is designed to help you crawl entire websites using distributed crawling tasks running on Cloudflare’s infrastructure.

Sure, the crawling mechanism works and is easy to launch, control, and implement. With just a few lines of code, you can get started. Still, several concerns should be raised:

1. Opaque pricing: Costs are tied to resource usage rather than the number of pages crawled, making them harder to predict.

2. Fixed User-Agent: The API doesn’t allow User-Agent customization, meaning even basic server-side checks can block it.

3. Limited effectiveness on protected sites: The API has an intended very low success rate against anti-bot–protected websites (unless you specify in Cloudflare Bot Protection settings that you allow it against your site).

4. Rate limiting constraints: It strictly respects robots.txt directives and crawl delays, which can significantly slow or limit large crawls.

In simple terms, if you want to use it for general-purpose, large-scale web crawling, I wouldn’t recommend it. The market offers more effective solutions that can actually bypass anti-bot limitations. Plus, remember that around 35% of the entire Internet is estimated to be protected against bots (i.e., you won’t be able to crawl it with this API).

Yet, if you know the target site is not protected, budget isn’t a concern, and you want to remain (overly?) ethical and compliant, the Cloudflare Crawl API can be an option.

I hope this breakdown helps you better understand this new solution and make an informed decision on whether to adopt it. Lastly, remember that the Cloudflare Crawl API is still in beta, so things may change soon. Just keep an eye on the docs for updates. Until next time!

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

The short answer is that a couple of tools finished the task, just one with any consistency. The story behind why is worth telling, because it explains what is happening at the intersection of AI agents and web data right now. We ran a similar exercise against Cloudflare earlier this year, and the conclusion is broadly the same: each anti-bot needs its own answer, and the answer changes every quarter.

From workflows to agents, and why that changes the data problem

Most code shipped under the AI banner is not really agentic. It is workflow code with an LLM dropped into a slot: generate a summary here, classify a record there, draft an email at the end. The control flow is hard-coded, and the model is one component among many.

The definition of an agent is quite different. The model decides the next action, observes the outcome, and decides again. The control flow lives inside the loop, not outside it. The agent has goals rather than scripts, and it picks tools and steps based on what it sees. That is what makes the engineering interesting, that is what makes it hard, and that is what sometimes makes it unreliable.

It also forces a different relationship with data. An agent that only sees its training corpus is stuck in the past. To make decisions worth anything, it has to read prices that change daily, stocks that move minute by minute, listings that did not exist last week. Some of that data sits behind APIs. Most of it does not. The web is still the largest and most current dataset in the world, and most of it is reachable only through a browser. So if we want our agents to act on real information, we have to give them a way to browse: opening a page, reading it, clicking a link, typing into a search bar, following a result, filling a form, all on sites that were never built for machines.

Start your scraping journey with Byteful: 10GB New Customer Trial | Use TWSC for 15% OFF | $1.75/GB Residential Data | ISP Proxies in 15+ Countries

Claim your 10GB here

This is the constraint that produced the wave of “agentic browser” launches we have seen over the last twelve months. Y Combinator alone has backed a long string of them. Hyperbrowser (S21) was an early entry: scalable cloud browser infrastructure with built-in CAPTCHA solving, proxy management, and now a multi-agent playground. The newer cohort followed the agent wave more directly: BrowserOS (S24) is an open-source agentic browser that runs the agent locally on the user’s machine; Browser Use (W25) offers an open-source agent loop on top of Playwright, plus a cloud version. Skyvern is a self-hostable browser agent that uses an LLM and computer vision instead of fixed selectors. Outside the YC pipeline, Lightpanda is doing something different again, a headless browser engine written from scratch in Zig and aimed squarely at agents and crawlers (claiming roughly 9x faster execution and 16x lower memory than Chrome). It fits the “browser built for machines” line of thought we covered in Rethinking the web browser earlier this year. Browserbase ships a managed browser plus Stagehand for natural-language automation. And the big AI labs are now in the same space: OpenAI shipped Operator and the ChatGPT Atlas browser, Anthropic shipped Computer Use, Perplexity launched Comet. Each project attacks the same problem from a slightly different angle, but the goal is identical: a browser an agent can drive without immediately tripping every detection mechanism on the other side.

Your scraping workflows deserve a proxy infrastructure that just works. With Swiftproxy on your side, consistency is built-in.

Try Swiftproxy today

The same problem scrapers have been chasing for a decade

For anyone who has worked in web data, none of this is new. The fight over whether a request looks human or automated has been going on as long as commercial scraping has existed. The product names have changed but the purpose not.

What has changed is who is selling the bypass. The companies that have spent years selling residential proxies and unblockers noticed quickly that the agentic boom is good for their business. They already have the IP networks, the fingerprint research, the bypass code, the cat-and-mouse experience. They know what TLS handshake Chrome sends in October 2025 and what it sent in October 2024. Pivoting all of that into a managed browser is a smaller leap than building one from scratch. Bright Data, Oxylabs, Rayobyte, ZenRows have all added a managed browser product alongside the proxy.

The other side of the line is moving in the opposite direction. Bot traffic has grown faster than human traffic for years, and the operators of large public sites care more about it than ever. DataDome, Cloudflare Bot Management, Akamai Bot Manager, HUMAN, Kasada: every one of them ships updates that target the exact tools we just listed. Fingerprint checks get stricter. Behavioral models get more sensitive. The JavaScript challenge changes shape every few weeks. There is no silver bullet, and there is no tool, browser, proxy, or service that bypasses every anti-bot on every site at all times. Anyone who claims otherwise is selling something that worked last quarter and might still work this week. The useful question is what works on a given target, today, at what cost.

Check the TWSC YouTube Channel

Picking a hard target

To answer that question concretely, we needed a target where the anti-bot was good and the signal was clean. We picked leroymerlin.fr, the French DIY retailer. Leroy Merlin runs DataDome standalone, with no other anti-bot layer on top, so attribution is straightforward. It also runs one of the more verbose DataDome configurations we have come across: response headers expose x-datadome-riskscore, x-datadome-protection, x-datadome-cid, and x-datadome-endpointid. Most DataDome-protected sites only show us the outcome. Here we see the score the engine assigns at every request, which is rare and very useful when comparing tools side by side.

The task we picked is small but realistic. From the homepage, the agent has to type “ampoule B22 led blanc” into the search bar, click the first product result, and add the product to the cart. Four steps. We dropped the login step on purpose: leroymerlin.fr requires an OTP to sign in, and we did not want OTP friction to confound an anti-bot test.

A run is a pass if the agent reaches the cart confirmation. Otherwise we record where it stopped and what DataDome said about it. Each tool runs ten times back to back, and we aggregate the results. Tools that support an external proxy use the same residential pool: Bright Data residential FR for the Bright Data runs, Geonode residential FR for the Geonode runs. Tools that ship their own proxy use it. The reason behind two different providers was because we wanted to diversify the IP addresses, to be sure that blocks were not a matter of IP reputation.

Need help with your scraping project?

The contestants

As you’ve seen before, the browser landscape is quite crowded and we could not cover all the tools. We picked four open-source projects and seven commercial products. Let’s start with the open source.

Camoufox is the stealth Firefox fork most people in the scraping world have already met (we introduced it on TWSC back in September 2024). It rotates real-world fingerprints, patches the obvious automation tells, and ships a Playwright-compatible API. We pair it with both Bright Data and Geonode residential proxies in France.

Pydoll takes a different route: it drives Chromium directly over CDP without WebDriver, with built-in humanized cursor movement and typing. Importantly, Pydoll implements an explicit Fetch.authRequired handler, which lets it authenticate proxies that require Basic auth.

Scrapling is a higher-level Python library. We use it in two modes. DynamicFetcher launches vanilla Playwright Chromium driven by Scrapling’s session manager. StealthyFetcher does the same, but under the hood uses an improved and customized version of patchright, a stealth-patched Playwright fork. Each gets its own row in the comparison.

RayoBrowse is the self-hosted stealth Chromium fork from Rayobyte, distributed as a Docker container that exposes a CDP endpoint on port 9222. Here we hit a wall worth flagging: for some reason RayoBrowse could not use the Bright Data residential proxy in our setup. Every navigation through that proxy failed instantly, even though the same credentials worked fine through curl from inside the same container. The same RayoBrowse setup worked fine with Geonode. We did not isolate the root cause, so we report RayoBrowse on Geonode only.

The commercial side is more crowded.

Browser Use exists in two flavors, and we tested both. The cloud version is the managed Browser Use, with its own residential proxy, its own stealth fingerprinting, and a fixed set of supported models; we drove it once in raw CDP mode (we steer it ourselves with Playwright) and once in agent mode (we hand the LLM the task in natural language and let it plan the steps).

Browserbase is a managed Chromium with optional residential proxies, Cloudflare Web Bot Auth verification, and the Stagehand agent SDK. We discovered during the test that the free tier excludes proxies entirely; without one, the session egresses from a US datacenter. We left this configuration in the test because it is what a free user would experience.

Browserless is a managed browser-as-a-service whose anti-bot story is a stealth path (/chromium/stealth) plus optional residential proxies for paid plans. The free plan caps sessions at 60 seconds, which is tight for a four-step flow. We tested it with the built-in residential proxy targeting France, and tried to test it with our external proxies via the externalProxyServer parameter; the external mode failed at connection time on every run, in the same Chromium-side authentication way that broke RayoBrowse, so we drop those configurations from the comparison.

ZenRows Scraping Browser is a managed Chromium with a built-in residential proxy network and built-in CAPTCHA solving; we connect via the WSS endpoint with proxy_country=fr to get a French exit point.

Bright Data Browser API sits at the other end of the same product category: a managed Chromium with built-in residential rotation and CAPTCHA solving, on a dedicated Browser API zone we configured on their dashboard.

As always, the code can be found in our GitHub repository reserved to paying users, inside the folder 103.BROWSERS.

What we had to fix before the numbers made sense

Read more

This means that if you route all your traffic through a single egress IP, you will be rate-limited in seconds and blacklisted in minutes. To survive at scale, you need to distribute your requests across a massive pool of IP addresses.

Traditionally, the web scraping industry has solved this issue thanks to commercial proxy providers. However, this is not the only approach. This article responds to the following question: “Is there a way to scrape at scale without burning budget on proxies?”

The answer is yes. But let’s be clear from the beginning: This approach is not a universal silver bullet. Let’s see how it works, how to build it, and what its limitations are.

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

The Typical Solution for Scraping at Scale: Proxy Provider Services

Let’s start this discussion with the typical choice for scraping at scale. IP bans and rate limits are the #1 operational problem in scraping, especially at scale. The typical solution every web scraping engineer integrates is using proxy servers, for a simple reason: proxies act as intermediaries between your scrapers and the Internet, avoiding your scrapers from getting banned. To do so, companies buy proxy IPs from proxy providers. The most common categories, both with their flaws, are the following:

• Datacenter proxies: These are cheap and fast, but their ASNs(Autonomous System Numbers) are heavily scrutinized. WAFs maintain databases of known datacenter CIDR (Classless Inter-Domain Routing) blocks, so hitting a target with a static list of 100 datacenter proxies usually results in those IPs being flagged and blocked within hours.

• Residential proxies: These route traffic through actual consumer devices. They have highly trusted IP reputations, making them excellent for bypassing anti-bot systems. However, they are priced by bandwidth, so they are very expensive, especially when scraping at scale.

The main limitation of this approach is that it is highly expensive. So, what if you need to scrape at scale but don’t have enough budget for doing so?

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

An Alternative Approach: Scraping at Scale With Dedicated Infrastructure

To escape metered billing, you can move egress back to dedicated infrastructure. But before presenting the solution, let’s first point out shortly what happens when you buy and use proxies, at the infrastructure level.

Buying Proxies Means Delegating Your Infrastructure

When you buy proxies from providers, you are delegating 100% of your infrastructure. When your scrapers make the requests, under the hood, the proxy provider connects to a gateway, which is a massive load balancer controlled entirely by the provider itself.

Let’s consider the case of residential proxies, for simplicity. Behind the gateway is a peer-to-peer (P2P) network of millions of consumer devices that the provider has acquired bandwidth from. When your request hits the gateway, their proprietary routing algorithm decides which consumer device in which country will act as your final exit node.

The second you route traffic through their gateway is the exact moment where you delegate the 100% of your scraping infrastructure.

Your scraping workflows deserve a proxy infrastructure that just works. With Swiftproxy on your side, consistency is built-in.

Try Swiftproxy today

NyxProxy: The Infrastructural Solution

NyxProxy is a self-hosted HTTP/SOCKS5 proxy server that exploits a well-known IPv6 networking trick: When a cloud provider gives you a /64 subnet, you legally own 18.4 quintillion IPv6 addresses.

Let’s explain the number and the trick around IPv6s. An IPv6 address looks like this:

 2a05:f480:1800:25db:0000:0000:0000:0001

They are 128 bits long. That gives 2^128 possible addresses. The number is so large that the designers said: “We can afford to give every organization a massive block and never worry about running out”.

Now, here is the trick. An IPv6 address is split into two halves, 64 bits each:

2a05:f480:1800:25db : 0000:0000:0000:0001
|___________________|   |_________________|
   Network prefix            Host part
   (your subnet)          (you control this)

The /64 notation means: the first 64 bits identify the network, the last 64 bits are yours to assign however you want. The last 64 bits can be any value from 0000:0000:0000:0000 to ffff:ffff:ffff:ffff: That’s 2^64 = 18.4 quintillion combinations. All valid addresses, all routable to your server.

Thanks to this trick, NyxProxy can assign a pool of those addresses to your network interface at startup, then rotate your outgoing traffic across them. This means having a fresh IP per request. The tool handles pool management, background rotation, NDP proxying via ndppd, and exposes a monitoring endpoint.

The best part is, indeed, in the NDP proxying. When your server uses a random address like 2a05:f480:1800:25db:a3f1:9922:beef:1234 as a source IP, your router upstream needs to know your server is responsible for that address. Otherwise, the response packets have nowhere to go.

IPv6 uses NDP (Neighbor Discovery Protocol) for this. The router sends an NDP query: “who has 2a05:f480:1800:25db:a3f1:9922:beef:1234?” and your server must answer.

ndppd (NDP Proxy Daemon) runs on your server and answers those queries automatically for your entire /64 subnet, essentially saying “yes, all of those addresses are mine”. Without it, your packets go out, but responses never come back.

Below is a summary schema of how this whole process works:

Provider gives you:  2a05:f480:1800:25db::/64
                     ↓
Your server can use: 2a05:f480:1800:25db:[anything]
                     ↓
NyxProxy assigns 200 random IPs to your interface
                     ↓
Each outgoing request binds to a different one
                     ↓
Target sees 200 different source IPs
                     ↓
ndppd makes sure responses route back correctly

Check the TWSC YouTube Channel

How To Use NyxProxy

Let’s now see how to use NyxProxy with a practical implementation.

Environment Setup & Prerequisites

To replicate this tutorial for deploying NyxProxy and utilizing it in your scraping scripts, you must have the following system and hardware requirements:

• Hardware: A Virtual Private Server (VPS) or bare-metal server with at least 512 MB of RAM and 100 MB of disk space. Supported architectures are amd64 or arm64.

• Subnet: A cloud provider that natively delegates a full IPv6 /64 subnet to your network interface. Note that not all the VPS providers are supported: Check out the NyxProxy documentation to learn more about supported VPSs.

• Operating system: A modern Linux distribution, specifically Ubuntu or Debian, to ensure compatibility with the automated setup scripts and sysctl kernel modifications.

• Python: Python 3.7 or higher installed on your local machine to run the scraping scripts.

To get your server ready to run the proxy daemon, you need to verify your IPv6 setup and gain root access. Ensure you are logged into your VPS via SSH as the root user, or have sudo privileges.

First, verify that your server has a globally routable IPv6 /64 subnet assigned to it. You can check this by running the following command in your server’s terminal:

ip -6 addr show | grep "scope global"

If done correctly, you should see an output similar to the following:

inet6 2a05:f480:1800:25db::1/64 scope global

If you do not see a /64 subnet, you will not be able to rotate IPs, and you must review your cloud provider’s network settings.

Next, prepare your local development environment. Suppose you call the main folder of your Python project nyxproxy_scraper/. At the end of this step, the folder will have the following structure:

nyxproxy_scraper/
    ├── main.py
    └── venv/

Where:

• main.py is the Python file that will store your proxy request logic.

• venv/ contains the standard Python virtual environment.

You can create the venv/ virtual environment directory like so:

python -m venv venv

To activate it, on Windows, run:

venv\Scripts\activate

Equivalently, on macOS and Linux, execute:

source venv/bin/activate

As a final prerequisite, install the Requests library in your activated virtual environment so your Python script can make HTTP calls:

pip install requests

Well done! You are now ready to test and use Nyxproxy.

Need help with your scraping project?

Installing and Configuring NyxProxy

NyxProxy provides a quick setup script that handles the infrastructural heavy lifting. It auto-detects your network interface, installs ndppd, tweaks the Linux kernel parameters via sysctl to allow non-local binding, and downloads the compiled Go binary.

You can launch it with the following single command:

wget <https://raw.githubusercontent.com/jannik-schroeder/nyxproxy-oss/main/scripts/quick-setup.sh> && chmod +x quick-setup.sh && sudo ./quick-setup.sh

During the setup, you will be prompted to configure your proxy credentials and set your rotation rules. Behind the scenes, the script generates a config.yaml file. Let’s look at the crucial subset of that configuration:

network:
  rotate_ipv6: true
  ipv6_subnet: "2a05:f480:1800:25db::/64"

  # The rotation mechanics:
  ipv6_pool_size: 200
  ipv6_max_usage: 100
  ipv6_max_age: 30

Below is an explanation of what these three parameters mean for your scraping pipeline:

• ipv6_pool_size: NyxProxy keeps 200 mathematically unique IPs “hot” and bound to your network interface at any given time. This keeps proxy startup times under 100ms while maintaining IP diversity.

• ipv6_max_usage: After a specific IP has been utilized for 100 requests, it is considered “burned.” NyxProxy destroys the route and spins up a fresh address to dynamically replace it.

• ipv6_max_age: If an IP hasn’t hit 100 requests but has been alive for 30 minutes, it gets forcefully rotated out. This prevents time-based algorithmic tracking by the target WAF.

Once the daemon is running as a systemd service, your VPS is officially acting as a rotating proxy gateway. When NyxProxy receives a scraper request, the underlying Go binary takes over. It looks at its internal memory, picks one of the 200 rotating IPv6 addresses in its pool, and binds to that specific address to establish the outbound connection.

The expected output is as follows:

IPv6 rotation mode: IP Pool with dynamic rotation
  Interface: enp1s0
  Subnet: 2a05:f480:1800:25db::/64
  Pool size: 200 IPs
  Rotation: Every 100 uses or 30m0s
  Initializing IP pool...
  Progress: 50/200 IPs added
  Progress: 100/200 IPs added
  Progress: 150/200 IPs added
  Progress: 200/200 IPs added
  IP pool ready with 200 addresses
  Background IP rotation started

Starting https proxy on 0.0.0.0:8080 (Protocol: IPv6)

Testing the Proxy Logic

At this point, NyxProxy has done its job. To verify it works correctly, you can use the following Python script that hits api6.ipify.org, which is an API that simply bounces back the IP address it sees:

import requests

# Point this to your VPS IP and the credentials you set during setup
proxies = {
    'http': '<http://admin:password@your-vps-ip:8080>',
    'https': '<http://admin:password@your-vps-ip:8080>'
}

# Test 5 consecutive scraping requests
for i in range(5):
    response = requests.get('<https://api6.ipify.org>', proxies=proxies)
    print(f"Request {i+1}: Target sees IP -> {response.text}")

(NOTE: If you are already familiar with ipify.org, note that the “api6” prefix can be used for IPv6 requests only.)

The result should be similar to the following:

Request 1: Target sees IP -> 2a05:f480:1800:25db:1a2b:3c4d:5e6f:7890
Request 2: Target sees IP -> 2a05:f480:1800:25db:9988:7766:5544:3322
Request 3: Target sees IP -> 2a05:f480:1800:25db:aaaa:bbbb:cccc:dddd
Request 4: Target sees IP -> 2a05:f480:1800:25db:1122:3344:5566:7788
Request 5: Target sees IP -> 2a05:f480:1800:25db:dead:beef:cafe:babe

This shows that every single HTTP request utilizes a completely different, globally routable IPv6 address generated from your subnet block. To the target server, these look like entirely distinct users connecting from across the internet.

Perfect! You have successfully built a self-healing, infinitely rotating proxy pool without handing over your budget for metered residential bandwidth.

The Illusion of Infinity: Critical Limitations of IPv6 Subnet Rotation

At this point, you may think you have found a solution to all of your budgeting problems for scraping at scale. But before you tear down your commercial proxy infrastructure, you must understand that a $5/Mo VPS and an open-source rotation daemon are not a universal silver bullet. If it were that simple, the commercial proxy industry would not exist.

This architecture has the following main limitation:

• The IPv4 compatibility wall: This entire architecture is built on one absolute prerequisite: Your target endpoint must support IPv6. If you are scraping legacy enterprise systems or platforms that haven’t migrated to dual-stack networking, this setup is useless. You cannot route an IPv6 packet to an IPv4-only server.

• Subnet-level bans (/64 prefix blocking): Enterprise WAFs are fully aware of IPv6 prefix delegation standards. They know that hosting providers allocate a /64 subnet to a single client. If their heuristics detect highly concurrent behavioral patterns (like missing browser fingerprints or anomalous TLS handshakes) originating from 2a05:f480...:1a2b, they will ban the entire /64 CIDR block. Once your /64 prefix is banned, all 18 quintillion of your “infinite” IPs are simultaneously dead. To recover, you must physically destroy the VPS and provision a new one in a different IP range.

• ASN reputation: No matter how many IPs you rotate, your traffic still originates from a Datacenter Autonomous System Number (ASN). Target firewalls assign a baseline trust score to every ASN. Traffic originating from a Datacenter ASN always starts with a highly degraded trust score compared to a Residential ASN. For highly restrictive targets, any request from a datacenter IP is instantly met with an unpassable CAPTCHA or a hard 403 Forbidden, regardless of whether it’s IPv4 or IPv6.

• nf_conntrack and hardware exhaustion: You cannot push enterprise-grade throughput on a $5, 1-vCPU server without consequence. Rotating thousands of IPv6 addresses requires the Linux kernel to aggressively maintain the nf_conntrack table and the NDP proxy table. At high concurrencies, the overhead of establishing, tracking, and tearing down thousands of TCP sockets across rotating interfaces will exhaust the memory or CPU of a low-tier VPS. The kernel will begin dropping packets natively, your latency will spike to useless levels, and your scrapers will be greeted with errors.

Conclusion

In this article, you learned how to leverage your hosting provider’s IPv6 /64 subnets to build an infinitely rotating proxy pool with NyxProxy, escaping the metered billing of residential proxy networks.

The competitive advantage of engineering your own proxy infrastructure is in your unit economics and architectural control. However, you also learned that this solution is not a universal silver bullet for every scraping scenario: It comes with trade-offs and constraints.

So, let us know: Have you already experimented with bare-metal IPv6 rotation for your scraping pipelines? What targets did it work best for? Let’s discuss in the comments!

Behind the scenes, it relies on hydration to make server-rendered pages interactive. And here’s the interesting part: the same mechanism that makes Next.js fast and popular also exposes a significant amount of structured data in the HTML sent by the server. From a scraping perspective, that’s a huge opportunity!

In this post, I’ll show you a simple trick to scrape data from virtually any Next.js website. Follow along as I break down how it works and how you can apply it yourself.

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

Next.js in Numbers

Next.js needs no introduction, but it’s worth giving some context to truly understand how popular it is (and therefore how useful the trick I’m about to present for Next.js web scraping can be):

• According to the 2025 Stack Overflow Developer Survey, 20.8% of respondents used Next.js extensively over the past year.

• Next.js is the 14th largest repository on GitHub, with over 138k stars (and still growing!).

• According to W3Techs, Next.js has a 2.9% market share among JavaScript libraries.

• Major brands such as Nike, Stripe, and Notion have chosen this full-stack framework to build their official websites.

Before Getting Started: A Bit of Context on Hydration

I know you probably just want the trick… Still, let me take a minute to explain why it works in the first place, why it’s even possible, and what kind of data you’ll actually retrieve with it!

Start your scraping journey with Byteful: 10GB New Customer Trial | Use TWSC for 15% OFF | $1.75/GB Residential Data | ISP Proxies in 15+ Countries

Claim your 10GB here

What Is Hydration?

Hydration is the process that makes a server-rendered page interactive in the browser.

Frameworks like Next.js, Remix, Nuxt, and SvelteKit employ this mechanism to combine the performance benefits of server-side rendering (SSR) with the interactivity of client-side applications.

The idea is that the server first sends fully rendered static HTML to the browser. Then, hydration happens next.

The browser downloads the JavaScript bundle, and the frontend framework reconstructs the component tree in memory, attaches event listeners, and links that virtual tree to the existing DOM instead of re-rendering it from scratch. The result is a fully interactive application built on top of server-rendered HTML.

How Does the Hydration Mechanism Work?

It’s now clear that in Next.js and similar frameworks, hydration is the process where a static, server-rendered HTML page “comes to life” and becomes fully interactive in the browser. But what’s actually happening under the hood?

At a high level, hydration is a 3-step process:

1. The server generates and sends a fully rendered HTML snapshot. The user immediately sees the content (great for First Contentful Paint). At this point, though, the page is just static HTML. Buttons, forms, and other interactive elements are visible, but they don’t work yet because no JavaScript is attached.

2. The client’s browser downloads the JavaScript bundle (which includes React and your frontend application code) and executes it.

3. React rebuilds the component tree in memory and attaches event listeners to the existing DOM nodes. Instead of discarding the HTML and re-rendering everything from scratch, React “hydrates” the existing markup, meaning it reuses it and wires it up with state and interactivity.

Once hydration completes, the page behaves like a normal single-page application: it responds to clicks, manages state, and updates dynamically.

And here’s an important detail: if the browser doesn’t support JavaScript (or it fails to load), the user still sees the server-rendered HTML. It won’t be interactive, but the core content is there. That’s great for SEO and perceived performance!

Why It Matters for Scraping Next.js (and Other Full-Stack Frameworks…)

The key insight you need to understand is simple: hydration requires data, and that data must be embedded somewhere in the HTML sent by the server!

In Next.js, when the server renders a page, it doesn’t only send markup. It also serializes the data required to rebuild the React component tree on the client. That serialized payload is embedded directly into the page’s HTML.

That’s exactly why hydration matters for scraping. Instead of parsing the DOM or simulating user interactions through browser automation, you can extract the structured data that React itself uses to hydrate the page.

In many cases, hydration data is cleaner and easier to parse than the rendered HTML. It can also contain more information than what’s visibly displayed on the page, including hidden and interesting metadata.

Keep in mind that this principle applies not only to Next.js! All other full-stack frameworks that rely on hydration, such as Remix, Nuxt, Angular Universal, and SvelteKit, tend to dehydrate state on the server and rehydrate it on the client.

So remember this simple rule. If a framework hydrates, it must serialize data. And if it serializes data into the HTML, you can scrape it.

How to Scrape Next.js Websites: 2 Approaches

The approach to scraping Next.js by targeting hydration data depends on how that data is embedded in the HTML generated on the server side.

I won’t go too deep into framework internals here (if you’re a Next.js dev, you already know things shift depending on whether you’re using the App Router or the Pages Router), but there are essentially two scenarios you’ll run into.

In this section, I’ll walk through both of them and show you exactly how I retrieve data from each!

Check the TWSC YouTube Channel

Approach #1: Target the __NEXT_DATA__ Script

As a target site, I’ll use a Nike product page as a reference:

That’s actually a great example because Nike.com is even showcased on the Next.js homepage as a real-world site built with the framework.

Now, right-click on the page and select the “Inspect” option in your browser to open the DevTools. Scroll through the DOM and get familiar with the page structure. If the Next.js site is using the Pages Router, you’ll notice a <script> tag with the id __NEXT_DATA__ containing a large JSON blob:

That JSON data is precisely the hydration data I was referring to earlier.

When a site uses the Pages Router approach in Next.js, the server embeds all the page data directly into that <script> tag. From a scraping perspective, that’s gold, as the data is already structured and ready to be captured.

Below’s a simple JavaScript snippet to extract it:

const hydartionScript = document.querySelector("#__NEXT_DATA__")
const hydrationData = JSON.parse(hydartionScript.innerHTML)
console.log(hydrationData)

What’s happening here is straightforward. The JS script:

• Selects the <script> element with id __NEXT_DATA__.

• Reads its inner HTML (which is a JSON string).

• Parses it into a JavaScript object.

• Logs it to the console.

Run this directly in the DevTools Console, and you’ll immediately see the result:

What’s interesting is how much structured data you get right away. This includes product details, images, metadata, and more. All is neatly organized, and it only took three lines of code!

If you want to store the JSON hydration object, just right-click the object in the Console and select the “Copy object” option:

From there, you can paste it wherever you need (e.g., into a local .json file, a MongoDB collection, etc.).

Need help with your scraping project?

Approach #2: Target the self.__next_f.push() Elements

Another, more complex approach to scraping Next.js involves pages built with the App Router.

Even if the App Router has been the recommended direction for a while, in my experience, it’s still not as widely adopted as the Pages Router. And honestly, that’s a bit of a gift for us (as scraping hydration data in App Router sites is definitely more complex!)

As a reference, let’s look at the “Business Overview” page on the OpenAI website, which is built with Next.js App Router:

Just like before, open DevTools and inspect the page. This time, focus on the <script> tags inside the <body>:

You’ll notice several scripts containing content like:

self.__next_f.push(<some_data>)

That “<some_data>” is serialized using the React Flight protocol for React Server Components (RSC). I won’t go too deep into the internals here (it’s a dense topic!), but what matters is that deserializing that data is not straightforward!

React Flight isn’t plain JSON. It mixes control records (HL, I, J, etc.), module references, streaming boundaries, and serialized model fragments into a transport format that React incrementally resolves at runtime.

You might think: “Why not just reuse the frontend deserialization library?” In practice, that doesn’t work well because:

• The client decoder (react-server-dom-webpack) expects a full React runtime.

• It relies on module maps and webpack IDs generated at build time.

• It resolves component references against the exact bundle that produced the stream.

• It assumes streaming semantics and internal React wiring.

Basically, outside that exact environment, you don’t have the module graph, build manifest, or hydration context. So even if you import the decoder, you can’t reconstruct the component tree the way the browser does.

There have been recent security issues in the React Flight payload deserialization system, highlighting just how sensitive and complex this layer is. For more details, refer to:

• Security Advisory: CVE-2025-66478

• Critical Security Vulnerability in React Server Components

Thus, instead of fighting the protocol, I’d simplify and accept that in this case, it’s better to extract the unparsed React Flight string data. Achieve that with the JS script below:

const nextFlightScripts = [...document.querySelectorAll("script")]
  .filter(script => script.textContent.includes("self.__next_f"))
  .map(script => script.textContent.trim())
console.log(nextFlightScripts)

This selects all <script> elements containing “self.__next_f” and builds an array of their raw contents.

Run it in the Console, and you’ll get an array of React Flight chunks:

From there, the simplest way to extract structured data is often to copy the array, feed it to an AI, and ask it to reconstruct a parsed JSON representation of the meaningful payload sections:

Is this more complicated than the __NEXT_DATA__ trick? Absolutely! Yet, it’s still a powerful way to access a large amount of page data with just a few lines of code.

Final Script to Quickly Access Data From Next.js Sites

If you combine the two approaches, you can build a production-ready script for brute-force hydration data scraping in Next.js:

// Pages Router approach (__NEXT_DATA__)
const hydrationScript = document.querySelector("#__NEXT_DATA__")
let nextData = null
if (hydrationScript) {
  try {
    nextData = JSON.parse(hydrationScript.textContent)
    console.log("__NEXT_DATA__ found:")
    console.log(nextData)
  } catch (err) {
    console.warn("Failed to parse __NEXT_DATA__:", err)
  }
} else {
  console.log("No __NEXT_DATA__ script found.")
}

// App Router approach (self.__next_f)
const nextFlightScripts = [...document.querySelectorAll("script")]
  .map(script => script.textContent.trim())
  .filter(content => content.includes("self.__next_f.push"))

if (nextFlightScripts.length > 0) {
  console.log("React Flight scripts found:")
  console.log(nextFlightScripts)
} else {
  console.log("No React Flight scripts found.")
}

To test it, just open the Console in DevTools, paste the script, and run it.

Important: The <script> components containing hydration data aren’t loaded dynamically via client-side rendering. They’re embedded directly in the HTML generated by the server.

That means you can:

1. Fetch the target Next.js-powered page with an HTTP client.

2. Parse the HTML using an HTML parsing library like Beautiful Soup or Cheerio.

3. Apply a similar version of the JavaScript script above, but adapt it to the API provided by your HTML parser.

In other words, this trick for scraping Next.js doesn’t only work in the browser DevTools. It also works perfectly in regular scraping scripts!

Pros and Cons of This Approach to Next.js Scraping

👍 Pros:

• Simple and effective, requiring only a few lines of code.

• Works on all Next.js websites (and, more generally, on most sites that rely on hydration).

• Can let you access more data than what is actually displayed on the page.

• No need for browser automation, waiting for client-side rendering, or simulating user interactions.

👎 Cons:

• You may only get partial data, meaning you might still need to complement it with a more traditional scraping approach.

• React Flight data is difficult to parse and may require custom logic or even AI-assisted parsing.

Conclusion

Here, I’ve shared a trick I personally documented years ago, and that still works to this day. It allows you to quickly scrape data from virtually any Next.js site by targeting the hydration data embedded in the HTML document generated by the server and sent to the client for rendering.

As you’ve seen, with just a few lines of JavaScript, you can extract hydration data from any Next.js-powered page. What you get back is clean, or at least almost clean, data that you can process directly in your data pipelines.

Instead of fighting the frontend, this Next.js web scraping approach helps you leverage the data the framework itself needs to function!

I hope you found this useful and insightful. If you have questions or thoughts, feel free to share them in the comments below!

But what is Polymarket, and how does it work?

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

What Polymarket is, and why it keeps making headlines

Polymarket is a prediction market, a platform where you buy and sell shares tied to the outcome of real-world events. If the event happens, your share pays $1. If it doesn’t, it pays $0. The trading price at any moment reflects what the market collectively believes the probability of that outcome is. You can bet on elections, geopolitics, sports, crypto prices, and increasingly anything else with a verifiable resolution.

It is the largest prediction market by volume, built on the Polygon blockchain. Its main competitor, Kalshi, operates as a CFTC-regulated exchange in the US. Both are attracting billions in volume, and Wall Street firms are now building dedicated trading desks around them.

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

The platform handled at least $7.5 billion in volume during 2025 (a conservative figure, since Polymarket volume is commonly double-counted due to how OrderFilled events are summed, and set a single-day record of $425 million in February 2026 when Iran-related markets resolved simultaneously. Those are not toy numbers. And with that kind of money flowing through, the headlines have followed.

In January 2026, a newly created Polymarket account invested $30,000 and walked away with $436,759 after correctly betting on Maduro’s removal from power. The account was created less than a week before the U.S. military operation, and the bulk of bids were placed hours before Trump’s announcement. In a separate case, an Israeli Air Force reservist was indicted for leaking classified details about a strike on Iran to guide Polymarket bets, netting roughly $244,000. A different trader has made nearly $1 million since 2024 from dozens of well-timed bets correctly predicting U.S. and Israeli military actions against Iran, winning 93% of five-figure wagers. These incidents triggered at least eight prediction market bills in Congress since January 2026, and federal prosecutors in Manhattan are actively exploring whether certain prediction market bets violate insider trading laws.

But insider trading is not the only way people make money on Polymarket. There is a quieter, more interesting story happening in parallel.

Start your scraping journey with Byteful: 10GB New Customer Trial | Use TWSC for 15% OFF | $1.75/GB Residential Data | ISP Proxies in 15+ Countries

Claim your 10GB here

The efficiency gap

The Efficient Market Hypothesis, formalized by Eugene Fama in the 1960s, states that asset prices reflect all available information, making it impossible to consistently beat the market. In traditional equity markets, this largely holds because massive institutional capital from hedge funds, pension funds, and proprietary trading firms constantly hunts for and eliminates mispricings. The S&P 500 trades roughly $500 billion daily. Any pricing error gets corrected in milliseconds by algorithms running in colocated data centers.

Polymarket’s individual markets often have only tens of thousands of dollars in liquidity. The ratio of “smart money” to “total market cap” is fundamentally different from equity markets, and that is why edges persist longer than they would on Wall Street. A 2025 study by IMDEA Networks Institute documented $40 million in arbitrage profits extracted from Polymarket alone between April 2024 and April 2025, analyzing 86 million bets. Arbitrage opportunities on the platform last an average of 4 seconds, with 73% of profits captured by bots, executing in under 100 milliseconds.

The institutional side is catching up. DRW is hiring dedicated prediction market traders at a $200,000 base salary. Susquehanna International Group became the first official market maker on Kalshi (a competing platform). Jump Trading is building specialized desks. But the market is not there yet. Liquidity is too thin for these firms to deploy serious capital without moving prices, leaving room for smaller, faster actors.

Check the TWSC YouTube Channel

Speed as edge: what people are building

I’ve been studying Polymarket for some months now, and I’ve probably ended up on a bubble on Instagram and other social media. I’m seeing a growing number of traders share systems designed to exploit exactly this kind of market inefficiency. The approaches vary, but the pattern is the same: faster information, faster execution, profit.

One notable case involves a trader who claims to use computer vision models processing live football match video feeds. His system watches the match in real time, detects events (goals, red cards, penalties) through frame analysis, and places bets on prediction markets seconds before the event registers on official data feeds and bookmaker odds adjust. He claims an 8-second advantage over other traders (unfortunately, I cannot find the post on Instagram about it anymore). Whether that specific claim holds up or not, this is nothing new: courtsiders have been doing this in tennis for years, attending live matches and transmitting scores faster than official data feeds reach bookmakers. In 2016, tennis umpires from Kazakhstan, Turkey, and Ukraine were banned for deliberately delaying score updates for courtside accomplices.

The same principle applies at a larger scale. Bill Benter built a multinomial logit model with over 120 variables per horse for Hong Kong racing and extracted over $1 billion between 1987 and 2001. Tony Bloom’s Starlizard syndicate employs 160 people to model Asian handicap football markets and reportedly generates 600 million GBP per year. On Polymarket itself, 14 of the top 20 most profitable wallets are bots. One bot turned $313 into $414,000 in a single month, exploiting temporal arbitrage in 15-minute crypto markets.

All of these systems share two requirements: data and speed. They need real-time access to market prices, order books, and event outcomes, and they need to act on that data faster than everyone else. All of this is possible because Polymarket provides a full set of APIs that can be used to operate programmatically on the platform.

Need help with your scraping project?

Polymarket’s API architecture

Polymarket exposes three distinct APIs, each serving a different purpose. Understanding which one to use and when is the first step toward building anything that trades or monitors this market.

Gamma API: market discovery

The Gamma API is the browsing layer. It returns human-readable market data: questions, descriptions, outcome prices, volume, liquidity, event metadata. No authentication required.

Base URL: https://gamma-api.polymarket.com

Key endpoints:

- GET /markets returns a paginated list of markets with filtering options (limit, offset, closed, tag_id)

- GET /markets/{id} returns a single market by ID or slug

- GET /events and GET /events/{id} return event-level data (events group related markets)

- GET /search?query=... performs keyword search across markets and events

A single call to /markets?limit=1&closed=false returns something like this:

{
  "id": "540816",
  "question": "Russia-Ukraine Ceasefire before GTA VI?",
  "conditionId": "0x9c1a953fe92c8357f1b646ba25d983aa83e90c525992db14fb726fa895cb5763",
  "outcomes": "[\"Yes\", \"No\"]",
  "outcomePrices": "[\"0.545\", \"0.455\"]",
  "volume": "1516211.89",
  "liquidity": "62104.61",
  "clobTokenIds": "[\"850149715...\", \"252731249...\"]"
}

The `clobTokenIds` field is the bridge to the trading layer. Each outcome (Yes/No) gets its own token ID, which is what you pass to the CLOB API to get real-time prices and order book data.

The Gamma API is rate-limited at roughly 60 requests per minute. It is useful for discovery and metadata, not for real-time price monitoring.

CLOB API: the order book

The CLOB (Central Limit Order Book) API is where trading happens. It has both public and authenticated endpoints.

Base URL: https://clob.polymarket.com

Public endpoints (no authentication):

- GET /price?token_id=X&side=BUY|SELL returns the current best price

- GET /midpoint?token_id=X returns the midpoint between best bid and ask

- GET /spread?token_id=X returns the current spread

- GET /book?token_id=X returns the full order book with all bids and asks

- GET /last-trade-price?token_id=X returns the last executed trade price

- GET /tick-size?token_id=X returns the minimum price increment

A call to /midpoint returns a minimal payload:

{"mid": "0.545"}

The /book endpoint returns the full depth:

{
  "market": "0x9c1a95...",
  "asset_id": "850149715...",
  "bids": [
    {"price": "0.54", "size": "15234.50"},
    {"price": "0.53", "size": "8920.00"}
  ],
  "asks": [
    {"price": "0.55", "size": "12100.00"},
    {"price": "0.56", "size": "6500.00"}
  ]
}

These public endpoints are what matters for price monitoring. They are lightweight, return small payloads, and have no authentication overhead.

Authenticated endpoints require a two-level authentication system:

Level 1 (L1) uses EIP-712 wallet signatures. You sign a structured message proving you control a specific Ethereum wallet address. This is a one-time operation that generates API credentials:

POST /auth/api-key
Headers: POLY_ADDRESS, POLY_SIGNATURE, POLY_TIMESTAMP, POLY_NONCE
Returns: { apiKey, secret, passphrase }

Level 2 (L2) uses HMAC-SHA256 signing on every request. Every authenticated call requires five headers: POLY_ADDRESS, POLY_SIGNATURE (computed HMAC of the request), POLY_TIMESTAMP, POLY_API_KEY, and POLY_PASSPHRASE.

Even with L2 auth, placing an order requires the user to sign the order payload locally with their private key. Three cryptographic operations total: key derivation (once), request signing (per call), order signing (per order).

The authenticated endpoints are:

- POST /order places a single order

- POST /orders places a batch of orders

- DELETE /order cancels an order

WebSocket feeds provide real-time streaming at wss://ws-subscriptions-clob.polymarket.com/ws/ for order book updates, price changes, and user-specific events.

Data API: analytics

The Data API at https://data-api.polymarket.com provides analytics-oriented data: user positions, trade history, leaderboards, and holder information. It is less documented and less stable than the other two. Some endpoints returned 404 or empty responses during our testing. Useful for research, not reliable for production.

The speed game: calling the APIs as fast as possible

If arbitrage opportunities on Polymarket last 4 seconds on average, and 73% of profits go to bots executing in under 100 milliseconds, then the speed at which you can read prices and place orders is a direct competitive advantage. We set up a benchmark to answer two questions: where should you run your code, and which language and HTTP strategy gets you there fastest?

We did our tests and chose the /midpoint endpoint for the benchmark because it requires no authentication, returns the smallest possible payload, and isolates HTTP client performance from payload parsing. Each benchmark runs 1,000 requests in two modes: sequential (one request at a time, measuring per-request latency) and concurrent (50 simultaneous workers, measuring throughput).

As always, the code can be found in our GitHub repository reserved to paying users, inside the folder 102.POLYMARKET.

Read more

In particular, your scrapers leak information at four distinct layer levels. Modern anti-bot systems, in fact, fingerprint your browser, analyze your TLS handshake, trace your network infrastructure, and track your behavioral patterns. And a single inconsistency across these layers triggers permanent blocking.

This means your scrapers aren’t competing only against rate limits anymore. Today, they are competing against machine learning models trained on billions of legitimate requests, and any deviation from the expected pattern is a signal. So, if you want to scrape at scale, your infrastructure must be indistinguishable from a real user’s browser, network stack, and behavior.

This article guides you through a systematic approach: First, understanding where leaks occur, then learning how anti-bot systems detect them, and finally building a layered defense that makes your scraper invisible.

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

Identifying the Leaks: Where Your Scraper Exposes Itself

Before fixing anything, you need to understand the complete attack surface. Modern anti-bot systems analyze your scraper at four distinct layers, and a leak at any layer can expose you.

Layer 1: The Browser Level

Headless browsers are loud by default. Launch a Puppeteer instance and check the navigator.webdriver flag. It surely returns true, and that’s a signal every major anti-bot system checks in the first 100ms of page load.

But this obvious flag is just the beginning. Anti-bot systems probe deeper:

• Error messages and stack traces: They differ between headless and headed modes. The execution context leaves fingerprints in error objects.

• Window dimensions: Properties like window.outerWidth and window.outerHeight reveal a headless operation because headless mode doesn’t render a visible window frame.

• Canvas rendering: They can produce pixel-level differences. Software rendering (headless) creates different anti-aliasing and color values than GPU-accelerated rendering (headed). Color channels can differ by 1-2 units per pixel.

• WebGL shader timing: This can vary a lot, depending on the underlying technology. GPU-accelerated browsers complete WebGL operations in microseconds. Software-rendered headless browsers take milliseconds.

• Font rendering: Headless environments often lack the full system font stack. This creates detectable layout differences when JavaScript measures text dimensions.

• Performance benchmarks: When run, they can reveal software rendering. For example, there are websites that run JavaScript stress tests, creating thousands of DOM elements, calculating layouts, and triggering reflows. In such scenarios, real browsers with GPU acceleration show consistent performance. Headless browsers, instead, show different timing patterns.

• The window.chrome object behaves differently: Real Chrome populates this object with specific properties for extension management and runtime APIs. Headless Chrome, instead, either lacks this object or provides an incomplete implementation.

Layer 2: The Network Level

Your SSL/TLS handshake identifies you before you send any application data. When your scraper connects over HTTPS, it sends a TLS Client Hello message containing supported encryption methods, protocol versions, and extensions. All in a specific order.

Here’s what makes this dangerous:

• Every browser and HTTP library has a unique TLS pattern: Real browsers send their TLS parameters in a specific sequence that matches their version and underlying platform. Python’s standard HTTP libraries send a completely different pattern. So do Node.js, Go, and any other programming language you use for coding your scrapers.

• Anti-bot systems fingerprint your TLS handshake: They capture these patterns and convert them into a fingerprint, commonly called a JA3 hash. They maintain databases of known fingerprints for every major browser and HTTP library.

• Mismatches between User-Agent and TLS fingerprint are instant red flags: When you claim to be Chrome in your User-Agent header but your TLS handshake matches Python’s urllib library, that inconsistency triggers blocking.

• Detection happens before you send any application data: The first TCP connection already identifies you as automated traffic.

• HTTP/2 fingerprinting adds another layer: Beyond TLS, the order and priority of HTTP/2 frames, settings, and window updates create additional fingerprints. Your HTTP library’s frame ordering must match your claimed browser identity.

For your scraping needs, having a reliable proxy provider like Decodo with high reputatation IPs, on your side improves the chances of success.

Try Decodo Now

Layer 3: The Infrastructure Level

Your proxy configuration can expose your real infrastructure through network-level leaks via the following main mechanisms:

• DNS leaks: They happen when your browser resolves domain names using your local DNS server instead of routing through the proxy. Your scraper might send requests through a Miami residential proxy, but if DNS queries go through your AWS datacenter in Virginia, the target site knows your real location.

• WebRTC leaks: WebRTC is a browser API designed for peer-to-peer communication. Even with a proxy configured, WebRTC will attempt to discover your real local IP and public IP through STUN servers, completely bypassing your proxy.

• IP reputation: Not all IPs are created equal. Cloudflare and similar services maintain databases of every AWS, Google Cloud, and Azure IP range. Requests from known cloud providers receive instant higher suspicion scores before any other analysis happens.

Layer 4: The Behavioral Level

Even if your browser, network, and infrastructure are perfectly disguised, your behavior patterns can still expose you:

• Timing patterns: Requesting data at fixed and precise intervals creates a perfect periodicity. No human browses with mathematical precision.

• Mouse and scroll behavior: Real humans accelerate and decelerate smoothly. Instant jumps from point A to point B are mechanically impossible.

• Session state: Stateless scrapers that never accumulate cookies or maintain persistent sessions across days look like fresh bots on every run.

• Interaction sequences: The time between page load and first click, between mouse-over and click, or the pattern of how you scroll through content. They all follow detectable human patterns.

Understanding the Detection: How Anti-Bot Systems Catch You

Now that you know where leaks occur, let’s understand how anti-bot systems actually detect them.

Fingerprint Consistency Checks

Anti-bot systems cross-reference your claimed identity with actual behavior. If your User-Agent says “Chrome 120 on Windows 10,” they verify that your JavaScript features, WebGL capabilities, canvas rendering, and TLS handshake all match Chrome 120 on Windows 10.

A single mismatch anywhere flags the entire request. You can’t be Chrome in your User-Agent, Firefox in your TLS handshake, and headless Chrome in your canvas fingerprint. Anti-bot systems create composite fingerprints combining dozens of properties, then compare them against databases of known legitimate and bot patterns.

Machine Learning Pattern Recognition

Modern anti-bot systems use ML models trained on billions of requests. They learn what “normal” looks like for each type of visitor. This means that consumer browsers from residential IPs have different behavioral patterns than datacenter scrapers.

For ML models, statistical anomalies trigger investigation. Perfect timing intervals, impossible mouse movements, or timing patterns that don’t match human variance distributions are scored as anomalous. These models adapt continuously, so when new stealth techniques emerge, the models retrain on that data. This means that what works today might fail tomorrow.

Progressive Trust Scoring

Anti-bot systems block or allow requests, but they also score. This means that lower trust scores receive degraded service: slower response times, rate limits, or CAPTCHA challenges before blocking.

Also, scores accumulate across sessions. If you leak information across multiple visits, the system builds a profile associating your various identities. In other words, one leak can poison future requests, and even fixing the leak might not restore trust if your IP or fingerprint is already marked.

Check the TWSC YouTube Channel

Building the Defense: A Layered Approach to Stealth

Building a defense from data leaks in web scraping requires addressing each layer systematically. Your stealth stack must work from the inside out: browser → network → infrastructure → behavior. Each layer must remain consistent with your claimed identity.

Defense Layer 1: Hardening the Browser

The goal at this layer is to make the browser fingerprint indistinguishable from a real user’s browser and ensure every property is consistent with your claimed identity.

Step 1: Mask Automation Signals

Start with stealth libraries that patch the most common detection vectors:

• For Puppeteer: Use puppeteer-extra-plugin-stealth to automatically override navigator.webdriver, DevTools Protocol signatures, and plugin arrays.

• For Selenium: Use undetected-chromedriver, which patches automation signals and uses real Chrome binaries instead of ChromeDriver.

• For Playwright: Leverage native evasion features that handle many detection vectors out of the box.

Additionally, disable automation flags at launch. For example, in Playwright:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(
        headless=True,
        args=[
            '--disable-blink-features=AutomationControlled',
            '--disable-dev-shm-usage',
            '--no-sandbox'
        ]
    )

But remember: Stealth libraries handle the most common 20-30 leak vectors but miss advanced fingerprinting techniques. They’re your foundation, not your complete solution.

Step 2: Spoof Hardware Signatures

Cloud server canvas and WebGL fingerprints are obvious red flags. AWS, GCP, and Azure rendering signatures are well-known to anti-bot systems.

You have two approaches for your defense here:

• Add consistent noise: Inject deterministic noise into canvas operations so the fingerprint remains stable across sessions but doesn’t match your server’s real hardware. Override canvas methods to modify pixel data slightly before it’s read back. Keep noise minimal: just enough to mask the real hardware signature without appearing obviously manipulated.

• Emulate common consumer hardware: Spoof WebGL parameters to mimic common consumer GPUs. Override vendor and renderer strings returned by WebGL APIs to match your chosen hardware profile. Use existing libraries designed for canvas fingerprint defense or implement your own parameter overrides.

Step 3: Ensure Version Consistency

This is where most scrapers fail, even with stealth libraries. Your User-Agent string must match your actual browser engine behavior precisely. Consider the following rules of thumb:

• Use real browser binaries instead of spoofing: Tools like Playwright can launch actual Chrome, ensuring perfect consistency between claimed version and actual behavior.

• If you must spoof, maintain complete version profiles: Track which JavaScript features, WebGL capabilities, and API behaviors correspond to each browser version. Every property must align.

• Never mix components from different versions: If you claim Chrome 120 on Windows 10, every single API, from JavaScript features to WebGL renderers, must behave exactly like Chrome 120 on Windows 10.

Defense Layer 2: Hardening the Network Stack

Your goal at this layer is to make your TLS handshake and HTTP traffic indistinguishable from the browser you’re claiming to be.

Step 4: Match TLS Fingerprints to Your Browser Identity

Standard HTTP libraries can’t mimic browser TLS fingerprints because they use different SSL/TLS implementations. The solution requires specialized libraries that replicate browser behavior at the protocol level:

• For Python: Use curl_cffi or similar wrappers. These libraries use libcurl compiled with BoringSSL, which is the same SSL library Chrome uses. This creates identical JA3 fingerprints to real browsers.

• For Node.js: Use cycletls or equivalent libraries that allow you to specify exact JA3 fingerprint strings matching real browsers.

Critical requirement: Your TLS fingerprint must match your User-Agent. Chrome 120’s JA3 fingerprint is different from Firefox 115’s fingerprint. The browser identity must be consistent across all layers.

Step 5: Match HTTP/2 Fingerprints

Beyond TLS, HTTP/2 frame ordering creates additional fingerprints. Libraries like curl_cffi handle this automatically when you specify a browser to impersonate, but verify that:

• Settings frames match your target browser.

• Window update sequences align.

• Priority headers follow the correct pattern.

In Python, you can do so with the following code:

response = requests.get(
    '<https://tls.peet.ws/api/all>',
    impersonate='chrome120'
)
print(response.json()['http2']['sent_frames'])

Need help with your scraping project?

Defense Layer 3: Hardening Infrastructure

Your goal at this layer is to ensure your network traffic originates from legitimate-looking IPs and doesn’t leak your real location or identity.

Step 6: Choose the Right Proxy Type

IP reputation is the first filter that anti-bot systems check. This means that your proxy choice determines your baseline trust score. Consider the following guidelines:

• Datacenter IPs = instant red flag: Requests from AWS, Google Cloud, and Azure IP ranges receive instant higher suspicion scores.

• Residential proxies = highest legitimacy: These IPs come from real ISP connections, so they look legitimate because they are legitimate consumer connections.

• Mobile proxies = premium legitimacy: These IPs originate from cellular networks (4G/5G) and receive the highest trust scores. Mobile IPs rotate naturally as devices move between cell towers, making them appear even more organic than static residential connections.

Step 7: Prevent DNS Leaks

Force all DNS resolution through your proxy tunnel. For SOCKS5 proxies, use the SOCKS5h protocol variant, which forces DNS resolution on the remote proxy server instead of locally.

For example, in Python, write the following:

import requests

proxies = {
    'http': 'socks5h://proxy.example.com:1080',
    'https': 'socks5h://proxy.example.com:1080'
}

response = requests.get('<https://example.com>', proxies=proxies)

For browser automation, configure DNS-over-HTTPS to prevent local DNS leakage. The following is an example that applies to Playwright:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(
        args=[
            '--dns-over-https-server=https://cloudflare-dns.com/dns-query'
        ]
    )

Step 8: Disable WebRTC Completely

WebRTC will expose your real IP unless you completely disable it in browser automation. For example, in Playwright, you can do so as follows:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    
    # Remove WebRTC entirely
    await page.add_init_script("""
        delete window.RTCPeerConnection;
        delete window.RTCSessionDescription;
        delete window.RTCIceCandidate;
        delete navigator.mediaDevices;
    """)

When youìve done this, verify it’s actually disabled before deploying your scraper. Visit browserleaks.com/webrtc with your scraper. You should see “WebRTC is not supported by your browser”, or only your proxy IP should be visible. Never your real IP.

Defense Layer 4: Mimicking Human Behavior

Your goal at this layer is to make your interaction patterns indistinguishable from those of real human users.

Step 9: Add Timing Jitter and Randomization

Humans are inconsistent. Perfect patterns are robotic. The solution here is not to just add randomness. You also need to match the statistical distribution of real human behavior. To do so, consider the following example in Python:

import numpy as np
import time

# Wrong example (do not use this)

# Fixed interval
time.sleep(5)  # Always 5 seconds - DETECTABLE

# Random uniform
time.sleep(random.uniform(3, 7))  # Still doesn't match human patterns

------------

# Correct example (use this!)

# Log-normal distribution (matches real human reaction times)
delay = np.random.lognormal(mean=1.5, sigma=0.5)
time.sleep(delay)

For improving randomization, model different action types with appropriate distributions. Use the following rules of thumb:

• Clicks: 0.3-2 seconds (short delays)

• Reading: 5-45 seconds (high variance)

• Scrolling: 1-8 seconds (irregular intervals)

Step 10: Implement Realistic Mouse and Scroll Behavior

High-security sites like banking, ticketing, and heavily protected e-commerce websites track interaction patterns in real-time. To defend from leaking your information on such websites, you have to define mouse movements and scrolling for your automated scripts.

For mouse movements, you can:

• Use Bezier curves to create natural arcing movements between points.

• Add slight randomness to destination coordinates.

• Include hover delays before clicking.

• Vary the number of intermediate steps based on distance.

The following is an example you can try in Python:

import numpy as np
from playwright.sync_api import sync_playwright

def bezier_curve(start, end, control_points, num_steps=20):
    """Generate points along a Bezier curve for natural mouse movement"""
    t = np.linspace(0, 1, num_steps)
    points = []
    
    # Simplified cubic Bezier
    for t_val in t:
        x = (1-t_val)**3 * start[0] + \\
            3*(1-t_val)**2*t_val * control_points[0][0] + \\
            3*(1-t_val)*t_val**2 * control_points[1][0] + \\
            t_val**3 * end[0]
        y = (1-t_val)**3 * start[1] + \\
            3*(1-t_val)**2*t_val * control_points[0][1] + \\
            3*(1-t_val)*t_val**2 * control_points[1][1] + \\
            t_val**3 * end[1]
        points.append((x, y))
    
    return points

async def human_like_click(page, selector):
    element = await page.query_selector(selector)
    box = await element.bounding_box()
    
    # Add slight randomness to destination
    target_x = box['x'] + box['width']/2 + np.random.normal(0, 2)
    target_y = box['y'] + box['height']/2 + np.random.normal(0, 2)
    
    # Move mouse along curve
    current_pos = await page.mouse.position()
    control_points = [
        (current_pos['x'] + np.random.uniform(-50, 50), 
         current_pos['y'] + np.random.uniform(-50, 50)),
        (target_x + np.random.uniform(-20, 20), 
         target_y + np.random.uniform(-20, 20))
    ]
    
    points = bezier_curve(
        (current_pos['x'], current_pos['y']), 
        (target_x, target_y), 
        control_points
    )
    
    for x, y in points:
        await page.mouse.move(x, y)
        await page.wait_for_timeout(np.random.uniform(5, 15))
    
    # Hover briefly before clicking
    await page.wait_for_timeout(np.random.uniform(100, 300))
    await page.mouse.click(target_x, target_y)

For scrolling, you can:

• Pause between scroll actions for variable amounts of time (simulating reading).

• Scroll in chunks of varying size, not uniform pixels.

• Occasionally scroll backwards (humans re-read).

• Don’t scroll in perfect increments or at constant speeds.

Use the following Python code to try such scrolling behaviour:

async def human_like_scroll(page, total_distance):
    """Scroll with human-like patterns"""
    scrolled = 0
    
    while scrolled < total_distance:
        # Vary chunk size
        chunk = np.random.randint(100, 400)
        
        await page.mouse.wheel(0, chunk)
        scrolled += chunk
        
        # Pause to simulate reading
        pause = np.random.lognormal(mean=1.2, sigma=0.8)
        await page.wait_for_timeout(pause * 1000)
        
        # Occasionally scroll backwards (humans re-read)
        if np.random.random() < 0.15:
            await page.mouse.wheel(0, -np.random.randint(50, 150))
            await page.wait_for_timeout(np.random.uniform(500, 1500))

Step 10: Maintain Persistent Session State

Stateless scrapers look like stateless bots. Real browsers, instead, accumulate state over time because:

• Cookies persist across requests and sessions.

• LocalStorage accumulates tracking data over time.

• Session IDs remain stable across days or weeks.

To mimic real browser states, you can use the following Python code:

import pickle
import requests

# Save cookies to disk after each session
session = requests.Session()

# ... perform scraping ...

with open('cookies.pkl', 'wb') as f:
    pickle.dump(session.cookies, f)

# Before next scraping session
with open('cookies.pkl', 'rb') as f:
    session.cookies.update(pickle.load(f))

In case you use a browser automation tool:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    
    # Save browser storage state
    context = browser.new_context()
    # ... perform scraping ...
    context.storage_state(path='state.json')
    
    # Reload in next session
    context = browser.new_context(storage_state='state.json')

As a final note, consider keeping sessions alive for weeks to allow third-party tracking cookies to build up. Long-lived sessions with accumulated tracking data appear more legitimate than constantly refreshed clean states.

Conclusion

In this article, you learned that, if you don’t want your data to be leaked while scraping, you have to take several defensive measures, as no single technique makes you invisible. Anti-bot systems analyze multiple signals simultaneously, and any inconsistency across layers triggers detection and blocks your scrapers.

Also, detection methods evolve. So, what works today might fail tomorrow. This means you should also monitor the defenses you implemented and test new ones.

Now, let us know: How do you prevent data leaks in your scrapers? Did we miss some technique?

Rayobyte, the proxy and web scraping solutions provider, has taken a different approach. They recently released rayobrowse, a closed‑source yet Docker‑based, self‑hostable stealth browser built for local browser automation and web scraping.

In this post, I’ll take a deep look at this solution and walk you through everything you need to know about it. By the end, you’ll understand what rayobrowse is, how its stealth browser approach works, how to set it up, and whether it’s actually worth paying attention to.

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

An Introduction to rayobrowse

Let me introduce you to the world of rayobrowse, helping you understand what it is and what makes this project special.

What is rayobrowse?

rayobrowse is a self-hosted, Chromium-based stealth browser engineered for web scraping, AI agents, and automation workflows. It’s available as a Docker image, with optional support via a Python SDK (rayobrowse on PyPI) for simplified connection. The project is developed and maintained by Rayobyte.

The stealth browser runs inside Docker and is available via the Chrome DevTools Protocol (CDP). That means tools like Playwright, Puppeteer, and Selenium (or any other tool that speaks CDP) can natively connect to it for automation purposes.

What makes it noteworthy is its approach to device fingerprinting. User agents, screen size, WebGL, fonts, timezone, and other signals are tuned so each session looks like a real browser. That way, it helps your automation avoid detection on protected websites.

Core Principles Driving the Solution

These are the core principles and goals behind the project:

1. It should run on Linux server environments without GPUs or a GUI/desktop interface.

2. It should patch Chromium at the C++ level, rather than at higher layers like CDP, which are easier for anti-bot systems to detect.

3. It should work with Playwright, a common framework in browsing automation stacks.

4. It should support both headful mode (via Xvfb) and headless mode.

5. It should emulate fingerprints from real-world devices across different regions.

6. It should be self-hostable, so you can run it locally without relying on cloud infrastructure.

7. It should be free to test and use for certain user segments.

8. It should reliably bypass major anti-bot systems and scraping targets, including complex ecommerce and SERP platforms.

Note: If you’re not familiar with Xvfb, that’s an in‑memory display server for Unix-like systems that implements the X11 display protocol without requiring a physical display or input devices. In simpler terms, it allows GUI applications to run in headless environments. rayobrowse relies on it to launch headful browser sessions even on servers without a graphical interface (that’s beneficial as headful sessions are harder to detect than purely headless ones).

Main Features for Stealth Browsing and More

Here is a list of the most relevant rayobrowse features:

• Fingerprint spoofing: Each browser session comes with a real-world realistic device fingerprint drawn from a database of thousands of profiles. Signals include user agent, OS metadata, screen resolution, fonts, WebGL, hardware concurrency, and timezone.

• Human‑like mouse movement: Optional human‑style cursor behavior (inspired by HumanCursor) makes automation appear more natural. When using standard Playwright actions like page.click() or page.mouse.move(), the library applies realistic curves and timing.

• Proxy Integration: Traffic can be routed through any HTTP proxy, including authenticated and rotating proxies.

• Headless and headful Support: rayobrowse supports both execution modes, even on GUI-less Linux servers.

• Live session viewer: A built‑in noVNC interface (available at http://localhost:6080) lets you watch browser sessions in real time directly from the browser. This is particularly useful for debugging scraping flows and visually verifying fingerprint behavior.

• Official integrations: The browser integrates with common automation frameworks, namely Playwright, Puppeteer, Selenium, and Scrapy (via scrapy-playwright), as well as emerging AI‑driven tools such as OpenClaw. As of this writing, additional integrations (e.g., Firecrawl and LangChain) are planned.

• Remote/Cloud mode: rayobrowse can run as a remote browser service. Your server requests new browser instances through a REST API, and workers connect directly to the returned CDP WebSocket endpoint. This is still a beta feature.

• API‑driven browser management: The daemon exposes REST endpoints for creating, listing, and deleting browser sessions, allowing you to orchestrate multiple browsers across a distributed scraping infrastructure.

Technical Details About the Project

Now that you know what the project is and the features it provides, you’re ready to dive into the technical aspects.

How rayobrowse Works

At a high level, rayobrowse follows these steps:

1. Chromium patching: The project tracks upstream Chromium releases and applies a focused set of patches (relying on an approach similar to Brave’s “plaster” model). These patches normalize exposed browser APIs, reduce fingerprint entropy leaks, improve automation compatibility, and preserve native Chromium behavior whenever possible.

2. Fingerprint assignment: When a browser session starts, rayobrowse assigns a realistic device fingerprint.

3. Automation integration: Browser automation libraries connect to rayobrowse through the native CDP.

Architecture

Architecturally, rayobrowse follows a clean separation between the browser runtime and the automation code.

In particular, the system runs as a Docker container that bundles three core components:

1. A daemon server that manages browser sessions.

2. A browser manager that downloads and retrieves the correct version of Chromium, a fingerprint engine that injects realistic device profiles, and a stealth browser layer containing a custom Chromium build with stealth patches.

3. A noVNC viewer, which lets you watch browser sessions in real time. This is useful for debugging and demos.

As you can see, the automation scripts don’t run inside the container. Instead, they run on the host machine and connect to the browser remotely through the Chrome DevTools Protocol.

When a new session starts, rayobrowse assigns a real-user-looking fingerprint from a large database of actual devices, containing thousands of permutations collected from websites Rayobyte owns.

Requirements

The rayobrowse project is designed to run on Linux servers without GPUs (which is a common deployment environment).

These are the required prerequisites:

• Docker, as the browser runs entirely inside a container.

• ~2GB of available RAM, as each browser instance uses ~300MB.

The main benefit of this Docker-based approach is that you don’t need to install Chromium locally, configure fonts, or set up Xvfb manually. All of those dependencies live inside the container, which keeps the host machine clean, portable, and reproducible.

It also makes the project well-suited for self-hosted environments without exposing its internal Chromium patching logic, making it much harder for anti-bot solution providers to reverse engineer how it works.

In terms of compatibility, rayobrowse works on Linux, Windows (native or WSL2), and macOS. The supported architectures are x86_64 (amd64) and ARM64 (Apple Silicon and AWS Graviton). Still, you don’t have to worry about the architecture, as Docker automatically pulls the correct image for the host machine.

Optional: If you plan to use the stealth browser through the Python SDK, an additional requirement is Python 3.10+.

Check the TWSC YouTube Channel

How to Access rayobrowse

There are two main ways you can access rayobrowse:

1. The /connect endpoint.

2. The built-in Python SDK.

Method #1: Use the /connect Endpoint

The first rayobrowse usage method involves connecting directly to the /connect endpoint. This allows any CDP‑compatible tool (including Selenium, Playwright, and Puppeteer) to open a browser session simply by pointing to a WebSocket URL like ws://localhost:9222/connect.

For instance, take a look at the Playwright connection example below:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    # Connect to rayobrowse via CDP
    browser = p.chromium.connect_over_cdp("ws://localhost:9222/connect")
    page = browser.new_context().new_page()

    # Automation logic...

    browser.close()

Keep in mind that the WebSocket browser connection URL can be customized using query parameters, as follows:

ws://localhost:9222/connect?headless=false&os=android&proxy=http://user:pass@host:port

This URL creates a rayobrowse Chromium browser session in headful mode, using Android-based fingerprints, while routing all requests through the proxy http://user:pass@host:port.

Explore all /connect query parameters in the docs.

Method #2: Use the Python SDK

You can also interact with rayobrowse through the built-in Python SDK. This exposes a create_browser() function that returns a CDP WebSocket URL for a newly created browser instance. From there, connect using Playwright or another automation framework, as shown below:

from rayobrowse import create_browser
from playwright.sync_api import sync_playwright

# Configure the rayobrowse connection to run in headful mode 
# while simulating a Windows-based fingerprint
ws_url = create_browser(headless=False, target_os="windows")

with sync_playwright() as p:
    # Connect to rayobrowse with the configured URL via CDP
    browser = p.chromium.connect_over_cdp(ws_url)
    page = browser.contexts[0].pages[0]
 
    # Automation logic...

    browser.close()

This approach gives you more control over the browser lifecycle, but it also involves more configuration and setup.

For more examples (e.g., proxy integration, multi-browser management, etc.), check out the docs.

Get Started with rayobrowse: Step-by-Step Guide

In this guided section, I’ll show you how to build a simple Playwright script that connects to rayobrowse.

For the sake of simplicity, I’ll assume you already have:

• A Unix-based system (Linux, macOS, or Windows via WSL).

• Docker installed and running on your machine.

• Git installed locally.

• A Python environment set up with Playwright installed.

Follow the instructions below!

Step #1: Clone the rayobrowse Repository

The first step is to clone the rayobrowse repository to your machine:

git clone https://github.com/rayobyte-data/rayobrowse

Then, enter the project folder with:

cd rayobrowse

The cloned folder already includes everything you need to get started, including:

• docker-compose.yml: For running the browser container.

• requirements.txt: For installing the Python SDK.

Step #2: Set Up the Environment

rayobrowse requires a .env file that contains the configuration needed to run the browser daemon. For a full list of available environment variables and what they enable, explore the official documentation.

Start by creating a .env file as a copy of the .env.example file coming with the repository:

cp .env.example .env

Then open the .env file and make sure it contains:

STEALTH_BROWSER_ACCEPT_TERMS=true

This confirms that you accept the project’s LICENSE. Without that setting, the daemon will refuse to create browser sessions.

Step #3: Start the Docker Container

Launch the rayobrowse Docker container:

docker compose up -d

Docker will automatically pull the appropriate image for your system architecture (x86_64 or ARM64). Then, it’ll start the container, as explained earlier.

Step #4: Connect via CDP and Apply the Automation Logic

You can now connect to the running rayobrowse instance through the /connect endpoint using any CDP-compatible client. In this example, I’ll use Playwright with Python:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    # Connect to the rayobrowse browser through the CDP WebSocket endpoint
    browser = p.chromium.connect_over_cdp(
        "ws://localhost:9222/connect?headless=false&os=windows"
    )

    # Create a new browser context and page
    page = browser.new_context().new_page()

    # Navigate to the target (sample) page
    page.goto("https://quotes.toscrape.com/")

    # Print the page title to verify the session is working
    print(page.title()) # Output: "Quotes to Scrape"

    # Add your scraping logic here...

    # Close the browser session
    browser.close()

At this point, write your scraping or automation logic, which will run inside the stealth Chromium browser provided by rayobrowse.

For debugging, you can watch the browser session live through noVNC at http://localhost:6080/vnc.html. While the script is running, you should see a headful Chromium session opening and navigating to the target page specified in the script:

As you can tell, the server creates a headful Chromium session (due to the headless=false query parameter) and connects it to the page requested by the script.

Optional: If you want more control over the browser lifecycle, install the Python SDK with:

pip install -r requirements.txt

Take a look at the official examples in the repository for more guidance.

Need help with your scraping project?

Pricing and Limitations

This is how the rayobrowse pricing model works:

What matters most for us, developers, is that you can run rayobrowse for free via self‑hosting. In practice, the only real cost comes from proxies, which are necessary for scaling scraping workloads and avoiding IP bans (something that’s standard in most production scraping setups).

The main thing to keep in mind is that rayobrowse is still in beta. Rayobyte already uses it to scrape millions of pages per day, but results can vary depending on the target site and configuration.

Fingerprint coverage is currently strongest for Windows and Android, while macOS and Linux profiles are less mature. In addition, Canvas and WebGL fingerprinting are still evolving, which means some websites may detect the current implementation.

Benchmarks and Final Comment

To put rayobrowse to the test, I ran a simple script against a single page for each of the most popular anti‑bot detection systems. These are the results I obtained:

Note: These tests were performed on my local machine using my ISP’s IP address.

As you can see, in this simple experiment rayobrowse achieved a 100% success rate, while Playwright failed consistently in headless mode and even struggled in some headful scenarios.

This suggests that the project is definitely worth keeping an eye on, especially thanks to its self‑hosted nature.

To be honest, and this is just my personal opinion as an expert who works in this field, I don’t usually get very excited about projects like this…. In my experience, many libraries of this type either get cracked down on or simply don’t receive the long‑term support they deserve. In this case, however, things are a bit different. The project is closed‑source and backed by a well‑known company in the industry, which makes the expectations for its future understandably much higher!

Here, I covered what the project is about, what it offers, how it works, and how to use it. As always, remember to use rayobrowse only for legal and ethical web scraping. Until next time!

FAQ

Why is rayobrowse based on Chromium and not Chrome?

rayobrowse is based on Chromium simply because Chrome is closed-source. Plus, tests performed on difficult websites show no meaningful difference in detection rates between Chrome and Chromium. Using Chromium also avoids false positives and reflects the broader ecosystem of Chromium-based browsers like Brave, Edge, and Samsung Internet.

Is rayobrowse open source?

rayobrowse isn’t open-source to prevent anti-bot companies from reverse-engineering it. Similar projects, like Camoufox, were quickly studied and countered once their code became public. Rayobyte decided to keep the project closed-source to help maintain its effectiveness and reliability over the long term.

Can everyone use rayobrowse?

No, not all companies can use rayobrowse. Its license prohibits organizations listed in Rayobyte’s restricted list from using the software. For everyone else, the project is free to download and run locally.

Does rayobrowse support proxy integration?

Yes, Rayobrowse fully supports proxy integration. You can route traffic through any HTTP proxy using the proxy query parameter on the /connect endpoint or via the proxy option exposed by the create_browser() function from the Python SDK. The proxy support includes authentication and rotating proxies.

We have seen this pattern in every scraping operation we have worked with. The knowledge exists. It is just scattered across wikis, Slack threads, internal repos, and people’s heads. The real problem is not documentation; it is retrieval. People write things down. They just cannot find them when it matters.

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

In THE LAB #77, we explored the concept of RAG (Retrieval-Augmented Generation) applied to scraped data and showed how to build a basic knowledge assistant using FAISS. That was a proof of concept. This time we are going deeper. We are showing the production system we actually built and use daily, and we are explaining the reasoning behind each design choice: why markdown, how embeddings work, which chunking strategy actually performs better, and what role auto-tagging plays in retrieval.

After reading this article, we hope you will understand the mechanics well enough to build the same system for your team.

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

What we are building and why

At TWSC, we have published around 300 articles over the past four years. Tutorials, reverse-engineering deep dives, tool comparisons, anti-bot analysis. When we sit down to write a new article, we need to remember what we have already covered, find previous work to link to, and check whether a technique we are about to describe was already explained in a past issue. Doing this by memory or by searching Substack’s archive stops working after the first hundred articles.

We also follow what the broader community publishes. Projects like Crawl4AI, which appeared on Hacker News, show that the need to ingest web content into structured, LLM-ready knowledge bases is shared across the industry. The tools for crawling and extracting content keep getting better, but the retrieval side, finding the right piece of information in a growing archive, still requires a purpose-built system.

So we built one. Here is what the complete pipeline looks like:

Sources                                  Processing              Storage & Retrieval
─────────                                ──────────              ───────────────────
Substack articles                   ──┐
                                      ├──> HTML-to-Markdown ──> Frontmatter + Tagging ──> Markdown files
Hacker News and other sources       ──┘

Markdown files ──> Chunker ──> Embedder (e5-large-v2) ──> PostgreSQL + pgvector

Search query ──> Query embedding ──> Cosine similarity search ──> Ranked results

Three stages, each independent and replaceable. You scrape content from your sources. You process and embed it. You search it.

If your team writes in Confluence instead of Substack, you swap the scraper. If you prefer Qdrant over pgvector, you swap the vector store. The architecture remains the same.

And here’s the hardware used for most of the steps, from embedding to the storage and retrieval: my DGX Spark.

Yes, I know, probably an overkill.

Check the TWSC YouTube Channel

The tools

Playwright handles browser-based scraping for our own Substack articles. Substack serves content dynamically and requires authentication for premium posts, so a plain HTTP client is not an option.

Algolia API (via Hacker News) provides structured search over HN stories. No scraping needed: HN exposes its full search index through public endpoints.

ScrapegraphAI and Firecrawl convert external article URLs into clean markdown. ScrapegraphAI is the primary extractor, Firecrawl is the fallback.

sentence-transformers with the intfloat/e5-large-v2 model generates 1024-dimensional embeddings. We will explain why we chose this model later in the article.

PostgreSQL with pgvector stores embeddings and handles similarity search. We chose it over dedicated vector databases because we already need PostgreSQL for metadata, and pgvector with HNSW indexing handles our scale without adding infrastructure.

Docker Compose ties everything together as three containers: PostgreSQL, the API server, and the indexer.

You can find the code in our GitHub repository reserved to paying users, inside the folder 101.KNOWLEDGE_BASE.

Need help with your scraping project?

Why markdown as the universal format

The first design choice we had to make was what format our knowledge base would store. We had content from Substack (HTML), Hacker News links (various formats), and potentially Confluence, Google Docs, or Slack in the future. We needed a common representation.

We chose markdown for three reasons.

First, markdown preserves document structure without carrying rendering noise. An HTML page contains navigation bars, ad slots, JavaScript, CSS classes, and layout dividers. None of that is content. When you convert to markdown, you keep headings, paragraphs, code blocks, links, and lists. Everything the embedding model needs, nothing it would choke on.

Second, markdown is readable by humans and machines alike. When something goes wrong in the pipeline, you can open a markdown file and immediately see what the system is working with. Try doing that with a serialized HTML DOM or a JSON blob from an API response.

Third, YAML frontmatter is a natural fit for markdown and gives us a structured metadata header without mixing it into the content. Each file gets an `id`, `type`, `title`, `publish_date`, `topics`, and `visibility` field. This metadata drives filtering at search time and never enters the embedding model. The separation is important: embeddings capture meaning, frontmatter captures facts.

There are two paths to get content into markdown. You can build your own converter using open-source libraries, or you can use commercial services that handle extraction and conversion for you. In this article we show both approaches deliberately. For our own Substack articles, we built a converter from scratch with BeautifulSoup and markdownify. It costs nothing, we control every detail, and it works because we know the source HTML structure intimately. For external content discovered on Hacker News, we use commercial services like ScrapegraphAI and Firecrawl instead, because every URL leads to a different site with a different HTML structure. Building custom converters for thousands of unknown domains would be impractical. The trade-off is clear: when you control the source, build your own; when you are scraping the open web, commercial extraction services save an enormous amount of development time.

Our Substack HTML-to-markdown converter is deliberately simple. It strips scripts, styles, buttons, navigation, and footers, then converts the remaining HTML:

def html_to_markdown(html: str) -> str:
    soup = BeautifulSoup(html, "lxml")
    for tag in soup.find_all(["script", "style", "button", "form", "nav", "footer"]):
        tag.decompose()

    md = markdownify(
        str(soup),
        heading_style="ATX",
        bullets="-",
        strip=["script", "style", "button", "form", "nav"],
    )
    md = re.sub(r"\n{4,}", "\n\n\n", md)
    return md.strip()

The final output for each document looks like this:

---
id: a1b2c3d4e5f6...
type: twsc_article
title: "THE LAB #94: Using Cookies and Session Persistence"
slug: the-lab-94-using-cookies-and-session
canonical_url: https://substack.thewebscraping.club/p/the-lab-94-using-cookies-and-session
publish_date: 2025-11-15
visibility: premium
topics:
  - browser-automation
  - cloudflare
  - scraping-infra
---

[article body in markdown]

Scraping your own content

The first source we built was a scraper for our own Substack articles. The pattern applies to any CMS: discover URLs, authenticate if needed, extract content, convert to markdown with frontmatter.

URL discovery and authentication

Most publishing platforms expose a sitemap. We fetch it, filter for article URLs (Substack uses /p/ in the path), and track the lastmod date to detect changes:

def fetch_sitemap(sitemap_url: str) -> list[dict]:
    req = Request(sitemap_url)
    req.add_header("User-Agent", "Mozilla/5.0 ...")
    with urlopen(req) as response:
        content = response.read()

    root = ET.fromstring(content)
    ns = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}

    articles = []
    for url_elem in root.findall("sm:url", ns):
        loc = url_elem.find("sm:loc", ns)
        lastmod = url_elem.find("sm:lastmod", ns)
        if loc is not None and "/p/" in loc.text:
            articles.append({"url": loc.text.strip(), "lastmod": lastmod.text or ""})
    return articles

Substack gates premium content behind authentication. We handle this with a persistent Playwright browser context that stores cookies across runs. On the first run you log in manually; after that, the saved session keeps you authenticated. For cron jobs, we verify the session by loading a known premium article and checking if the full content appears.

We try multiple CSS selectors for extraction because Substack has changed its DOM structure over time. The extracted HTML goes through the markdown converter we showed earlier.

Ingesting external sources: Hacker News

Read more

The traditional way to answer that question involved surveys, focus groups, and expensive reports from firms that charge you a fortune for data that’s already a few months old by the time you read it. Today, the data you need is sitting on public web pages: You just need to collect it.

In this article, we’ll discuss how to scrape data for market research, what sources actually matter, how to build a pipeline that doesn’t fall apart after a week, and where the legal lines are.

Let’s dive into it!

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

What “Market Research” Actually Means Web Scraping Professionals

Market research needs to answer three questions:

• “What are our competitors doing?”

• “What are our customers saying?”

• “How is the market moving?”

That’s it. Everything else is a variation of those three. And if you think about it, the web gives you access to all three, if you know where to look.

In practice, scraped market intelligence sits on three pillars:

• Competitive data: Pricing, product catalogs, feature changes, hiring signals. This is the “what are they doing?” pillar.

• Customer sentiment: Reviews, forum discussions, social media posts. This is the “what are people saying?” pillar.

• Market signals: Job postings, regulatory filings, trend volumes, new product launches. This is the “where is the market going?” pillar.

Now, why scraping instead of traditional research? Because scraping is real-time, it’s continuous, and it doesn’t depend on people filling out forms. A survey tells you what 500 people said last month. A scraper tells you what thousands of customers are saying right now, every single day, without anyone having to opt in.

That’s the competitive advantage. And it’s a big one.

For your scraping activity, you need IPs with good reputation. For this reason, we’re using a proxy provider like our partner Ping Proxies, that’s sharing with TWSC readers this offer.

💰 - Use TWSC for 15% OFF | $1.75/GB Residential Bandwidth | ISP Proxies in 15+ Countries

Where to Scrape: Sources That Actually Matter

Not all sources are worth your time. You could scrape the entire Internet and still end up with nothing useful if you’re not targeting the right places. Below is a list of high-value targets for market research and what you can extract from each:

• Competitor websites: Pricing pages, product pages, feature matrices, changelog, and blog posts. This is your primary source for understanding what competitors are offering and how they position themselves. Pricing pages, in particular, are gold. They change more often than you’d think, and tracking those changes over time tells you a lot about a competitor’s strategy.

• Review platforms (G2, Trustpilot, Amazon, Yelp): Customer pain points, feature requests, sentiment shifts. Reviews are unfiltered customer feedback. Nobody writes a G2 review because they were asked nicely in a survey. They write it because they feel strongly about something—and that’s exactly the kind of signal you want.

• Job boards (LinkedIn, Indeed): Hiring patterns reveal where a company is investing. If a competitor suddenly posts 20 machine learning engineer roles, that tells you something no press release will. Job postings are one of the most underrated market research signals out there.

• Social media and forums (Reddit, X, niche communities): Unfiltered opinions, emerging trends, early complaints about products. Reddit threads and niche forums are where people say what they actually think, not what they’d say in a focus group.

• Government and public data portals: SEC filings, patent databases, import/export records. These are slower-moving signals, but they’re authoritative. A patent filing can tell you what a competitor is building 18 months before it ships.

Here’s the key question to ask yourself before adding a source to your scraper: “Does this data answer a specific research question, or am I just hoarding?”. If you can’t tie a source to a concrete insight, skip it. You’ll save yourself storage costs, maintenance headaches, and potential legal issues.

Building the Pipeline: From Raw HTML to Market Intelligence

A market research scraper is not a one-off script you run from your terminal. It’s a pipeline. And pipelines need structure. If you treat it like a quick script, you’ll end up with a mess of cron jobs, inconsistent data formats, and no idea whether your data is fresh or stale. So, build it properly from the start.

A scraping for market intelligence pipeline should have four stages:

1. Collection: Fetch the pages, extract the fields you need, throw the rest away. Don’t store raw HTML “just in case” (you’ll learn why in the legal section of this article).

2. Storage: Store facts and metadata (source URL, timestamp, extracted fields). Use a structure that makes deduplication and versioning easy. In practice, this means designing your schema around a composite key (for example: source + entity ID + scraped timestamp) so you can track how a data point changes over time without overwriting previous records.

3. Transformation: Normalize the data across sources, deduplicate records, and enrich with additional context (geocoding, industry classification, entity linking).

4. Analysis: Turn rows into insights. This is where the actual market research happens. And to be clear: “Analysis” doesn’t mean opening a CSV and scrolling through it. The goal is to turn your pipeline’s output into dashboards, scheduled reports, or Slack alerts that reach the people who make decisions. If the data sits in a database and nobody looks at it, the whole pipeline is wasted effort.

Scheduling Matters More Than You Think

Different data types have different freshness requirements. Getting this wrong means either wasting resources or working with stale data. The main ideas to consider when engineering the triggering times are the following:

• Price tracking: Daily or hourly, depending on the market. Consider that e-commerce prices can change multiple times a day. SaaS pricing pages, instead, change less often. But when they do, it’s significant.

• Review monitoring: Monitoring reviews daily is usually enough. Reviews don’t appear in real-time, and sentiment trends are measured in weeks, not minutes.

• Job postings: A weekly schedule works for trend analysis of the job market. Remember that you’re looking for patterns, not individual listings.

• Social media: This depends on your use case. If you’re tracking a product launch or a PR crisis, you might need near-real-time. For general trend analysis, daily or even weekly batches work fine.

Tools That Work Well for Market Research Scraping

You don’t need to reinvent the wheel. The software industry already provides you with the best tools for your market research scraping pipeline. Here’s a solid stack for a market research pipeline:

• Scrapy for structured crawling. Scrapy’s architecture is designed for exactly this kind of work: You define spiders per source, plug in middleware for proxy rotation and retry logic, and use item pipelines to clean and store data as it flows through. For market research specifically, Scrapy’s built-in feed exports let you dump results straight to JSON, CSV, or even S3 without writing custom I/O code. And if you need to coordinate multiple spiders (say, one per competitor), Scrapy’s project structure keeps things organized as your source list grows.

• Playwright or Puppeteer for JS-heavy pages. The key difference from Scrapy is that you’re running a real browser, which means you can handle dynamic content, infinite scroll, and client-side rendering. The trade-off is resource cost: Each browser instance eats memory and CPU, so you don’t want to use this for targets that serve static HTML.

• A task queue for scheduling and orchestration. This is what turns a collection of scrapers into an actual pipeline. Instead of running scripts manually or relying on cron jobs, a task queue lets you schedule scrapes per source at different intervals, retry failed jobs automatically, and control concurrency so you’re not overwhelming a target site with parallel requests. It also gives you visibility: you can see what’s queued, what’s running, what failed, and why.

• PostgreSQL for structured market data that needs querying and versioning. Relational databases shine here because market research data is inherently relational: competitors have products, products have prices, prices change over time.

The point is this: Pick tools that let you build a maintainable system, not just a working script. Every tool in this stack solves a specific problem, and none of them requires you to build infrastructure from scratch. The best market research pipeline is the one that’s boring to operate, because boring means reliable.

Check the TWSC YouTube Channel

Scaling Without Getting Blocked

If you’re scraping one competitor once a week, you don’t need this section. If you’re tracking 50 competitors daily across thousands of pages, you do.

Here’s the reality: The moment you start scraping at scale, you become visible. But sites don’t like bots, even polite ones. So you need to be smart about how you scale. Consider the following rules of thumb to avoid getting blocked:

• Proxy rotation: Residential proxies for sensitive targets (sites with aggressive anti-bot systems), datacenter proxies for everything else. Rotate per request or per session, depending on the site’s detection mechanisms. The key is to not send thousands of requests from the same IP in an hour.

• Rate limiting and backoff: Be a good citizen. If you hammer a site with concurrent requests, you’ll get blocked, and you’ll deserve it. Implement exponential backoff on failures, and set reasonable delays between requests. A 2-3 second delay between requests is a good starting point for most sites.

• Fingerprint management: Headers, TLS fingerprint, and browser-level signals matter on sites with serious anti-bot systems. Make sure your request headers look consistent and realistic.

• CAPTCHAs: If you’re hitting CAPTCHAs regularly, your approach is too aggressive. Fix the root cause (rate, fingerprint, proxy quality) before reaching for solver services. CAPTCHA solvers are a band-aid, not a solution.

The general principle is simple: Scrape at a pace that doesn’t degrade the target site’s performance.

Need help with your scraping project?

Turning Scraped Data into Actual Market Insights

Let’s be clear about something: Raw scraped data is not market research. It’s just data. A CSV with 50’000 rows of competitor prices is not an insight. A chart showing that competitor X has dropped their enterprise tier price by 15% over three months: That’s an insight.

Here’s where the value gets created:

• Price tracking and competitive benchmarking: Track changes over time, visualize trends, and set alerts for significant moves. The goal is not to know what a competitor charges today. It’s to understand their pricing trajectory. Are they moving upmarket? Are they running more frequent discounts? Are they simplifying their tier structure? This is where predictive analytics meets scraped data with the goal of predicting future moves from your competitors.

• Sentiment analysis on reviews: Use NLP to extract themes from customer reviews. This is powerful for product teams who want to understand what customers love and hate about competitors. But remember: You’re analyzing the data internally, not republishing the reviews.

• Hiring signal analysis: Aggregate job postings by role type, department, and location. A competitor suddenly posting 15 ML engineer roles tells you they’re investing in AI. A wave of sales hiring in EMEA tells you they’re expanding geographically. This is a signal that’s almost impossible to get from any other source.

• Trend detection: Time-series analysis on product launches, feature changes, pricing moves, or social media mentions. The goal is to spot patterns or anomalies before they become obvious. If three competitors all add the same feature within two months, that’s a market trend, not a coincidence.

Overall, the output of your scraping pipeline should be dashboards, reports, or automated alerts, not a database dump that someone has to manually dig through. If the insights don’t reach decision-makers in a usable format, the whole pipeline is wasted effort.

Legal and Ethical Considerations: Don’t Skip This Section

I know, I know. You’re a developer, not a lawyer. But here’s a thing I’m sure you know: Most legal problems in scraping are self-inflicted. They happen because someone scraped “everything on the page,” stored it “for later,” and only then asked: “Wait, can we actually use this?”

As discussed in detail in “How to Avoid Copyright Violations While Scraping”, let’s go through the key legal and ethical principles of web scraping shortly:

• Scrape facts, not expression: Copyright protects expression, not facts. Prices, SKUs, dates, availability, and job titles are facts. No one owns the fact that a SaaS product costs $49/month. On the other hand, product descriptions, review text, and blog posts are creative expressions.

• Don’t store raw pages by default: Storing the HTML of entire pages means creating copies of copyrighted content. Instead, parse in-memory, extract only the fields you need, and discard the rest. If you need to debug, store a small sample with short retention.

• Respect robots.txt: The robots.txt file is not the law, but ignoring it is evidence of bad faith if things go sideways. In disputes, it can be used to show that you knew you were unwelcome and kept going anyway.

• Terms of Service matter: If the ToS explicitly forbids scraping and you scrape anyway, you may have a breach-of-contract problem. This is often easier for the site owner to prove than copyright infringement, because the argument is straightforward: you agreed to a contract, then you violated it.

• Don’t scrape behind a login: Once you log in, you’ve affirmatively agreed to a contract. Breaking that contract to scrape is a fast track to legal trouble. If your plan requires authenticated access, treat it as a licensing problem, not an engineering challenge.

• GDPR/CCPA: If you’re scraping anything that could be personal data (usernames, reviewer names, profile information), you need to know which privacy laws apply. This is especially relevant for review scraping and social media monitoring.

Here’s the mental model that works: A price comparison tool that shows prices and links back to the source? Generally safe. A product catalog that copies descriptions, images, and reviews so users never need to visit the original site? That’s where you get into trouble, even if you don’t publicly display the results because you use them for internal analysis.

Keeping Your Scrapers Alive: Monitoring and Maintenance

Scrapers in production break for several reasons. Sites change layouts, add anti-bot measures, restructure their URLs, or just go down for maintenance. If you don’t monitor your scrapers, your data goes stale silently, and you won’t know until someone asks why the pricing dashboard hasn’t updated in three weeks.

Here’s a breakdown of what you need:

• Dead selector detection: Alert when a CSS selector or XPath returns empty across multiple consecutive runs. A selector that worked yesterday and returns nothing today means the site changed its HTML structure. The keyword here is “multiple consecutive runs”. A single empty result could be a transient issue, so consider not triggering alerts on the first failure. Instead, set a threshold, like three consecutive empty results, before flagging it. When it does fire, you need to inspect the current page structure and update your selectors. Alternatively, try to go beyond the DOM using AI and LLMs, to make your extraction more resilient to layout changes in the first place.

• HTTP status monitoring: A spike in 403s means you’re getting blocked. A spike in 429s means you’re hitting rate limits. A spike in 404s means URLs have changed. Each of these requires a different response. For 403s, check your proxy pool and rotation logic: You might need fresher IPs or a lower request rate. For 429s, back off and increase your delays between requests; the site is telling you exactly what the problem is. For 404s, the target has likely restructured its URL patterns, which means you need to update your URL generation logic, not just retry the same broken links. Log these status codes per source and per run so you can spot trends early. A gradual increase in 403s over a week is a warning sign that your current setup is losing effectiveness, even if individual runs still return some data.

• Data quality checks: Row counts, null rates, value distributions. If your price tracker suddenly shows all prices as $0 or your review scraper returns empty text fields, you want to know immediately. Build quality checks into your pipeline as a post-scrape validation step, not as something you run manually. Compare each run’s output against baseline expectations: If you normally get 200 rows from a source and today you got 12, something is wrong, even if those 12 rows look fine individually.

• Automated tests against fixture HTML: Save sample HTML pages from your targets and write tests against them. When a test fails, you know the site has changed before your production scraper breaks. Treat your scrapers like production code, because they are. In practice, this means saving a snapshot of a relevant section in the target page as a local HTML file. Then, write unit tests that run your extraction logic against that fixture and assert expected outputs. Store these fixtures in version control alongside your scraper code. When a site changes and your production scraper breaks, update the fixture with the new HTML. This gives you a repeatable workflow for handling site changes instead of scrambling every time something breaks.

The goal is simple: You should know when something breaks before your stakeholders do. A Slack alert that says “Competitor X pricing scraper returned 0 results” is infinitely better than a product manager asking why the dashboard is empty.

Conclusion

In this article, you learned that market research scraping is about building a reliable pipeline that collects the right facts, transforms them into insights, and doesn’t get you in legal trouble.

The competitive advantage of scraping for market research is in what you do with the data. Anyone can code a scraper. But building a system that delivers reliable, actionable market intelligence week after week? That’s where the real value is!

So, let us know: Are you using web scraping for market research? What sources have you found most valuable? How did you structure your scraping pipeline? Let’s discuss in the comments!

The Proxy Price Benchmark is now updated weekly

If you haven't checked it yet, the Proxy Price Benchmark is the tool I built to answer a simple but important question: how much should you actually be paying for your proxies?

Every week, I (or better, my fleet of agents) update the pricing data directly from the vendors, so you always have a reliable reference to compare offers or negotiate with your current provider.

This week, we added two new vendors: Dataimpulse and AnyIP, bringing the total number of monitored providers to 27.

Check the latest prices

If you use proxies at scale and would find API access to this data useful, I am considering a paid API plan. If you are interested, join the waitlist and tell me about your use case. I want to understand demand before I build it.

This week on Scraping News: stealth browsers are getting serious

The Scraping News feed has been tracking an interesting trend this week: two new stealth browser projects worth watching.

Owl Browser is a purpose-built browser engine for automation at scale. Not a Playwright wrapper but a full engine built on Chromium (CEF) with a custom C99 HTTP server, 256 parallel contexts, and sub-12ms cold start. Self-hosted, Docker-ready, with Python and TypeScript SDKs. If you are running high-volume scraping and hitting the limits of standard headless setups, this is worth a closer look.

Rayobrowse is Rayobyte's open-source stealth Chromium browser, released from their production scraping infrastructure. It handles fingerprint randomization at the browser level (user agent, WebGL, fonts, screen resolution, timezone) and connects via CDP, so it works with Playwright, Puppeteer, Selenium, or any custom script. Runs on headless Linux with no GPU required.

Both address the same problem from different angles: standard headless Chromium is detected, and the solution is now moving from patch-level evasion to full browser-level stealth. We will be covering both in depth on TWSC soon.

See all the latest news on Scraping News

Keep in mind that both the Proxy Price Benchmark tools and Scraping News are in an early version; feel free to suggest improvements and bug fixes.

You might think that, in a web scraping scenario, the solution is simply to connect directly to the WebSocket channels. Sure, that’s possible, but there are a few obstacles along the way. The main ones are WebSocket anti-bot techniques and bot detection measures.

In this post, I’ll walk through the most common ones, explain how they work, and share proven tips and tricks to help you avoid them.

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

A Quick Intro to WebSockets

Before diving into WebSocket bot detection, let me first provide some context about WebSocket as a protocol and its role in web scraping.

What Is the WebSocket Protocol?

WebSocket, also abbreviated as WS for short, is a web protocol standardized in RFC 6455 that enables full-duplex, bidirectional communication between clients and servers over a single, persistent TCP connection.

Unlike HTTP, which is stateless and request-driven, WebSockets establish a long-lived connection through an initial HTTP handshake. After the handshake, both client and server can send messages independently, with data transmitted in frames that can be text, binary, or control frames (ping, pong, close).

WebSockets support fragmentation, masking, and optional compression via extensions like per-message-deflate, while newer HTTP/2 and HTTP/3 mechanisms allow multiplexing, reduced latency, and better proxy traversal.

For your ethical scraping activity, you need IPs with good reputation. For this reason, we’re using a proxy provider like our partner Ping Proxies, that’s sharing with TWSC readers this offer.

💰 - Use TWSC for 15% OFF | $1.75/GB Residential Bandwidth | ISP Proxies in 15+ Countries

Why and When Web Pages Use WebSockets

The WebSocket protocol opens the door to live, bidirectional web communication. Unlike HTTP’s request-response model, it lets servers and clients exchange data continuously over a single, persistent connection.

In general, WebSockets are essential for any application where low latency and frequent updates are required. Common use cases include:

• Live streaming: YouTube Live, TikTok LIVE, Kick, Twitch, and similar platforms.

• Chat applications: Slack, Discord, and other messaging services.

• Collaboration tools: Google Docs, Figma, and online whiteboards.

• Gaming and multiplayer experiences: Browser-based MMO games, turn-based games, and PvP games.

• Financial data feeds: Stock tickers, cryptocurrency price updates, and trading dashboards.

• IoT and telemetry: Sensor updates, home automation, and device monitoring.

• Notifications and alerts: Push updates for social networks, dashboards, or monitoring systems.

In short, WebSocket comes into play wherever instant, continuous communication is necessary (and standard HTTP polling would be too slow or resource-intensive).

Main Challenges of Scraping Data from WebSockets

Connecting to a WebSocket server for collecting data isn’t as straightforward as spoofing API requests for web scraping. In particular, the main challenges of scraping data straight from WebSockets include:

• Finding the right client implementation: You must use a WebSocket client (and there are way fewer than HTTP clients…) that supports the correct protocol version and any negotiated extensions, such as compression or subprotocols.

• Limited documentation and examples: WebSocket scraping is less common than API scraping, so there are fewer guides, tools, and community resources available.

• Proxy integration complexity: Not all clients support proxy integrations, making IP rotation a challenge.

• No request–response model: You can’t simply send a request and receive a response, as with API scraping. Instead, you must send the right messages and then listen to a continuous stream of events.

• Real-time data handling: You require a system to collect, process, and store messages in real time, often dealing with high-frequency updates.

Main WebSocket Anti-Bot Techniques and Solutions

Now you’re ready to discover the most important WebSocket-specific bot detection techniques, along with practical tips to avoid and bypass them. The idea here is to target a WebSocket server from an automated script, relying on a WS client in Python, Node.js, or another programming language of your choice.

WebSocket Handshake Issues

The WebSocket handshake is a transition phase in which an HTTP connection is upgraded to a persistent WebSocket connection. During this step, both the client and the server negotiate the connection parameters, and either side can abort the process if the conditions aren’t acceptable.

Because the handshake is where the protocol upgrade happens, it’s also a pivotal security and bot-detection point. The server must carefully validate everything the client requests. Otherwise, protocol misuse or security issues may occur.

In detail, during the handshake, a WebSocket client must send a valid HTTP/1.1 GET request with specific headers, for example:

GET /live-data HTTP/1.1
Host: example.com:9000
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: JKjeFfYU8mti9re0prPQrw==
Sec-WebSocket-Protocol: chat, superchat
Sec-WebSocket-Version: 13

In practice, browsers also include additional headers such as Origin, User-Agent, Referer, Cookie, as well as authentication headers (e.g., Authorization). While these HTTP headers aren’t strictly required by the WebSocket specification, they are extremely valuable for fingerprinting and bot detection.

Now, the server should respond with 400 Bad Request and immediately close the connection if it encounters:

• An unknown or malformed header.

• An invalid Sec-WebSocket-Key.

• An unsupported WebSocket version.

Instead, if the WebSocket version is unsupported, the server should return a Sec-WebSocket-Version header listing the versions it supports (most modern servers only accept version 13).

In practice, repeated handshake failures or non-browser-like handshake patterns are often treated as a bot indicator. Those may result in blocking, particularly after repeated handshake attempts from the same IP or when fingerprinting enables identification even across IP changes.

📌 Tips:

• Always send a valid Origin header: All major browsers include it, and many servers automatically reject WebSocket requests without one.

• Replicate real browser handshakes as closely as possible: Inspect the WebSocket request made by a real browser and match all headers (e.g., User-Agent and similar extra headers).

• Avoid excessive handshake attempts from the same machine: Too many connection attempts in a short time window are a common bot signal.

• Use IP rotation carefully: Rotation can help avoid rate-based blocks, but it doesn’t protect against fingerprint-based detection if the handshake remains identical.

Honeypot WebSocket Events and Channels

If you’re familiar with common anti-bot techniques, you’ve probably heard of honeypots. A honeypot is a decoy mechanism designed to attract bots by exposing fake or hidden resources, allowing systems to detect automated behavior when those resources are accessed or interacted with (e.g., invisible links or fake pages created to study bots).

In the context of WebSockets, honeypot events are a possible anti-bot technique to detect automated clients. With this approach, the server deliberately sends fake, misleading, or non-actionable events over the WebSocket connection. Similarly, the server might expose channels that aren’t meant to be accessed by regular clients.

Yet, automated scraping bots may react incorrectly to WebSocket honeypots by:

• Processing incoming data that is fake or intentionally invalid.

• Requesting access to or subscribing to channels they aren’t supposed to use.

📌 Tips:

• Study real browser behavior carefully: Inspect WebSocket traffic in your browser’s DevTools (“Network” → “Socket”) and observe which server messages actually trigger data flow or UI updates.

• Avoid assuming every message is meaningful: Remember that reacting to every event can lead to detection.

Connection Lifecycle Anomalies and Patterns

Since WebSocket channels are stateful (unlike stateless HTTP requests), servers can detect bots by analyzing connection behavior over time. Scraping bots tend to prioritize speed over realistic user behavior, which can produce identifiable patterns.

In this regard, popular bot-like indicators include:

• Very short-lived connections: Opening and closing sockets rapidly to collect data.

• Immediate reconnections after closure: Reconnecting instantly without human-like delays.

• High connection churn per IP: Multiple connections from the same IP within a short period.

• Missing browser events: Typical browser WebSocket clients trigger events like proper socket closure, whereas bots often skip them.

• Unnatural latency patterns: Servers use ping frames as heartbeats to check responsiveness. Real users on home Wi-Fi or mobile networks exhibit variable latency (jitter), while automated scripts deployed on data centers generally show extremely stable, low-latency responses.

📌 Tips:

• Introduce some randomness: Introduce realistic delays between connections and reconnections.

• Replicate intended behavior: Emulate browser close events if testing automated clients.

• Add latency variation: Consider latency variation when sending and receiving frames to mimic real-world network jitter.

• Rotate connection IPs: Use proxies to distribute WebSocket connections across multiple IPs.

WebSocket Binary Data Transmission

WebSocket servers sometimes choose to send binary data instead of plain text or JSON. The main technical reasons for this are:

• Reduced bandwidth: Binary messages omit field names and whitespace, making packets smaller than JSON strings and supporting high-frequency updates.

• Faster parsing: Binary data can be read as typed arrays or fixed-size fields, avoiding JSON parsing overhead.

• Custom protocols: Web apps can define their own compact binary format for predictable, high-frequency data.

• Efficient number storage: Numeric values can be stored in 1–4 bytes rather than as multi-character strings, saving space.

For instance, TikTok LIVE pages use WebSockets to stream updates (e.g., chat messages, view counters, and other statistics) in binary format:

Sure, binary data can be converted to text. So, you may think that’s not a problem…

Well, keep in mind that most web applications using binary data implementations include some form of compression or encryption. This adds significant complexity!

Reverse-engineering these systems is technically possible by inspecting browser WebSocket clients, analyzing request headers for compression hints, or trial-and-error with common compression methods. Still, that’s time-consuming and error-prone. Plus, encryption keys, salts, or other details can easily change with each deployment.

📌 Tips:

This time, the only piece of advice I have is to look for alternative data sources. Many WebSocket-based pages, including TikTok LIVE, use regular HTTP APIs to retrieve initial data.

Note: Why aren’t these APIs called server-side when the HTML page is generated? In the case of live data, it’s more reliable to fetch it on the client, because even a single second of latency could result in outdated or inconsistent information.

Thus, polling over those RESTful APIs instead of the WebSocket data streams can allow you to retrieve the information of interest without dealing with binary encoding, compression, or encryption challenges.

Check the TWSC YouTube Channel

WebSocket-Based Bot Detection Measures

The WebSocket protocol is built on top of HTTP, so they inherit many anti-bot techniques commonly used for HTTP requests. At the same time, due to its stateful and persistent nature, anti-bot solutions like WAF (Web Application Firewalls) can leverage WebSockets to detect automated behavior even more effectively…

As a result, WebSocket-based anti-bot measures are not only relevant when connecting directly to WS servers, but also when interacting with web pages through browser automation tools like Playwright and Selenium. That’s why you must know them!

Advanced TLS Fingerprinting

Traditional HTTP fingerprinting checks headers and TLS details. WebSockets extend this by combining the TLS handshake with WebSocket-specific framing, which is much harder to spoof. Signals include JA3/JA4 fingerprints, unusual cipher suite ordering, frame fragmentation patterns, and incorrect masking behavior.

Need help with your scraping project?

Continuous Device Fingerprinting

HTTP allows basic fingerprinting on a per-request basis, but it can’t verify whether the client’s environment remains consistent. The stateful nature of WebSockets enables servers to continuously validate device fingerprints over time. For example, servers can request Canvas/WebGL renders, available fonts, and other browser characteristics repeatedly. Any inconsistency can lead to an immediate block.

Real-Time User Behavior Monitoring

WebSockets allow live streaming of mouse, keyboard, and scrolling events back to the server. This enables a much deeper level of user behavior analysis compared to static HTTP requests.

After all, most browser automation scripts produce perfectly straight mouse movements or instantaneous clicks, while human interactions naturally include slight jitter, variable speed, and reaction delays. These differences make automated clients easier to detect when behavior is constantly monitored over a WebSocket connection.

Conclusion

Here, I introduced the WebSocket protocol and explained why and when it comes in handy. Specifically, you learned that it powers live data updates on web applications. Want to access that data? Well, it’s not as straightforward as you might think due to WebSocket anti-bot techniques.

In this post, I explored the most relevant WS bot detection methods, along with useful advice for bypassing them successfully. You also saw how WebSocket’s stateful, continuous data streaming can be used by WAFs and other advanced anti-bot systems for enhanced detection.

I hope you found this helpful and informative. If you have any questions or comments, drop them below. Until next time!

But a browser does everything the hard way. It downloads the full page, parses HTML, executes JavaScript, renders the DOM, loads images, fonts, and stylesheets. For each request, it allocates hundreds of megabytes of RAM and takes seconds to complete what an HTTP client could do in milliseconds. When a pipeline needs to scrape ten pages, this overhead is irrelevant. When it needs to scrape ten thousand pages, the browser becomes the bottleneck.

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

Consider a concrete scenario: we need to monitor the wishlist of an e-commerce account, pulling product data, stock levels, and price changes every hour across hundreds of items. Running Camoufox for every single API call would mean spinning up a full browser instance, navigating to each page, waiting for JavaScript to execute, extracting the data, and closing. For a hundred items, that is minutes of execution time and gigabytes of memory. The same API calls through an HTTP client would complete in seconds using a fraction of the resources.

As we measured in THE LAB #96, HTTP clients with TLS impersonation can be 27x faster than browsers on the same target. The difference is not marginal. It is the difference between a pipeline that runs on a single machine and one that requires a cluster.

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

The problem is that these two approaches are usually treated as mutually exclusive. Either you use a browser for everything, accepting the overhead, or you try an HTTP client and hope the anti-bot system does not block it. But many websites only need a browser at the gate: for the login, the initial challenge, or the session establishment. Everything after that is plain API calls.

If we can use a browser to earn a valid session and then hand it off to an HTTP client, we get the reliability of browser automation where it matters and the speed of HTTP everywhere else. That is the pattern we want to build. But the handoff is not as simple as copying a few cookies, and the traps along the way are worth understanding before building a pipeline around this idea.

The hybrid pattern

The idea is simple in principle. Many websites require a browser only at the gate: the login flow, the initial anti-bot challenge, or the session establishment. Once that gate is passed, subsequent requests are plain API calls or page fetches that do not require JavaScript execution. If we can extract the session state from the browser and replay it through an HTTP client, we skip the browser for 99% of the work.

The session state, in practice, means cookies. An authentication flow sets session cookies that the server trusts for subsequent requests. If we transfer those cookies from the browser to an HTTP client, the server should treat the HTTP client as the same authenticated user.

But cookies alone are often not enough. Modern anti-bot systems like Akamai do not just check whether you have the right cookies. They also check whether the client presenting those cookies looks like the same client that earned them.

This is where TLS fingerprinting enters the picture: if the browser that logged in was Firefox, but the HTTP client that reuses the cookies presents a Python TLS fingerprint, the server may reject the request or simply drop the connection without responding.

So the real challenge is not just transferring cookies. It is maintaining continuity across two different execution models: the browser and the HTTP client must look like the same entity to the server.

Check the TWSC YouTube Channel

Tool landscape

For this experiment, we used two tools.

Camoufox is a custom Firefox build designed for stealth. It spoofs fingerprints (WebGL, canvas, audio, navigator properties), patches headless detection vectors, and uses Playwright’s Juggler protocol for automation. We covered it extensively in THE LAB #65: Scraping Datadome-protected websites with Camoufox. Its role here is limited to one thing: logging in.

curl_cffi is a Python binding for curl-impersonate, a modified version of curl that mimics the TLS and HTTP/2 fingerprint of real browsers. It supports impersonating Chrome and Firefox at specific versions, which means it can present the same TLS fingerprint as the browser that established the session. Unlike a browser, it uses negligible resources per request and can process thousands of pages per minute.

The key property that makes this pairing work: Camoufox is Firefox-based, and curl_cffi can impersonate Firefox’s TLS fingerprint. The server sees a consistent Firefox identity across both steps.

You can find the code in our GitHub repository reserved to paying users, inside the folder 100.HYBRID_SCRAPING.

Need help with your scraping project?

The target: Net-a-Porter

We chose Net-a-Porter as our target. It is a luxury e-commerce platform protected by Akamai Bot Manager, with authenticated features (wishlists, account details) exposed through internal JSON APIs. This gives us a clean test case: the login requires a real browser (Akamai blocks automation tools at the login endpoint), but the authenticated API calls are plain HTTP requests that return structured JSON.

Please keep in mind that this is an experiment for study purposes, and we’re not inciting you to scrape Net-a-Porter or any other website, especially the part behind a login.

Before diving into code, we need to understand what we’re dealing with. Net-a-Porter’s architecture has three layers relevant to us:

Akamai Bot Manager sits in front of everything. It sets a cluster of tracking cookies (_abck, bm_sz, bm_s, ak_bmsc, and others) that are generated through JavaScript execution on the client side. These cookies prove that a real browser visited the page. Without them, API calls either fail or hang indefinitely.

The login API at /api/nap/wcs/resources/store/nap_il/loginidentity/v2 accepts a JSON payload with email and password. On success, it returns a 201 status with an Ubertoken in the response body. This token is the key to all authenticated endpoints.

Authenticated API endpoints like the wishlist API at /api/nap/wcs/resources/store/nap_il/wishlist/v2/{id} require both the session cookies and the Ubertoken passed as an x-ubertoken header. They return clean JSON with product details, stock levels, and metadata.

The experiment: what worked and what did not

We did not arrive at the final solution directly. The investigation path itself reveals the constraints of session handoff, so it is worth walking through each attempt.

Read more

Lots of tools have emerged to solve the issue of browser fingerprinting, which, for example, is one of the primary reasons for blocks even when using high-quality residential proxies. So, the need companies have for stable, scalable data collection makes anti-detect solutions essential for survival in the current status of the industry.

In this article, you’ll discover Dolphin{anty}: A powerful anti-detect browser that lets you orchestrate hundreds of unique, isolated browser profiles. You’ll learn its strengths, why you should consider it for your scraping or multi-accounting projects, and how it works with a practical guide.

Let’s dive into it!

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

What is an Anti-detect Browser?

An antidetect browser is a specialized web browsing tool designed to mask a user’s digital fingerprint, allowing them to appear as a distinct, unique visitor to websites and tracking systems. Standard browsers like Chrome or Firefox broadcast a user’s hardware and software data. An anti-detect browser, instead, enables users to customize and spoof these parameters for every session.

In the context of web scraping, web scraping professionals use this technology to bypass anti-bot measures that rely on browser fingerprinting to identify and block automated traffic. Anti-detect browsers can also be used in “multi-accounting” strategies. You can use them to create isolated browser profiles, each with its own unique fingerprint, cookies, and proxy IP. The common use case is that a single user can manage hundreds of social media, e-commerce, or ad accounts simultaneously without triggering security flags that would normally link the accounts together and lead to mass bans.

A successful data pipeline is made not only by the right tool to use, but also from the right IP address. Proxy providers like Decodo help you achieving your scraping goals.

Try Decodo Now

What is Dolphin{anty} and Why Consider it for Your Web Scraping Projects?

Dolphin{anty} is an anti-detect browser that allows you to manage hundreds of unique, isolated browser profiles for web scraping and multi-accounting. You can use it via its desktop application or programmatically, as it provides a flexible API for deep integration with your scripts.

The best part of using it is that you can orchestrate wide scraping operations without worrying about browser fingerprinting. Forget about immediate IP bans, CAPTCHAs triggered by suspicious metadata, or complex cookie management. Dolphin{anty} handles the masking of your digital identity for you very simply. Also, thanks to its built-in “Scenarios” builder and synchronizer, it can automatically replicate human-like actions across multiple profiles simultaneously. So, say goodbye to manual warm-up routines and the fear of losing accounts to anti-fraud systems.

The top reasons why you should consider it for your projects are the following:

• Advanced anti-detect capabilities: If you’ve been scraping for a while, you know that standard headless browsers often leak metadata that triggers anti-bot defenses. Dolphin{anty} solves this by providing real, unique digital fingerprints for every profile. It mimics user behaviors at a granular level, allowing you to bypass sophisticated detection systems without the constant headache of being blocked.

• Mass profile management: Managing a few accounts is easy, but scaling to hundreds or thousands is a different beast. Dolphin{anty} is built for scale. It allows you to orchestrate hundreds of isolated browser profiles from a single interface. Whether you are managing a massive farm of accounts for data collection or need to segment your scraping tasks, the tool provides the infrastructure to keep everything organized and efficient.

• Flexible API integration: For those who prefer code, Dolphin{anty} offers a robust API that integrates deeply with your existing Python or Node.js pipelines. This allows you to automate profile creation, launch browsers programmatically, and integrate the anti-detect capabilities directly into your custom scraping infrastructure.

For your ethical scraping activity, you need IPs with good reputation. For this reason, we’re using a proxy provider like our partner Ping Proxies, that’s sharing with TWSC readers this offer.

💰 - Use TWSC for 15% OFF | $1.75/GB Residential Bandwidth | ISP Proxies in 15+ Countries

Dolphin{anty}’s Main Features

Dolphin{anty} is packed with features designed to make multi-accounting and scraping easier. The main features you should know about it are the following:

• Real fingerprint generation: The core of Dolphin{anty} is its ability to provide genuine device fingerprints. Instead of just blocking trackers, it creates a unique digital identity for every profile you run. In practice, it manages over 20 parameters—from WebRTC to Canvas—so your scrapers look exactly like real users on real devices.

• Built-in Automation: You don’t always need to be a coding wizard to automate tasks. Dolphin{anty} offers a “Scenarios” builder that lets you create automated workflows visually. Whether it’s warming up accounts or parsing data, you can set these scripts to run automatically. And for those who prefer code, the flexible API allows you to integrate these profiles directly into your existing scripts.

• Profile synchronizer: This is a game-changer if you need to perform the same action across multiple accounts. The Synchronizer allows you to perform an action in a “master” profile, and the tool automatically repeats that exact action across all other selected profiles in real-time. This saves you a massive amount of time on routine interactions.

• Team collaboration: If you work in a team, you know that sharing browser sessions and cookies can be a nightmare. Dolphin{anty} simplifies this by allowing you to transfer profiles, cookies, and proxies to colleagues in just a few clicks. You can also manage permissions, ensuring that team members only have access to the functionality they need.

• Smart profile management: When you are dealing with hundreds of profiles, organization is key. The tool provides a highly intuitive interface where you can use tags, statuses, and notes to sort and find your profiles instantly. It’s built to help you navigate a large farm of accounts without getting lost in the chaos.

Check the TWSC YouTube Channel

Hands-on Dolphin{anty}: Step-by-step Scraping Tutorial

In this section, you will see how easy and fast it is to use Dolphin{anty}. Get ready for the tutorial!

Setting Up Dolphin{anty}

First of all, you need to create a new login. After creating a new account on Dolphin{anty}, the system will ask you to download the software. As you can see from the image below, it supports all the major Operating Systems:

Below is how Dolphin{anty}’s interface appears after you installed it on your machine:

Good. Everything is set up. Time to create new profiles!

Need help with your scraping project?

Create New Profiles

Before using Dolphin{anty}, you have to create a new profile. To do so, click on CREATE PROFILE and fill in the fields:

Profiles are the core of Dolphin{anty}. This is where, for example, you can change the fingerprinting for your anti-detect strategies. To do so, you only need to click on NEW FINGERPRINT, and the tool will change all the fingerprinting data for you. And if the standard fingerprinting is not sufficient, you can manage advanced configurations:

Also, if your use case needs to use a specific social media like Facebook, you can set Facebook’s URL as the starting page and the credentials to log in to a profile you need to manage:

When everything is set up, click on SAVE, and your profile is completed! You are now ready to use Dolphin{anty} via UI or code.

Use Dolphin{anty} Via The UI

The power of anti-detect browsers rely in allowing you to create different profiles and letting you use the browser with one instance, but different profiles. So, after you created the profiles, click on START to launch the instances:

Dolphin{anty} will launch a new browser instance, allowing you to manage as many profiles as you have created and activated. Below is the expected result:

That’s it for using Dolphin{anty} via UI!

Use Dolphin{anty} Via Code

Before using Dolphin{anty} via code, you have to create an API key. To do so, navigate through the API panel in the web app:

Now you can connect to a profile through a port generated at startup and automate the browser using tools like Puppeteer, Playwright, Selenium, and others.

Basic automation you can do includes the following:

1. Start a profile via API with DevTools Protocol enabled.

2. Connect to the profile’s port using a browser tool.

3. Run your own automation script through the open connection.

Dolphin{anty} allows you maximum flexibility, so you can use your favourite programming language. For example, below is how you can write an authorization script:

import requests
api_url = "<http://localhost:3001/v1.0/auth/login-with-token>"
token = "your-api-key"
request_data = {"token": token}
headers = {"Content-Type": "application/json"}

response = requests.post(api_url, json=request_data, headers=headers)
if response.status_code == 200:
	print("OK", response.json())
else:
	print("Error", response.status_code)

If the response is successful, you will receive a message like the following:

{"success": true}

Discover how to use Doplhin{anti} via API by reading the documentation!

Pros and Cons of Dolphin{anty}

Like any tool, Dolphin{anty} has its strengths and weaknesses. Here is a breakdown of what you need to know before deciding if it fits your stack.

👍 Pros:

• Top-tier fingerprinting: The ability to generate real, unique fingerprints for every profile is its biggest selling point. It goes beyond simple user-agents, making your scrapers look genuinely human.

• Built-in automation tools: The “Scenarios” builder and the Synchronizer are massive time-savers. You can automate routine warm-up tasks or replicate actions across dozens of profiles without writing a single line of code.

• Team-centric design: If you work with a team, the ability to transfer profiles and share them instantly is invaluable. It removes the friction of sharing session data manually via files or text.

👎Cons:

• REST API complexity: This is a significant friction point for developers. Unlike other solutions that offer native SDK wrappers, Dolphin{anty} relies only on REST API calls for automation. This adds “boilerplate” complexity compared to simply importing a library.

• Resource intensive: Running multiple browser profiles with full fingerprinting requires significant system resources. You will need a powerful machine if you plan to run dozens of concurrent sessions locally.

Conclusion

In this article, you discovered Dolphin{anty}, a flexible anti-detect browser that can be used both via UI and via code. As you’ve learned, it comes packed with interesting features that can speed up your processes. In particular, we found that the “Scenarios” feature is the one that actually makes it stand out.

So, let’s discuss in the comments: Were you already using Dolphin{anty} before reading this article? What’s your experience with it?

We are not talking about a cease-and-desist letter or a Terms of Service dispute. Google did not send SerpApi any communication before filing the lawsuit. No cease-and-desist, no attempt to resolve their concerns directly. SerpApi told us this was highly unusual, and that had Google reached out, they might have learned that their claims lack merit.

Google is invoking the Digital Millennium Copyright Act, specifically Section 1201, the anti-circumvention provision. The same statute originally designed to prevent people from cracking DVD encryption is now being pointed at a SERP scraping API.

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

We reached out to both Google and SerpApi for comment on this case. Google did not respond. SerpApi did, and we will include their statements throughout this article where relevant.

Let us break down what happened, why it matters, and what it could mean for anyone who scrapes the web for a living.

The Facts

Google’s complaint tells a straightforward story. SerpApi, founded in 2017 by Julien Khaleghy, operates a paid API that sends automated queries to Google Search and returns the results as structured JSON. Google estimates that SerpApi sends hundreds of millions of artificial search requests per day, and that this volume has increased by as much as 25,000% over the past two years.

In January 2025, Google deployed a technological protection measure called SearchGuard. SearchGuard works by sending JavaScript challenges to incoming search queries. For regular browser users, the challenge is invisible: the browser runs the JavaScript, sends back the expected response, and the search results load normally. For automated systems, the challenge is a wall. Bots that cannot execute JavaScript or that fail behavioral checks get blocked.

According to Google’s complaint, SerpApi’s response to SearchGuard was to build circumvention mechanisms. The complaint alleges that SerpApi creates “fake browsers using a multitude of IP addresses that Google sees as normal users,” misrepresents device and location information when solving challenges, and syndicates authorization tokens from legitimate requests to unauthorized machines around the world. Google also alleges that SerpApi uses automated means to bypass CAPTCHAs that SearchGuard deploys as a secondary verification layer. SerpApi disputes these factual allegations.

The complaint cites SerpApi’s own blog posts, where the company reportedly described SearchGuard as making “web scraping more difficult” but claimed to be “fortunate to be minimally impacted” because its services had “already pre-solved Google’s JavaScript challenge.”

The Legal Theory

This is where it gets interesting for the scraping industry, because Google chose not to sue under the Computer Fraud and Abuse Act (CFAA). That would have been the traditional route. Instead, Google went with the DMCA.

The context matters. The CFAA path has been significantly narrowed by the hiQ Labs v. LinkedIn case. In that landmark decision, the Ninth Circuit held that scraping publicly available data does not violate the CFAA, and warned against allowing companies to create “information monopolies.” The Supreme Court vacated and remanded the case under its Van Buren ruling, but on remand, the Ninth Circuit reaffirmed its original position.

After hiQ, the CFAA is a much weaker weapon against scraping of publicly visible content. Google needed a different legal framework. Section 1201 of the DMCA provides one.

Section 1201 has two relevant provisions. The first, Section 1201(a)(1)(A), prohibits the act of circumventing a technological measure that effectively controls access to a copyrighted work. The second, Section 1201(a)(2), prohibits trafficking in technology designed to circumvent such measures. Google’s complaint invokes both.

The argument chain goes like this: Google’s search results contain copyrighted content, specifically images in Knowledge Panels licensed from third parties, merchant-supplied product images in Google Shopping, and licensed content from Google Maps. SearchGuard is a technological measure that controls access to these search results pages (and therefore to the copyrighted works within them). SerpApi circumvents SearchGuard. Therefore, SerpApi violates Section 1201.

Each act of circumvention carries statutory damages of between $200 and $2,500. Google alleges billions of individual circumventions. Do the math, and the potential damages exceed what SerpApi could ever pay. Google itself notes in the complaint that SerpApi “reportedly earns a few million dollars in annual revenue, but already faces liability that is orders of magnitude higher and growing.”

Check the TWSC YouTube Channel

SerpApi’s Position

When we reached out to SerpApi, they were clear about their stance. On the fundamental legality of what they do, SerpApi told us: “We embrace the term ‘scraping,’ and we practice it legally and transparently. SerpApi accesses publicly visible search results, the same ones available to any browser, and delivers clean, structured JSON back to our customers. We’ve operated this way since 2017, serving developers, researchers, and businesses who need reliable access to public information at scale.”

On the legal boundaries of automated access to search results, their position is equally direct: “The law on this is clear, and we’re prepared to defend that position in court. Scraping is legal, and we stand behind our products and customers. Our API replicates real-time searches with no login, no bypass of any paywall, and no access to anything that isn’t already available to anyone with a browser. U.S. courts have upheld this repeatedly; hiQ Labs v. LinkedIn is a key precedent. The data Google surfaces lives on the open web. Google didn’t create it.”

In February 2026, SerpApi filed a motion to dismiss. Their arguments include the assertion that the DMCA is a copyright protection statute, not a website protection statute, and that Google is improperly trying to use it to control access to public portions of its website. They also argue that mimicking browser behavior to access publicly available pages is not the same as cracking encryption or disabling authentication, and that any ambiguity in the definition of "circumvention" must be given its narrowest reasonable reading, citing the "First Amendment interest in maintaining accessibility of the Internet as an open forum."

SerpApi also pointed out what they see as an absurdity in Google’s theory. If statutory damages were calculated at scale, the total “would exceed U.S. GDP.” Congress, they argue, never intended Section 1201 to be used this way.

On the DMCA claim specifically, SerpApi told us: “The DMCA’s anti-circumvention provision was designed to protect copyrighted works, full stop. Google is not protecting access to copyrighted works. Google is improperly attempting to use the DMCA to limit access to the public portions of its website. We believe that the law is on our side.”

Need help with your scraping project?

The Hypocrisy Argument

SerpApi is not shy about making this point. In a blog post about the lawsuit, they argue that Google’s case threatens access to public data on the open internet and this resonates widely in the scraping community. As they told us: “Google indexed the web without anyone’s permission. That’s how search works. Now it’s trying to pull up the ladder behind it, prohibiting the practices that it used, and still uses today, to build its business empire. That’s why SerpApi is standing up to Google. Not just to protect our business, but to protect legal competition and open access to public information on the internet.”

Google Search operates by crawling, indexing, and presenting content from billions of websites. Many of those website owners never explicitly consented to being indexed. Google’s position has always been that robots.txt provides the mechanism for opting out, and that the default state of the open web is crawlable. Now Google is arguing that its own search results should be exempt from the same logic.

The irony is not lost on legal commentators either. Above the Law( described the case as Google “pulling up the ladder after climbing it.” Eric Goldman’s blog published an extensive guest analysis arguing that Google’s DMCA strategy represents an attempt to relitigate hiQ Labs through a different statutory framework.

Why This Matters Beyond SerpApi

If Google’s legal theory prevails, the implications extend far beyond one API company. The core question is whether deploying an anti-bot system on a publicly accessible website is enough to invoke federal copyright law against anyone who bypasses it.

Think about what that means in practice. Every CAPTCHA, every JavaScript challenge, every behavioral analysis system deployed on a public website could potentially become a “technological protection measure” under Section 1201. Any scraper that solves a CAPTCHA, executes JavaScript to render a page, or rotates IP addresses to avoid detection could be committing a federal offense.

This is not hypothetical. The legal theory applies to any website that hosts copyrighted content (which is almost all of them) and deploys some form of bot detection (which is increasingly all of them).

Eric Goldman’s blog highlighted this exact concern. The guest analysis by Kieran McCarthy warns that accepting Google’s theory would allow any website deploying anti-bot technology to invoke federal law against circumvention, “transforming speed bumps and CAPTCHAs into federally enforceable copyright perimeters.”

The Electronic Frontier Foundation has also weighed in. Staff attorney Tori Noble stated that “the right to scrape publicly available information keeps the Internet free and open,” cautioning that overly broad DMCA interpretations undermine innovation and research.

SerpApi made a similar point when we asked about the impact on consumers: “Scraping-powered services benefit all kinds of consumers who use the web every day. Scraping helps to maintain the free and open flow of information across the internet, ultimately encouraging things like price transparency, competition, and informed decision-making, all to benefit consumers. Expanding the DMCA as Google has suggested would only benefit the largest tech incumbents and hinder transparency and healthy competition.”

The Emerging Legal Pattern

Google’s lawsuit does not exist in isolation. In October 2025, Reddit filed a 41-page complaint against SerpApi, Perplexity AI, Oxylabs, and AWMProxy in the Southern District of New York. The complaint is far more aggressive than Google’s, both in tone and in scope: six legal counts including three separate DMCA claims, unfair competition, unjust enrichment, and civil conspiracy.

Reddit’s framing is vivid. It describes the defendants as “similar to would-be bank robbers, who, knowing they cannot get into the bank vault, break into the armored truck carrying the cash instead.” AWMProxy is characterized as “a former Russian botnet.” Perplexity is compared to “a North Korean hacker.” The language is clearly designed to make scrapers look like criminals.

The underlying theory is similar to Google’s. Reddit has signed licensing deals with both Google and OpenAI to grant them programmatic access to its data. Companies that want Reddit content at scale are expected to pay for it. But when scrapers circumvent SearchGuard to harvest Google’s search results, they also harvest Reddit content without paying a cent. According to data Reddit obtained through a subpoena to Google, the three scraping defendants accessed almost three billion Google SERPs containing Reddit content in just two weeks during July 2025. SerpApi alone accounted for over 1.8 billion of those page accesses. Like Google, Reddit did not send SerpApi any communication before filing suit. SerpApi disputes these figures and the other factual allegations in Reddit’s complaint, and has filed a motion to dismiss in that case as well.

Reddit also produced a piece of evidence that reads like a detective novel. It created a hidden “test post” that could only be crawled by Google’s search engine and was not otherwise accessible anywhere on the internet. Within hours, the contents of that post appeared in Perplexity’s “answer engine.” The only way Perplexity could have obtained that content was through scraping Google’s search results. Reddit calls this technique the equivalent of “marked bills” in a bank robbery investigation.

The Reddit complaint also reveals a detail that connects directly to our industry: after Reddit sent a cease-and-desist letter to Perplexity in May 2024, Perplexity’s citations to Reddit content did not decrease. They increased forty-fold.

And in December 2025, in Ziff Davis v. OpenAI, a federal judge in the Southern District of New York ruled that robots.txt files do not “effectively control access” under Section 1201. Judge Sidney Stein compared robots.txt to a “keep off the grass” sign that “relies on readers to decide to comply rather than enforcing any kind of access control itself.” The ruling is important because it sets a baseline: passive, voluntary measures are not enough to trigger DMCA protection.

But SearchGuard is not robots.txt. It is an active system that executes JavaScript, performs behavioral analysis, deploys CAPTCHAs, and makes real-time decisions about whether to grant access. Whether this kind of system meets the “effectively controls access” standard is the open legal question. The answer will likely set the direction for the entire industry.

Legal commentators have identified what they call the “DMCA 1201 scraping strategy”: platforms deploy technological protection measures specifically to create legal standing under Section 1201, then sue when those measures are circumvented. The sequence is intentional. Deploy, document, sue. Whether courts view this as legitimate copyright protection or as strategic rent-seeking will determine the outcome.

There is also a relevant doctrinal debate. The Lexmark case in the Sixth Circuit introduced the “front door/back door” argument: if a house’s front door is unlocked, putting a lock on the back door does not mean the house is “access-controlled.” Applied here: if anyone with a regular browser can access Google Search results, does deploying SearchGuard against automated systems meaningfully “control access” to the copyrighted works within those results?

The AI Angle

There is one more layer worth noting. As Search Engine Land reported, OpenAI used SerpApi to scrape Google Search results for ChatGPT responses on current events, after Google declined to provide direct access to its search index. SerpApi listed OpenAI as a customer on its website as recently as May 2024 before removing the listing. Other reported customers include Meta, Apple, and Perplexity.

This context matters because Google already has a massive structural advantage in the AI race when it comes to fresh web data. Cloudflare CEO Matthew Prince put numbers on it: “For every one page that OpenAI sees, Google is seeing 3.2 pages.” Against Microsoft, the ratio is 4.8 to 1. The reason is simple. Publishers cannot block Googlebot without disappearing from search results. So Google gets access to the web at a scale that no competitor can match, and it can use that data not just for search but also for training and running its AI products.

In this context, suing companies that make it easier for competitors to scrape Google’s search results is not just about protecting copyrighted images in Knowledge Panels. It is also an act of defense of a competitive advantage. If OpenAI or any other AI company can get structured search data through SerpApi, they partially close the gap that Google’s crawler monopoly creates. Shutting down that channel through litigation serves Google’s position in the AI race, even if the complaint is framed purely in terms of copyright protection.

What Happens Next

The case is still in its early stages. SerpApi filed its motion to dismiss on February 20, 2026. According to the court docket, the initial case management conference before Judge Yvonne Gonzalez Rogers is scheduled for March 30, 2026, and a hearing on the motion to dismiss is set for May 19, 2026.

If the motion to dismiss fails and the case proceeds to discovery and trial, it will force courts to answer questions that have been left open since hiQ. Is a JavaScript challenge a “technological protection measure” under the DMCA? Can anti-bot systems on publicly accessible websites invoke federal anti-circumvention law? Does the DMCA protect the act of accessing a public webpage, or only the copyrighted works behind genuine access controls like encryption and authentication?

For the scraping industry, the stakes are high. A ruling in Google’s favor would give any website with copyrighted content and a bot-detection system a federal cause of action against scrapers. A ruling in SerpApi’s favor would confirm that the DMCA was not designed to protect public webpages from automated access, regardless of the technical measures deployed.

We will follow the case closely. Whatever happens, the days of operating in a legal gray area are coming to an end. The courts will have to draw a line, and that line will define the rules for the next decade of web scraping.

*Disclaimer: We are not lawyers. This article represents our analysis of publicly available court filings and legal commentary. Consult legal counsel for advice specific to your situation.*

The scraping industry is well aware of this problem. A recent analysis by ScrapeOps found that even though proxy prices have dropped by 67% over the past five years, the cost per successful payload has actually increased by 133%, mostly because anti-bot defenses now require heavier infrastructure. When each request costs more, wasting them on unchanged pages hurts even more.

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

Several approaches try to solve this. Tools like changedetection.io monitor pages for visual or structural changes and alert you when something is different. On the more technical side, recently explored using SimHash as a client-side fingerprint to determine whether a document has changed since the last crawl, without downloading the full body. These are valid strategies, but they all share one trait: they require you to build and maintain the change detection logic yourself.

What you might not know is that the HTTP protocol already has a native mechanism for this, and it has been part of the spec since 1999. It is called conditional requests, and it lets the server itself tell your scraper “nothing has changed” by responding with a 304 status and zero bytes of body. No diffing, no hashing, no client-side state management beyond storing a single header value.

We have written about proxy cost optimization before in articles like Optimizing Proxy Usage for Large-Scale Scraping and Analyzing the Cost of a Web Scraping Project, but we have never covered this technique. In this article, we will test it against real e-commerce sites and measure exactly how much bandwidth and money it can save.

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

How HTTP caching works (the short version)

When a web server responds to a request, it can include headers that describe the freshness and identity of the content. Two of these headers are relevant for our purposes.

The first is ETag, short for Entity Tag. It is a string that uniquely identifies a specific version of a resource. Think of it as a fingerprint of the page content. When the content changes, the ETag changes.

The second is Last-Modified, a timestamp indicating when the resource was last updated.

These two headers enable what HTTP calls conditional requests. The idea is simple. After your first request, you store the ETag (or the Last-Modified value) returned by the server. On the next request to the same URL, you send it back using the `If-None-Match` header (for ETags) or `If-Modified-Since` (for timestamps). The server compares your stored value with the current one. If they match, the server responds with status 304 Not Modified and an empty body. If they do not match, you get a regular 200 response with the fresh content.

A 304 response contains zero bytes of body. For a proxy billed per GB, that is a request that costs almost nothing in bandwidth.

The tools we used

The HTTP caching technique itself is protocol-level and works with any HTTP client that allows setting custom headers. You could implement it with Python’s `requests`, httpx, or even raw curl.

For this article, we used curl_cffi, a Python HTTP client built on top of curl-impersonate. Its main strength for our purposes is TLS fingerprinting: it can impersonate the TLS handshake of real browsers (Chrome, Firefox, Safari), which prevents e-commerce sites from blocking the request before we even get to test caching behavior. Without TLS fingerprinting, some of the e-commerce targets we wanted to test would have returned 403 immediately, making it impossible to evaluate their caching support.

Then later in the article, we’ll see if we can use the same approach with Scrapy.

Check the TWSC YouTube Channel

The audit methodology

Before attempting conditional requests, we need to check whether a target supports them. We wrote a simple audit function that makes two requests to any URL.

The first request is a standard GET. We capture the ETag, Last-Modified, and Cache-Control headers from the response, along with the response body size.

If an ETag or Last-Modified header is present, we make a second request with the corresponding conditional header (If-None-Match or If-Modified-Since). If the server responds with 304, the site supports conditional requests and we measure the bandwidth saving.

import time
from curl_cffi import requests


def audit_caching(url: str) -> dict:
    headers = {
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        "Accept-Language": "en-US,en;q=0.9",
        "Accept-Encoding": "gzip, deflate, br",
    }

    resp = requests.get(url, headers=headers, impersonate="chrome", timeout=30)

    resp_headers = {k.lower(): v for k, v in resp.headers.items()}
    etag = resp_headers.get("etag")
    last_modified = resp_headers.get("last-modified")
    cache_control = resp_headers.get("cache-control")
    response_size = len(resp.content)

    result = {
        "url": url,
        "status": resp.status_code,
        "etag": etag,
        "last_modified": last_modified,
        "cache_control": cache_control,
        "response_size_bytes": response_size,
        "supports_304": False,
    }

    if etag or last_modified:
        time.sleep(2)

        cond_headers = dict(headers)
        if etag:
            cond_headers["If-None-Match"] = etag
        if last_modified:
            cond_headers["If-Modified-Since"] = last_modified

        cond_resp = requests.get(
            url, headers=cond_headers, impersonate="chrome", timeout=30
        )

        result["conditional_status"] = cond_resp.status_code
        result["conditional_size_bytes"] = len(cond_resp.content)
        result["supports_304"] = cond_resp.status_code == 304

    return result

You can find the code in our GitHub repository reserved to paying users, inside the folder 99.CONDITIONAL_SCRAPING.

Need help with your scraping project?

Shopify stores: full conditional request support

We focused our testing on Shopify stores because, while working on various scraping projects, we came across several Shopify-hosted sites that had this caching system enabled. Shopify powers hundreds of thousands of online stores and is one of the most common scraping targets in e-commerce, so the finding felt worth investigating systematically. The results were clear: Shopify stores with the native page cache enabled support conditional requests out of the box.

Allbirds, Kylie Cosmetics, and Brooklinen all returned 304 responses consistently. Here is what we measured on Allbirds:

URL: https://www.allbirds.com/products/mens-tree-runners.json
Status: 200
Response size: 7,961 bytes

Caching headers:
  ETag: "page_cache:11044168:ProductDetailsController:de822deb7906aa6f9932541f4fe3dae9"
  Last-Modified: not present
  Cache-Control: not present

Conditional request support:
  304 Not Modified: YES
  Conditional response size: 0 bytes
  Bandwidth saving: 100.0%

The saving is 100% because the 304 response body contains exactly zero bytes. The only cost is the request/response headers, which are a few hundred bytes.

This behavior was consistent across three types of Shopify endpoints. The Product HTML page is the standard storefront URL that a browser would load (e.g. /products/mens-tree-runners), which includes the full rendered page with images, reviews, and theme assets. The Product JSON endpoint is the same URL with .json appended (e.g. /products/mens-tree-runners.json), which returns only the structured product data: variants, prices, inventory, and metadata. The Catalog JSON endpoint (/products.json) returns the first page of the store’s entire product catalog in a single response.

We ran repeated conditional requests on each endpoint and confirmed that all returned 304 consistently. The ETag stayed stable as long as the product data did not change.

Read more

Lots of tools and libraries for the main programming languages have indeed driven web scraping to significant growth. The need companies have for data also makes such growth the actual reason for existing.

In this article, I’ll talk about Kadoa: A tool that lets you create resilient scraping workflows in minutes. I’ll show you its strengths, why you should consider it, and how it works, with a practical guide.

Let’s dive into it!

Before proceeding, let me thank NetNut, the platinum partner of the month. They have prepared a juicy offer for you: up to 1 TB of web unblocker for free.

Claim your offer

What is Kadoa?

Kadoa is a web scraping tool that automatically and programmatically extracts web data at scale. You can use it either via the UI or via code, as it has SDKs and provides you with REST APIs.

The best part of using it is that you can just paste the target URL and the tool retrieves the data for you. Forget about anti-bot measures, fingerprinting issues, or proxy management: Kadoa does all of that for you very simply. Also, thanks to its AI engine, it can automatically recognize the structure of the data you want to scrape from a target website. So, say goodbye also to CSS selectors and any other strategy you use to go beyond the DOM using LLMs.

Using the right tool is just the first steps for a successful data extraction pipeline. Having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

Why Consider Kadoa for Your Web Scraping Projects?

The top reasons why you should consider Kadoa are the following:

• Scrape via workflows: Kadoa’s UI is built to help you set scraping workflows step-by-step. Insert your target URL(s), define the data schema (or let AI make the work for you), and choose to scrape all the available pages or to remain on page and see the agent work for you.

• Write code only if you need it: Other than the UI, Kadoa provides you with Python and JavaScript SDKs and a wide set of REST APIs you can call. This allows you to create workflows via UI, but to manage and call them via code if you need to.

• Integrated data quality management: Before starting the scraping process of your target data, Kadoa allows you to manage data quality. In practice, it allows you to set data quality rules or to manage the rules it provides you, thanks to its AI agent.

• Easy proxy management: If you’ve been scraping for a while, you know that you have low chances of successfully scraping the majority of the content you need without using proxies. Using proxies is not a very big issue if you are used to it and if you already have a favourite provider. However, Kadoa simplifies proxy management. It already provides you with a list of countries you can choose from and, under the hood, it manages everything that’s needed to integrate proxies in your workflow.

• Scheduling feature: There are cases where you need to scrape the same target data from time to time. Or, eventually, you’d like to be notified when data in a target page has changed. Kadoa provides both these features. You can choose to schedule your workflow to scrape at precise time intervals. You can also choose among different notifications, one of which is getting notified when data is changed.

Kadoa’s Main Features

Below is a list of Kadoa’s top features to help better understand its potential:

• Simple and intuitive UI: Kadoa’s UI is simple and intuitive. It allows you to create workflows in minutes. Every scraping workflow is subdivided into steps, and Kadoa provides you with different screens. In a matter of a few minutes, you can define your preferred setup, insert the target page(s), and leave it scraping for you.

• Chrome extension: Other than the UI, Kadoa provides you with a Chrome extension. If you are a Chrome user, this feature allows you to define everything you need directly on the target page, then trigger the workflow to let Kadoa’s agent start scraping.

• Code integrations: If you are a developer or if you simply need to invoke your workflows via code, Kadoa offers you two possibilities. It provides you with Python and JavaScript SDKs in an open-source repository, so that you can use custom code to invoke your scrapers. Also, if you like to use code but prefer REST APIs, Kadoa provides you with several endpoints.

• Scraping suitable for structured or unstructured data: One of the difficult aspects you may encounter when manually scraping websites is defining how to grab unstructured data. This is one of the typical use cases where you could use AI to detect patterns in data in your scraping projects. The good news is that you don’t need to come up with imaginative solutions. Kadoa automatically retrieves unstructured data for you thanks to its AI engine.

• Data schemas definition: The tool provides you with a feature that allows you to define recurrent data structures. This can be helpful when you retrieve similar data from different websites. If you leave its AI engine to automatically define the data structure, in such cases, you could lose consistency across similar data.

• Proxy and anti-detection features: Forget about anti-bot measures and proxy management. Kadoa manages anti-bot solutions under the hood. It also provides you with a predefined list of locations you can choose from, and it will automatically set coherent proxies.

• Error handling: It provides you with advanced error handling management. Common cases are when the target site goes offline, is under maintenance, or encounters a technical issue. When this happens, Kadoa detects the problem, it notifies you, and automatically retries the data extraction. If recovery still fails, its support team is notified and investigates.

• Integration capabilities: The software allows you to integrate with several third parties. One interesting one is the integration between n8n and Kadoa, which allows you to get your scraping automation workflow a step forward.

• Pricing model and usage graphs: Kadoa offers a free tier option, for which you can use 500 credits. Its pricing model is based on credit consumption, and it provides you with a UI section where you can see a graph of the consumption.

• Extensive docs: Kadoa has extensive documentation that covers both UI and API usage.

Check the TWSC YouTube Channel

Hands-on Kadoa: Step-by-step Scraping Tutorial

In this section, I’ll show you how to use Kadoa on an actual scraping task via the UI. The workflow will retrieve Intel’s historical price from Yahoo Finance:

In this scraping workflow, I will:

• Set the target web page.

• Define the data schema.

• Set scheduling options and notifications.

• Retrieve the actual data.

Before starting the actual workflow, log in to Kadoa. Below is the first access page you will see:

Perfect! You are now ready to create your first scraping workflow with Kadoa.

Step #1: Create a New Workflow

From the main page, click on Add workflow to create a new one and paste the target URL. The Proxy location box allows you to select a country where proxies are localized; leave it to AUTO to let the tool automatically manage it. Click on Continue to proceed with the next step:

Note that inside the Enter one or more URLs box, you have to insert the target page. If the target page is more than one, you can insert all the target pages you are interested in.

Alright, you created a new workflow in Kadoa. Let’s proceed with the next step and customize it!

Step #2: Define the Data Schema

As the next step, define the data schema:

If you want to insert the schema manually, Kadoa already provides you with some predefined schemas. For this tutorial, I’ve chosen to let AI do the job. So I selected AI Suggest Fields.

The system, then, asks you how you want to navigate the data. For the sake of this example, I decided to scrape only the current page from the target one, but you can also choose among three different options:

After clicking on Continue, the agent will start doing its job:

Step #3: Review Extracted Fields and Schedule the Workflow

Because I let AI work, the agent automatically tries to extract the data from the target page. But before proceeding, Kadoa asks for your review:

As you can see from the previous image, the agent has correctly detected the data to extract from the target page. Also, this job is finely improved as the tool provides you with a screenshot of the data it will extract, so that you can visualize it even better.

In the next step, you have to define the scheduling:

For the sake of this example, I decided to run the workflow only once. But, as you can see, you can choose among several scheduling options.

Step #4: Set Notifications and Final Details

As the next step, define the way you want to be notified:

In this case, I decided to be notified via email if the workflow fails. You can add different notification channels by clicking on Add channel:

Next, define the latest details of your scraping workflow:

Before starting with the actual scraping, the system asks you to approve the sample data it proposes to you or to review the data quality rules:

By clicking on Review rules, the tool provides you with automated data quality rules. You can select them if you think this will improve the quality of the scraping result:

When you are done reviewing quality rules, click on Approve. The actual scraping workflow will start and will be queued:

Et voilà! You have launched your first scraping workflow with Kadoa.

Need help with your scraping project?

Download Data, See Logs and Statistics in Kadoa

The workflow section reports all the workflows you created, their status, and the token consumption for each scraper:

By clicking on one workflow, you can see the data it retrieved and can decide the format you want to download it:

The Activity log page reports detailed logs of every action occurred to your workflows:

The Usage page reports graphs of the trend in terms of active workflows and the number of rows extracted for workflow, as well as the remaining total tokens on your plan:

Manage Kadoa’s Workflows via APIs

As introduced before, Kadoa provides you with several endpoints for making calls via REST APIs. The APIs allow you to perform several actions that are not strictly necessary for workflows already created. For example, you can start crawling sessions and create data schemas.

Before using the API, get your API Key under the Settings page:

If you want to manage already existing workflows, either created via the UI or APIs, you have to use the specific workflow’s ID via the UI.

Then you can perform several actions by invoking the REST endpoints. For example, you can schedule a particular workflow for later:

curl --request PUT \\
  --url <https://api.kadoa.com/v4/workflows/{workflowId}/schedule> \\
  --header 'Content-Type: application/json' \\
  --header 'x-api-key: <api-key>' \\
  --data '
{
  "date": "2025-02-07T10:00:00.000Z"
}
'

Where you have to insert the following:

• workflowId : Is the ID of the workflow you want to schedule.

• <api-key>: Is your KadoaAPI key.

• The actual date you want your workflow to start the scraping task. You have to use the ISO format for the date in UTC.

Kadoa: Final Comments

After analyzing and testing the tool, I can say the following are its main advantages and disadvantages:

👍 Pros:

• Ready for AI integration. You can download the scraped data or integrate it into your AI projects directly via API.

• Suits all the user needs, as it provides APIs, SDKs, and the UI.

• Supports structured output formats, including JSON.

• Offers virtually unlimited scalability on the side of infrastructure management and the number of URLS to scrape.

• Focuses on data quality before scraping, not later.

👎 Cons:

• Currently, it supports only 5 proxy locations.

• You can’t scrape all the websites you’d like:

Conclusion

In this article, I’ve presented Kadoa: An AI-powered scraping tool that helps you simplify your scraping projects. As you’ve seen, this is a ready-to-use tool that creates scraping workflows in minutes via UI and also supports code.

Let us know in the comments: Did you know this tool before? Have you already tested it?